The endpoint that was switched off

Drift needs an answer to one question: given this article, which other articles are near it? That is what the threads under each card are. Without a decent answer the app is a random article button.

Wikimedia had a REST endpoint for exactly this, returning a page's related articles. I built around it. It returns 403 to anyone outside Wikimedia's own apps, and has for a while; the documentation had not caught up. I found out after the feed, the cards and the storage were already written.

The first replacement I tried was the list of links in the article, which is available through the API and looks like the obvious answer.

It is not, and the reason is worth writing down. The links in an article are the words that needed explaining, not the articles nearest to it. A composer's page links to a year, a country and a city. All are related in a technical sense and none is a direction anyone wants to take.

• Volume. Hundreds per article, with no useful ordering.
• Wrong kind. Years, places, units of measurement. Prose scaffolding rather than subject matter.
• One direction. Links point outward. What a page is about is often better indicated by what points at it.

What worked was Wikipedia's own search, which supports a more-like-this query: name an article and it returns pages whose text sits closest to it. That is a different question from “what does this link to”, and closer to the one I actually had.

The results are noticeably better. From Impressionism you get Post-Impressionism, Claude Monet, En plein air, rather than 1874 and France.

The old endpoint returned titles. Building a card from a title then needed another request each for the description, extract and thumbnail, so twenty related pages meant twenty one requests.

The search query can return all of those fields alongside the results in the same request, so it is one call instead of twenty one. That matters more than it would elsewhere: this is a donated public API and Drift is a hobby project pointing a feed at it.

Check the riskiest call first. I had built most of the app before discovering the one request it could not do without did not work, and a few minutes of curl at the start would have caught it.

And when a dependency goes away, do not assume the replacement should have the same shape. My first instinct was to reconstruct the old endpoint's behaviour from other calls, which would have been worse than what I ended up with.

What Drift asks of Wikipedia now is described on sources.