© 2026 Thibaud Geisler - SIRET 880 419 122 00036
Python/FastAPI REST gateway that aggregates Beatport, Bandcamp and SoundCloud behind a single output contract, used by about ten DJs. Reliable access to sources with no public API, centralised title cleanup, per-API-key usage tracking. In production.
Several of my tools need the same music metadata: tracks, artists, labels, release dates, BPM, key. Starting with a desktop application that re-tags audio files, used by about ten DJs I know. And a collection service for artists and labels (in development).
The problem: no single platform covers the whole catalogue. So I query three of them, each for what it does best:
So I built a gateway: it queries all three and always returns the same format, whichever one answered.
My role: design, development and operations, solo.
The tagging application is used by about ten DJs, each with their own API key. This makes it possible to revoke one access without touching the others and to track each user's usage. Sentry was not enough: it reports errors, but not usage.
Solutions: one API key per user, individually revocable, without adding a database. For tracking, a self-hosted logging stack (VictoriaLogs, Grafana): 30 days of history per API key, a usage dashboard and an alert when a user only gets errors. On the security side, a full review with no major flaw and a load-tested rate limit.
One route answered with a 200 status and a well-formed list, except part of the results were missing. The source page advertised a wrong item count: 5 announced for 355 actual, 1 for 736, 0 for 322.
The caller had no way of noticing: no error, a valid response, just items missing from the roll call.
Solutions: fifteen combinations tested to find where the problem came from, then a move to another access channel, properly paginated. I dismissed the quick fix, reading a page whose counter is correct: it capped at 21 items, so it reduced the error instead of removing it.
Titles returned by the platforms contain noise: promotional mentions, label and catalogue number stuck to the title, artists mixed into the track name. At first, I left that cleanup to each consuming application. The problem is that they all redid it their own way. Some titles could still be polluted and above all a lot of code was duplicated.
Solutions: I centralised the cleanup in the gateway, the only component that sees the platforms' structured fields. When the source provides structured data, it always takes precedence over parsing the title. The original text stays available: nothing is lost if the cleanup gets it wrong. A set of 914 real titles checks the rules on every test run.
Whichever platform is queried, the routes return the same objects. An application consuming the API writes its logic only once and switches source without rewriting anything.
Technical challenges: every source has its own structures and its own conventions. One puts remixers in dedicated fields, another settles for a free-text label name. On top of that, three pagination schemes that cannot be reconciled, one without a reliable total and another with no total at all.
Solutions: a shared pivot schema with one simple rule: nothing gets invented. A field the source does not provide comes back empty, never filled in by guesswork. Pagination goes through an opaque cursor the caller passes back as-is without reading it, which shields it from changes on the source side. Finally, an empty list is never an error, so the caller can tell "this source has nothing" apart from "this source is broken".
Every Beatport route got its token by reading a page of the Beatport website. One evening, that website was blocked by an anti-bot challenge for almost two hours: the whole source went down, while Beatport's data API, hosted separately, kept answering. Before touching the code, I checked from four different connections that the block hit everyone and not just my server.
Solutions: rather than waiting for things to return to normal, I removed that dependency: the token is obtained through another path, which no longer goes through the website. I also refused to keep the old path as a fallback, since it depended on the same website.
The real risk with a scraper is the silent failure: a source changes its structure and extraction breaks without anyone seeing it.
Solutions: errors go to Sentry, grouped on a key that includes the source and the status code returned. A broken source therefore raises one alert instead of several hundred. Data is scrubbed before sending and local-variable capture is turned off, otherwise the configuration ends up in the stack traces with an API key in clear text. Finally, a request ID follows the whole path, from the alert to its matching log line.
The OAuth API caps how many tokens you can generate. Creating one per incoming request is enough to burn through the quota at the first traffic peak.
Solutions: a centralised token service, with in-memory cache, early refresh and de-duplication when several requests ask for one at the same time. A circuit breaker remembers the failure for thirty seconds and answers without hitting the network; past that delay, a single request actually retries the call. Finally, a network cache can skew a token's age, so I rely on the expiry time returned by the server rather than on a "valid for one hour" style duration.
A call went through with curl but was refused from Python, even though I was using the same IP and the same headers. The deciding factor was therefore not the IP but the HTTP client's TLS fingerprint.
Solutions: a change of HTTP client, applied once and for all to the session shared by every outbound call. I documented the limit of this approach from day one: it acts on the transport layer, not on a JavaScript challenge. When an incident came up later, that note kept me from looking in the wrong place.
Source code available on request (not public by strategic choice).