Once I had to deal with a problem that at first looked like a piece of real black magic. In the end it turned out to be very simple and logical system behaviour, but catching it took far longer than it should have.
Some context about the project. At the time the team was just a few developers. The project itself was written in PHP with a fairly large codebase. In places the business logic was quite complex, so queues were used heavily. The queue driver was the database — that is, the queue consumers worked straight through the DB (it is one of the standard drivers). A very simple “out of the box” solution that usually gets set up at the start to keep the stack lean. Quite often it stays that way for a long time if there is no need to scale the project.
And then at some point an unpleasant bug showed up in the very heart of the project, the sales funnel. In some cases it broke the payment flow, so it had to be fixed as urgently as possible.
The bug itself was not particularly complex, so I solved it quickly and deployed the fix to the staging environment. But to my surprise, after running a few tests I noticed the bug had not gone anywhere — it just showed up a bit less often. That was a big surprise: locally the problem no longer reproduced (yes, yes, “it works on my machine!”), but on staging things were far from smooth.
After double-checking everything in detail, I was 100% sure the problem was solved. I confirmed that CI/CD had run correctly and that the codebase and containers on the server were up to date… But the bug was up to date too.
Here you can pause and think about what was really going on. Or just look under the spoiler. My own pause dragged on for about four hours of debugging and trying to understand what on earth was happening.
Spoiler
It turned out that one of the developers had connected the staging database to their local environment. On their machine the project was running on old code. As a result, their local consumers picked up jobs from the shared queue from time to time and processed them with the old logic — without my changes.
In the end I could have simply taken away everyone’s direct access to the database, but that would have hurt developer convenience a lot. So I moved the queues to Redis (which is not exposed to the outside) — and the problem was solved.
That’s team development for you…
