Not to trigger you, but remember during the pandemic when a certain person, unhappy about his poor record as a leader in the midst of an unprecedented health crisis, insisted that covid testing was problematic? “When you test,” he said, “you create cases.”
This fundamental confusion between actual causes and observed effects is a kind of magical thinking – or fetishism – and it’s as tempting in the software project as it is out in the real world.
I fell prey to it myself this week when one of our Selenium tests messed up a release by abruptly failing in a new and frustrating way after years of happily checking the operation of a crucial application flow.
I knew that that this particular issue could not be due to a recent commit, since nothing had changed that could possibly have affected this area of the code.
The logic in the system was sound. The code was behaving as it should given its input. Focussed as I was on getting the release out, I simply added an override that effectively said “if running in test mode, fire the modal anyway” and lo! all ran again. We were free to proceed.
But the issue – and, it has to be admitted, a smart team member – nagged at me as the day wore on. Eventually, we returned to investigate.
The assertion that failed assumed the presence of a switch in a database document which, if present and set to true, triggered the appearance of a modal. The flag duly arrived, but it was set to false. Therefore, no modal appeared and the test failed. If the code had not changed, we reasoned, obviously something must have altered the data.
Sure enough, after tracing all activities around our API, we discovered that a script had been run which accidentally reset a bunch of configuration fields in the database to default values. Not only that, but a misfiring modal was the least of our worries. The field resets were already causing all sorts of unexpected behaviours across our system.
So, it turns out, the test failure was righteous. Rigging it to pass with no further investigation was about as wise as muffling a fire alarm because you don’t like the sound of the bell. (The this-is-fine dog would sit well here if I had permission to use him).
Luckily, good sense prevailed and we did the (not entirely trivial) work to correct our configuration and the problems were resolved. But the experience reminded me of three aphorisms that are worth bearing in mind when coding and managing systems.
- Tests succeed when they fail.
- Never fix effects without first understanding causes.
- Beware fetishism (AKA cargo cults).
There are sutuations in which you might reasonably hobble a test. But, I’d argue that you should only do so when you know exactly why you’re doing it and when that reason is a good one. The fact that a failure inconveniences you is not an example of a good reason.
Photo by Patrick Tomasso on Unsplash
