When the CDN started classifying our own test suite as a bot

TL;DR

Overnight, the nightly Cypress suite went from fully green to 62 failing specs with no corresponding change in the application code or the tests. The cause was upstream of both: the CDN's bot-scoring had started flagging the Cypress browser, and a middleware in the application treats bot traffic as a signal to pin the visitor's country to the default market for SEO consistency. Every journey rendered as US, whichever market it had selected.

The more dangerous half of the problem was not the 62 red specs - it was the other 154. They were US-based journeys, so the forced country happened to match what they asserted and they reported green. Every one of them was failing silently: the suite had stopped testing what it claimed to test, and nothing on the dashboard said so.

The fix was a targeted server-side exemption for the test user agent, applied in two places, and deliberately not a change to the shared bot-detection helper.

The symptom

Between one nightly run and the next, the suite went from 0 failures to 62 out of 216. Two things made this look unlike a normal regression:

No triggering change. No application deploy, no test change, and no CI configuration change correlated with the step from green to red.

The failures clustered by country, not by feature. Every failing spec had a non-US background (GB, FR). Every US-background spec passed. Features that failed spanned unrelated areas - feeds, the country selector, checkout - with no shared code path in the test layer.

Typical failure mode: a GB-only category feed returned zero results, so the grid element the test waited on never appeared. The test failed on a timeout, which reads like flakiness and buries the real cause.

First action: stop the bleeding

Before any debugging, we paused the nightly schedule.

Two reasons, and we'd argue both are the default for any suite in this state:

A suite that fails every night stops being read. Once a red dashboard is the normal state, the team loses the one signal it exists to provide. The cost of a paused suite is honest; the cost of an ignored suite is invisible.

workflow_dispatch was kept. Pausing the cron did not remove the ability to run the suite on demand - which is exactly what the investigation and the later verification runs needed.

Investigation

Working back from the failures:

The failing assertions were all downstream of the rendered country. Not one test asserted on country directly; they asserted on catalogue content that only exists in specific markets. That pointed at the country resolution rather than at any individual feature.

The middleware resolves country by precedence. Reading the set_country middleware, the order was roughly: an explicit country URL parameter, then a bot check that pins to the domain's default market (US), then the authenticated user's stored country, then the session country, then the country cookie, and finally the CDN's geo header.

The bot branch sat above the session, the cookie and the geo header. So once a request was classified as a bot, everything the test had done to select a country was discarded on the next page load.

Proving the mechanism, with the CDN out of the path. This was the step that turned a theory into a fact. We sent the same request with the same GB country cookie directly to the origin load balancer, varying only the bot-score header. With no bot-score header it rendered GB; with a bot-score of 1 it rendered US; with a bot-score of 99 it rendered GB again.

Caching was ruled out. We also checked the other two branches of the bot helper to be sure this was the only one that could fire: the user-agent parser classified the Cypress browser as an ordinary unknown browser, not a bot, and the hardcoded bot IP ranges covered only search-engine crawlers. A CDN bot score of 1 was the sole trigger.

Why the score changed is still the open question. An existing CDN rule was supposed to exempt the test traffic - matching the test user agent and a runner egress country of GB or IE. That rule stops the traffic being blocked or challenged, but it does not stop the bot-score header reaching the origin - those are separate concerns. We reproduced the forced-US behaviour from a VPN'd laptop whose egress matched the rule exactly, which confirmed the exemption was never going to help here.

Why CI never caught it. Pull-request runs point at an internal load balancer, bypassing the CDN entirely. Only the nightly run went through the public edge. The suite that exercised the real production path was the only one that could see the bug - and it ran once a day, at midnight, to an audience of nobody.

The dead end worth documenting

Before landing a server-side change we spent four iterations on a test-side workaround: inject the country parameter - the one input evaluated above the bot check - into every request. It does not work, and the reasons generalise:

Setting the country parameter on the initial visit is not enough. Journeys continue via in-app clicks. Opening a product from a feed is a background request to an overlay endpoint, not a navigation, so the parameter never reaches it.

Putting it in the visited URL breaks assertions. A step asserting the query string equals exactly a specific value fails the moment you append anything.

A blanket same-origin intercept crashes the browser. It also broke the consent banner, and it matched Cypress's own runner assets served from the app origin.

Every server-side layer needs it separately - documents, the catalogue API, the checkout API - and the CDN's verdict varies run to run. Two consecutive runs with near-identical code failed at different stages.

The conclusion we took to the team: you cannot stabilise a suite by patching the test layer around a non-deterministic upstream input. The fix had to be where the decision was made.

The fix, part one: exempt the test agent

We took the evidence to a backend engineer on the owning team and agreed the shape of the fix together. The bot branch should not fire for our own test traffic, so we added a check that skips it when the request comes from the Cypress test agent.

Two design decisions are worth calling out, because both were deliberate:

It is scoped to a single call site, not to the shared helper. The obvious change would have been to teach the bot-detection helper about the test agent. There were 25 call sites, each with its own consequences. Changing the helper would have altered all of them at once, invisibly. Adding a separate predicate and applying it at exactly the sites that need it keeps the blast radius equal to the intent.

The abuse case was assessed and accepted. The check is only a user-agent string, so it is trivially spoofable. What does spoofing it get you? The ability to change your own country - which is to say, the ability to behave like a normal logged-out user. The bot branch exists to give crawlers a consistent market for SEO, not as a security control. Impersonating the test agent is equivalent to being scored anything other than 1. The trade was documented in the commit rather than left implicit.

A regression test for the bypass landed alongside it, and a redundant branch in the same block was removed while we were in there.

Verify before declaring victory

Staging has no CDN in front of it, so the fix could not be validated pre-merge. After it merged we triggered a manual dry run of the full nightly suite against production.

It was a partial success, and that was the point of running it. The silent failures were gone - the country selector now genuinely exercised GB and FR instead of quietly agreeing with itself - and most of the real failures cleared. But checkout specs still failed.

Same root cause, different code path. The checkout dispatcher redirects bots to the homepage, because bots probe the checkout page directly, never have a checkout session, and would otherwise render an empty, erroring page. Our test agent was still a bot as far as that check was concerned.

The fix, part two: the same exemption at the second site

This is exactly the case the part-one design decision anticipated: a second, independent site needing the same exemption, added deliberately rather than inherited silently from a helper change. The predicate was extracted into a shared utility at this point so the two sites agree.

Hardening, once the signal was trustworthy

With the suite genuinely passing, the failures that remained were real test-quality problems that the country bug had been masking. We cleared them out while the context was fresh:

Fixed a login that was not persisting across an order-history journey.

Fixed native checkout, editorial and category-feed failures, and removed the markers that had been papering over collection tests.

Removed hardcoded store-name assertions from generic category feed tests - they asserted on a specific brand that happened to be first in the results, so any catalogue change broke them. They now assert against the store the test actually clicked, derived from the page itself.

Upgraded Cypress to the current release.

Then the nightly schedule was restored.

Outcome

62 of the 62 failing specs restored.

154 silent failures eliminated. The specs that never went red were running against the wrong market and reporting green anyway. That was the more dangerous half of the failure and it produced no visible symptom.

Root cause fixed at the decision point, in two places, with a documented and bounded security trade-off - not worked around in the test layer.

Suite unpaused and green across all 216 specs, with the accumulated real flakiness cleared out.

What we'd take to the next incident

A red suite and a green suite can share the same bug. The 62 failures told us where to look; the 154 passes told us how bad it was. If a country, locale or feature-flag input is silently forced to one value, every test that happens to expect that value keeps reporting green while testing nothing. Ask what the passing tests are actually proving.

Test the environment the suite runs in, not the one it's convenient to run in. PR runs hit an internal load balancer and were structurally incapable of catching this. That is a reasonable trade for speed, but it means the nightly run is the only signal for a whole class of edge behaviour - and it deserves to be watched accordingly, not just when it goes red.

Pause loudly, keep the manual trigger. A permanently red dashboard is worse than an honestly paused one, and every step after this depended on being able to run the suite on demand.

Prove the mechanism in isolation before proposing a fix. Sending the same request to origin with only the bot-score header varying turned "we think the CDN is doing this" into a three-row table. That is what made the conversation with the backend engineer a five-minute agreement on the fix rather than a debate about the diagnosis.

A dry run is part of the fix, not a formality. The first fix was correct and incomplete. Only running the real suite against the real edge surfaced the second code path.

Match the blast radius to the intent. The tempting one-line change to a helper with 25 call sites would have shipped 24 unintended behaviour changes. Two explicit call-site changes are more code and less risk.

Tooling

Investigation, log analysis and the isolation testing against origin were carried out with Claude Code as an assistant; the diagnosis, the decision to reject the test-side workaround, and the fix design were ours.

‍