Your Test Harness Is Answering Your Questions: Stubs That Outlive the Test

While debugging a bot-challenge widget that would not render in production, we drove the live site
with an instrumented browser and found something striking: window.turnstile already existed on a
page that had not loaded the vendor script yet. Calling render() returned the string "stub".
The obvious conclusion was that something else on the page — the CDN, a tag manager, a legacy bundle — was defining a conflicting object. We shipped a loader designed to work around it, and told the customer that was the cause.
It was not. The stub was ours. An earlier test in the same browser context had installed it, and it was still there.
The mechanism
Playwright's page.addInitScript() registers a script that runs before every document in that
browser context, and it stays registered for the life of the context — across every navigation, and
across every new page opened from it.
await page.addInitScript(() => {
window.turnstile = { render: () => "stub", reset: () => {} };
});
await page.goto("/signup"); // stub installed — intended
await page.goto("/pricing"); // stub installed — not intended, and not obvious
await page.goto("/"); // still installed
The same applies to anything else attached to the context rather than the page: routes registered
with page.route(), injected cookies, granted permissions, storage state, and emulated geolocation.
Selenium, Puppeteer and Cypress each have their own equivalents.
The result is a debugging session where your test double is answering questions you believe you are asking production.
Why it was so convincing
Every property of a real third-party conflict was present:
- an object we did not create, on a page we did not expect it on
- behaviour differing between our local environment and production
- a plausible culprit — the site genuinely does sit behind a CDN that injects scripts
- a reproducible observation, because the stub was reliably there every time
Reproducibility felt like proof. It was proof of a persistent stub, which looks identical from inside the session.
The habits that prevent it
Start diagnosis in a fresh context
For any run whose purpose is to observe production rather than to test code, open a new context:
const ctx = await browser.newContext(); // no init scripts, no routes, no storage
const p = await ctx.newPage();
await p.goto(url, { waitUntil: "load" });
// ...observe
await ctx.close();
This is one extra line and it removes an entire category of false conclusion. We now do it by default for every live-site investigation.
Make stubs identify themselves
A stub that announces itself cannot be mistaken for the real thing:
window.turnstile = {
__isTestStub: true,
render: () => "stub",
};
Then a diagnostic can assert the absence of its own instrumentation:
const clean = await p.evaluate(() => !window.turnstile?.__isTestStub);
if (!clean) throw new Error("test stub present — this context is not clean");
Had that check existed, the wrong diagnosis would have lasted about four seconds.
Prefer page-scoped setup where the tool allows it
Put the stub as close to the test that needs it as possible, and remove it afterwards. A stub whose lifetime matches the assertion that needs it cannot leak into an unrelated observation.
Treat "unexpected global exists" as suspect-yourself-first
When instrumentation reports something surprising about a page, the first hypothesis should be your own harness, not the vendor's code. Confirm in a clean context before writing the hypothesis into a commit message — or a message to a customer.
What it cost, and the correction
We shipped a loader built around a conflict that did not exist, and we told the customer that
Cloudflare was injecting its own object. Both had to be corrected. The eventual real cause was
mundane and well documented: render() was being called before the vendor API finished initialising,
which is a silent no-op, and the fix is the documented onload handshake.
The wrong diagnosis was not harmless. It burned a deploy cycle, it put an inaccurate explanation in front of someone who was relying on it, and the commit message recording it is still in the history. We corrected the record in the next commit, naming the earlier one explicitly, because a repository that quietly revises its own explanations is worse than one that shows its working.
The generalisation
Any tool that makes the system under test more observable also makes it different. Debuggers change timing. Proxies change headers. Coverage instrumentation changes performance. Stubs change behaviour.
Most of the time that is an acceptable trade. It stops being acceptable the moment you use the instrumented system as evidence about the uninstrumented one — which is exactly what you are doing when you reach for the tool to diagnose a production problem.
Clean context, every time, before drawing a conclusion you intend to act on.


