Stub the provider, not the state machine
To make Whet's LLM-backed E2E suite deterministic and free, the stub sits only at the provider I/O seam while the orchestration, persistence, and UI around it still run. Cut at the narrowest seam so the test verifies real wiring minus cost and nondeterminism.
Whet’s end-to-end suite drove a real LLM-backed pipeline — admin seed,
settings, sources, the picker and drawer — which made it both
nondeterministic and expensive to run. The fix was to place a stub at
the narrowest possible seam: a WHET_E2E_LLM_STUB flag intercepts the
provider I/O call and returns a fixed result, while the machinery around
it keeps running. The web path already carried this seam; the backend
generate() path got the same treatment on Jun 7 (8b78d04). At that
point the suite reached 15/15 green spending zero tokens.
The seam is the point
A test earns its keep by exercising real behavior. Stub too high — mock the whole job, short-circuit the whole endpoint — and you skip the orchestration, persistence, and UI transitions the test exists to verify. What’s left green is a test of the mock. Stubbing at the provider call replaces only what is nondeterministic and paid: the model’s output, and the per-item curation score that derives from it. Everything around that seam still runs — the curation job executes, the rows get written, the drawer renders them. The seam is chosen so the model call below it is fake and the machinery above it is real.
Off by default is not optional
The flag defaults off, so production and any run without it opt into the stub explicitly. A test-only shortcut that alters the default path is a liability: it can mask a real regression the day someone forgets it’s active. Here the provider path is the norm and the stub is the exception you have to ask for, which keeps the escape hatch from becoming the front door.
Determinism from state, not time
The suite’s waits anchor to observable UI conditions
(e2e/fixtures/waits.ts) rather than fixed sleeps. Once the paid,
nondeterministic call is gone, the remaining flakiness lives in timing
assumptions. Waiting on state — the drawer rendered, the item present —
instead of on a clock removes it. A stubbed provider makes the suite
free; state-anchored waits make it stable. The two together are what
turn an LLM-dependent flow into something CI can run on every push.