Flaky Test Root Cause Analysis

A flaky test passes and fails without a related code change. Quarantining or rerunning it reduces noise in CI, but it does not explain why the assertion or timeout failed. Root cause analysis turns intermittent failures into a small set of cause classes so a QA engineer can choose a durable fix instead of another retry.

History and logs are the primary evidence. Patterns across builds (time of day, parallel shard, environment, recent dependency bumps) often matter more than a single stack trace. An LLM can read that evidence at scale, propose a cause label, and draft a fix recommendation. The engineer still owns the change: they validate the classification, edit the test or product code, and confirm stability in subsequent runs.

Tools such as BuildPulse help surface which tests are flaky and how often they fail. Classification sits downstream of that signal. If the pipeline cannot supply enough history or log context, the system should return empty output rather than guess.

Useful classification depends on structured inputs, not a free-form chat about "this test feels flaky."

Minimum inputs

• Test identity: suite, file, case name, and stable ID if your runner provides one
• Pass/fail history over a meaningful window (enough runs to show intermittency, not a single red build)
• Failure logs: assertion messages, timeouts, stack traces, and runner output for failing attempts
• Execution context: branch or commit range, CI job, shard or worker, browser or OS matrix when relevant
• Optional signals from flakiness detectors (for example BuildPulse severity or flake rate)

Empty output rules

Return no classification and no recommendation when:

• Pass/fail history is missing, too short to establish intermittency, or shows only consistent failures (those are usually regressions, not flakes)
• Failure logs are absent or truncated so that assertions, timeouts, and stack frames cannot be inspected
• The test identity cannot be resolved against the history store

Empty output is preferable to a confident wrong label. A QA engineer can still open the CI job manually; they should not inherit a fabricated cause class.

The model treats flakiness as a labeling problem over known cause families, then attaches a short, actionable recommendation.

Typical cause classes

• Timing and race conditions (waits, async completion, shared clocks)
• Order or isolation issues (shared state, unclean fixtures, cross-test pollution)
• Environment and infrastructure variance (resource contention, network, flaky third-party services)
• Selector or UI stability problems (dynamic DOM, animation, viewport)
• Data and fixture nondeterminism (seeded randomness, eventual consistency, shared test accounts)
• Parallelism and sharding conflicts (resource locks, port collisions)
• Known product nondeterminism that the test incorrectly treats as deterministic

Classification should cite which log lines or history patterns support the label. A useful output looks like: cause class, confidence as a qualitative note (for example "supported by repeated timeout on shard 3"), and a recommended fix shape (add an explicit wait for a condition, serialize fixture setup, stub an external call, harden a selector).

The model does not merge the fix. It does not quarantine the test or mute the CI check. Those remain pipeline and team decisions after an engineer reviews the recommendation.

1. Detect. Flakiness tooling or CI analytics flags intermittent tests (BuildPulse or equivalent).
2. Assemble evidence. Pull history and failure logs for the candidate test. Stop if evidence is incomplete.
3. Classify. The LLM proposes a cause class and a fix recommendation grounded in that evidence.
4. Review. The QA engineer accepts, rejects, or overrides the class. Ambiguous cases (race vs. isolation) may need a short experiment, such as running with a single worker.
5. Fix. The engineer implements the change in the test, fixture, or product code, or documents an accepted product nondeterminism with a narrower assertion.
6. Verify. Re-run under the same matrix and watch flake rate over following builds before removing any quarantine.

This loop keeps the model in the diagnosis lane. Speed comes from reading many logs and histories quickly; correctness still depends on someone who knows the suite and the system under test.

Limits

• Classification quality tracks log quality. Scrubbed or truncated CI logs hide races and shared-state clues.
• Cause taxonomies must match how your team remediates work. Labels that do not map to a fix pattern create busywork.
• High flake rates across an entire suite often point to infrastructure or shared fixtures, not dozens of independent test bugs. Prefer suite-level analysis when many tests fail together.

Failure modes to watch

• Treating a consistent regression as a flake because history was not long enough
• Overfitting to the last stack frame when history shows a different pattern
• Recommending retries as a "fix" (retries mask cause; they are not root-cause remediation)

Related practices

Pair cause classification with adversarial input generation when failures look like brittle edge handling rather than timing. Use autonomous E2E agents carefully: agent-driven flows can introduce new nondeterminism if waits and assertions are not explicit. Feed stable flake findings into risk-based prioritization so high-flake, high-impact areas get attention first.

The durable outcome is quality: fewer false reds, clearer ownership of intermittent failures, and fixes that address the labeled cause instead of burying it under retries.