Flaky Tests

REQUIREMENTS

• Test Insights must be configured

Flaky tests are tests that produce different results (pass or fail) when run multiple times with the same code. They erode trust in your test suite and waste developer time investigating false failures. Tuist automatically detects flaky tests and helps you track them over time.

How flaky detection works

Tuist detects flaky tests in two ways:

Test retries

When you run tests with Xcode's retry functionality (using -retry-tests-on-failure or -test-iterations), Tuist analyzes the results of each attempt. If a test fails on some attempts but passes on others, it's marked as flaky.

For example, if a test fails on the first attempt but passes on the retry, Tuist records this as a flaky test.

tuist xcodebuild test \
  -scheme MyScheme \
  -retry-tests-on-failure \
  -test-iterations 3

Cross-run detection

Even without test retries, Tuist can detect flaky tests by comparing results across different CI runs on the same commit. If a test passes in one CI run but fails in another run for the same commit, both runs are marked as flaky.

This is particularly useful for catching flaky tests that don't fail consistently enough to be caught by retries, but still cause intermittent CI failures.

Managing flaky tests

Automatic clearing

Tuist automatically clears the flaky flag from tests that haven't been flaky for 14 days. This ensures that tests that have been fixed don't remain marked as flaky indefinitely.

Manual management

You can also manually mark or unmark tests as flaky from the test case detail page. This is useful when:

• You want to acknowledge a known flaky test while working on a fix
• A test was incorrectly flagged due to infrastructure issues