Answer
Fixing 40% Flaky Cucumber Tests
Step 1: Identify and Categorize Flaky Tests
Bash
# Run suite 3 times back-to-back, compare failing scenarios
mvn test -Denv=staging -Dcucumber.filter.tags="@regression" -Drun=1
mvn test -Denv=staging -Dcucumber.filter.tags="@regression" -Drun=2
mvn test -Denv=staging -Dcucumber.filter.tags="@regression" -Drun=3
Tag consistently flaky scenarios:
Gherkin
@flaky @regression @P2
Scenario: Load more products on scroll
...
Step 2: Add Step-Level Screenshots for Diagnosis
Java
@AfterStep
public void screenshotAfterEachStep(Scenario scenario) {
if (ctx.driver != null) {
byte[] screenshot = ((TakesScreenshot) ctx.driver)
.getScreenshotAs(OutputType.BYTES);
scenario.attach(screenshot, "image/png",
"After: " + scenario.getName());
}
}
Now you can see the exact state at the step that fails.
Step 3: Root Cause Categories
▸Category A: Timing Issues (Most Common — 60%)
Java
// BAD — hardcoded sleep
Thread.sleep(3000);
driver.findElement(By.id("results")).click();
// GOOD — explicit wait
new WebDriverWait(driver, Duration.ofSeconds(15))
.until(ExpectedConditions.elementToBeClickable(By.id("results")))
.click();
▸Category B: Test Data Conflicts (25%)
Java
// BAD — all tests use same email
When the user registers with email "test@example.com"
// Second parallel run fails: email already exists
// GOOD — unique data per scenario
@Before
public void generateUniqueTestData() {
ctx.testEmail = "qa" + System.currentTimeMillis() + "@test.com";
}
▸Category C: Parallel Execution State Leakage (10%)
Java
// BAD — static driver shared between threads
static WebDriver driver;
// GOOD — PicoContainer gives each scenario its own driver
public class ScenarioContext { public WebDriver driver; }
▸Category D: Environment Instability (5%)
- ✓Staging server slow/down
- ✓Test DB not reset between runs
- ✓Network latency spikes
Step 4: Systematic Fixes
Java
// Fix 1: Replace all Thread.sleep with wait utilities
// BasePage.java
protected void waitForElementVisible(By locator) {
new WebDriverWait(driver, Duration.ofSeconds(15))
.until(ExpectedConditions.visibilityOfElementLocated(locator));
}
// Fix 2: API-based test data setup (no UI dependency)
@Before("@registration")
public void createUniqueUser() {
ctx.email = "qa" + UUID.randomUUID().toString().substring(0,8) + "@test.com";
}
// Fix 3: DB cleanup after each scenario
@After("@db-writes")
public void cleanDatabase() {
dbUtils.executeQuery(
"DELETE FROM users WHERE email LIKE 'qa%@test.com'");
}
// Fix 4: Retry flaky scenarios
@CucumberOptions(plugin = {"rerun:target/rerun.txt"})
// + RerunTestRunner reads failed scenarios
Step 5: Flaky Test Prevention Policy
CODE
1. Every PR must run @smoke 3 times — all must pass before merge
2. Scenario fails 2 of 5 runs → automatically tagged @flaky → moved to quarantine suite
3. Weekly flaky review session — fix or remove
4. No Thread.sleep() allowed (enforced by Checkstyle rule)
5. All test data generated uniquely (UUID/timestamp)
6. Parallel execution mandatory for all scenarios from day 1
Step 6: Metrics Tracking
CODE
Before fixes: 60% pass rate
After Week 1 (timing fixes): 82% pass rate
After Week 2 (data isolation): 94% pass rate
After Week 3 (state leakage): 98% pass rate
Target: < 2% flakiness = acceptable for 200-scenario suite
