</>

Technology

Cucumber

Difficulty

Advanced

Interview Question

Your team reports that 40% of Cucumber tests are flaky — they pass sometimes and fail other times. How do you investigate and fix this?

Answer

Fixing 40% Flaky Cucumber Tests

Step 1: Identify and Categorize Flaky Tests

Bash
# Run suite 3 times back-to-back, compare failing scenarios
mvn test -Denv=staging -Dcucumber.filter.tags="@regression" -Drun=1
mvn test -Denv=staging -Dcucumber.filter.tags="@regression" -Drun=2
mvn test -Denv=staging -Dcucumber.filter.tags="@regression" -Drun=3

Tag consistently flaky scenarios:

Gherkin
@flaky @regression @P2
Scenario: Load more products on scroll
  ...

Step 2: Add Step-Level Screenshots for Diagnosis

Java
@AfterStep
public void screenshotAfterEachStep(Scenario scenario) {
    if (ctx.driver != null) {
        byte[] screenshot = ((TakesScreenshot) ctx.driver)
            .getScreenshotAs(OutputType.BYTES);
        scenario.attach(screenshot, "image/png",
            "After: " + scenario.getName());
    }
}

Now you can see the exact state at the step that fails.

Step 3: Root Cause Categories

Category A: Timing Issues (Most Common — 60%)
Java
// BAD — hardcoded sleep
Thread.sleep(3000);
driver.findElement(By.id("results")).click();

// GOOD — explicit wait
new WebDriverWait(driver, Duration.ofSeconds(15))
    .until(ExpectedConditions.elementToBeClickable(By.id("results")))
    .click();
Category B: Test Data Conflicts (25%)
Java
// BAD — all tests use same email
When the user registers with email "test@example.com"
// Second parallel run fails: email already exists

// GOOD — unique data per scenario
@Before
public void generateUniqueTestData() {
    ctx.testEmail = "qa" + System.currentTimeMillis() + "@test.com";
}
Category C: Parallel Execution State Leakage (10%)
Java
// BAD — static driver shared between threads
static WebDriver driver;

// GOOD — PicoContainer gives each scenario its own driver
public class ScenarioContext { public WebDriver driver; }
Category D: Environment Instability (5%)
  • Staging server slow/down
  • Test DB not reset between runs
  • Network latency spikes

Step 4: Systematic Fixes

Java
// Fix 1: Replace all Thread.sleep with wait utilities
// BasePage.java
protected void waitForElementVisible(By locator) {
    new WebDriverWait(driver, Duration.ofSeconds(15))
        .until(ExpectedConditions.visibilityOfElementLocated(locator));
}

// Fix 2: API-based test data setup (no UI dependency)
@Before("@registration")
public void createUniqueUser() {
    ctx.email = "qa" + UUID.randomUUID().toString().substring(0,8) + "@test.com";
}

// Fix 3: DB cleanup after each scenario
@After("@db-writes")
public void cleanDatabase() {
    dbUtils.executeQuery(
        "DELETE FROM users WHERE email LIKE 'qa%@test.com'");
}

// Fix 4: Retry flaky scenarios
@CucumberOptions(plugin = {"rerun:target/rerun.txt"})
// + RerunTestRunner reads failed scenarios

Step 5: Flaky Test Prevention Policy

CODE
1. Every PR must run @smoke 3 times — all must pass before merge
2. Scenario fails 2 of 5 runs → automatically tagged @flaky → moved to quarantine suite
3. Weekly flaky review session — fix or remove
4. No Thread.sleep() allowed (enforced by Checkstyle rule)
5. All test data generated uniquely (UUID/timestamp)
6. Parallel execution mandatory for all scenarios from day 1

Step 6: Metrics Tracking

CODE
Before fixes: 60% pass rate
After Week 1 (timing fixes): 82% pass rate
After Week 2 (data isolation): 94% pass rate
After Week 3 (state leakage): 98% pass rate

Target: < 2% flakiness = acceptable for 200-scenario suite

Follow AutomateQA

Related Topics