A CI pipeline that runs tests but doesn't actually block anything on failure isn't a gate — it's a suggestion. What makes a test suite a real quality gate is which tests are required, how fast they report back, and what happens when one is flaky instead of genuinely broken.
4 min read
Running tests in CI is necessary but not sufficient — the test results have to be wired to something that prevents a bad change from merging or deploying. On GitHub, this is a required status check: a branch protection rule that makes the CI job's pass/fail state a hard requirement before the merge button is even clickable, not just a colored badge someone can choose to ignore.
# Repository settings → Branches → branch protection rule for `main`
# (not a file — configured in the GitHub UI or via the API)
required_status_checks:
strict: true # branch must be up to date with main before merging
contexts:
- "build-and-test" # the CI job name from the earlier lesson's workflowWithout this configured, a red CI check is a strong hint, not an actual blocker — someone in a hurry can merge past it, and eventually someone will.
Different test types have wildly different costs — a unit test runs in milliseconds, an end-to-end test that spins up a browser and a real backend can take tens of seconds. Running the slow, expensive tests first wastes time on every single run, including the runs that were going to fail a trivial unit test anyway. The pipeline should mirror the test pyramid shape: run the fast, cheap layer first, and only pay for the slow, expensive layer once the cheap layer has already passed.
A developer who breaks a unit test finds out in the time it takes to make coffee, not 15 minutes later after an E2E suite finishes running against a change that was never going to pass anyway.
Lint and type-checking don't depend on each other, and often don't need to wait for the full test suite either — running independent jobs concurrently instead of as one long sequential script cuts wall-clock time without cutting anything from what's actually checked:
jobs:
lint:
runs-on: ubuntu-latest
steps: [...]
typecheck:
runs-on: ubuntu-latest
steps: [...]
unit-tests:
runs-on: ubuntu-latest
steps: [...]
# This job waits for the others to finish before running,
# since E2E is expensive and only worth running on code
# that's already passed the cheaper checks
e2e-tests:
needs: [lint, typecheck, unit-tests]
runs-on: ubuntu-latest
steps: [...]needs: expresses the real dependency — E2E genuinely should wait for the cheap layer — while lint, type-checking, and unit tests run at the same time since none of them depend on each other's outcome.
A test that fails intermittently — not because the code is actually broken, but because of a timing race, a shared-state leak between test runs, or a network call to something unreliable — is worse than a test that's simply missing. A missing test provides no false signal. A flaky test actively teaches the team the wrong lesson: "CI failures are often nothing, just re-run it," which is exactly the habit that lets a genuine failure get waved through as "probably just flaky again."
The practical response, in order of preference:
Not every check needs to be a hard merge blocker. A reasonable split:
Getting this split wrong in either direction has a cost: too little required, and broken code merges regularly; too much required (especially anything flaky), and the team learns to route around the gate entirely — which is the same failure mode as having no gate at all, just with extra ceremony.
Check your understanding
A quick comprehension check — not tracked, not graded, just for you.
1. What actually makes a CI check function as a real gate rather than a suggestion?
2. Why should a pipeline run unit tests before end-to-end tests rather than running everything at once?
3. Why is a flaky test considered worse than a missing test?
4. What is the recommended way to handle a flaky test that can't be immediately fixed?
CI/CD & Deployment Pipelines