A Vibecoder's Survival Guide to Software Testing

You shipped a feature in an afternoon. Tests passed. CI was green. At 2 AM your phone lights up: something you “barely touched” broke a flow you forgot existed. Welcome.

This is a tour of every kind of test that exists, why each one exists, and which one would have saved you that night. The vibecoder superpower is shipping at the speed of thought. The vibecoder failure mode is shipping bugs at the speed of thought. Tests are how you keep the first without the second.

Each section is the same shape: one sentence to define it, a tiny example, the failure mode it uniquely catches, a war story when there is a good one, and the tool to reach for. Read it top to bottom once. Then come back to whichever section matches the bug that just woke you up.

Core Levels

These are the test types that show up in every codebase that survives more than six months.

Unit test

A unit test calls one function or one class in isolation and asserts the output. Martin Fowler’s definition is the canonical one: “low-level, focusing on a small part of the software system, fast, and done by the programmer.” The “unit” is conventionally a class in OO code and a function in procedural or functional code.

def test_add_handles_negatives():
    assert add(-1, 1) == 0

Catches: logic bugs inside a single function that no human will notice in code review at 11 PM.

Tools: pytest, jest, vitest, JUnit, Go’s testing, RSpec.

Integration test

A test that runs multiple units together with real or fake dependencies, checking that the seams between them actually hold. “Integration” lives between “unit” and “end to end” and is the layer most teams underinvest in.

def test_signup_writes_user_and_emits_event(db, bus):
    signup_service.run(email="[email protected]")
    assert db.users.count() == 1
    assert bus.events == [("user.created", "[email protected]")]

Catches: wrong data flowing across the boundary between two modules. Unit tests mock the boundary; integration tests do not, so they catch when one side changes its contract and the other does not notice.

Tools: pytest with real DB containers (testcontainers), Spring Boot test slices, NestJS testing module.

Component test

A test that boots one deployable component end to end inside its own process, with external collaborators stubbed at the network boundary. It is broader than integration but narrower than full system.

# spin up the service with an in-memory DB and stub HTTP server
docker compose -f compose.test.yml up -d
curl -X POST localhost:8080/orders -d '{"sku":"X"}' | jq '.id'

Catches: wiring bugs that only surface when the real HTTP / gRPC / queue layer is in play; routing rules, middleware order, dependency-injection mistakes.

Tools: Testcontainers, WireMock, hoverfly, Pact (in stub mode), the language’s built-in HTTP test client.

End-to-end / system test

A test that exercises the whole stack across services, like a user would. Slowest, brittlest, and irreplaceable when you actually need it.

// playwright
await page.goto('/');
await page.getByRole('button', { name: 'Sign up' }).click();
await page.getByLabel('Email').fill('[email protected]');
await page.getByRole('button', { name: 'Continue' }).click();
await expect(page.getByText('Welcome')).toBeVisible();

Catches: the bug where every service passes its own tests and the whole system still does not work, because integration was tested at the seam and nothing tested the journey.

Industry note. Google’s “Software Engineering at Google” (Chapter 11) reports that about 80% of their tests are unit, 15% integration, 5% end-to-end. E2E is precious; do not try to scale it.

Tools: Playwright, Cypress, Selenium, BrowserStack.

Acceptance test

A test phrased in business language, executable, that says “the feature is done when this passes.” Often written in Gherkin (Given / When / Then).

Feature: Refunds
  Scenario: customer asks for a refund within 30 days
    Given an order placed 5 days ago
    When the customer requests a refund
    Then the refund is approved

Catches: mismatch between what was built and what was asked for. Unit tests prove the code does what the developer thinks; acceptance tests prove it does what the product owner thinks.

Tools: Cucumber, Behave, SpecFlow, pytest-bdd.

User-flow and UI

The part of testing that mostly did not exist twenty years ago and is now the most painful.

UI / browser test

A test that drives a real browser, clicking and typing as a user would. Sub-type of end-to-end.

await page.goto('/checkout');
await page.getByPlaceholder('Card').fill('4242 4242 4242 4242');
await expect(page.getByText('Paid')).toBeVisible();

Catches: the JavaScript that throws only when the third-party script loads in this order on Safari.

Tools: Playwright (current default), Cypress, Selenium. Pick Playwright unless you have a reason; it handles auto-wait, network mocking, and trace viewer out of the box.

Visual regression test

A test that takes a screenshot of a page or component and fails if the next run looks different.

test('login page snapshot', async ({ page }) => {
  await page.goto('/login');
  await expect(page).toHaveScreenshot('login.png');
});

Catches: the CSS change three components over that accidentally broke the login button’s centering. No assertion ever covers “the layout looks right.”

Tools: Percy, Chromatic, Loki, Playwright’s built-in screenshot diffing.

Accessibility test

Automated checks for WCAG violations: contrast, alt text, ARIA roles, keyboard reachability.

import { injectAxe, checkA11y } from 'axe-playwright';
await injectAxe(page);
await checkA11y(page);

Catches: the form field with no label, the button with contrast ratio 2.1, the modal that traps keyboard focus. Things you do not see and one in five of your users absolutely does.

Tools: axe-core, Pa11y, Lighthouse CI.

Snapshot test

A test that serializes a component’s rendered output to a file and fails if the next render differs.

expect(renderer.create(<Button label="ok" />).toJSON()).toMatchSnapshot();

Catches: unintended renders. Different from visual regression: snapshots are over the tree (DOM, JSON), not the pixels. Useful, also widely overused; if every snapshot diff becomes a blind --updateSnapshot, the test is doing nothing.

Tools: Jest, Vitest, React Testing Library.

Non-functional

Tests that check things other than “does the function return the right answer.”

Performance / load test

A test that hits the system with realistic traffic and measures latency, throughput, and error rate.

// k6
import http from 'k6/http';
export const options = { vus: 100, duration: '30s' };
export default () => http.get('https://staging.example.com/api/list');

Catches: the regression where a junior dev’s nested loop turned an O(n) endpoint into O(n²) and now p99 is 3 seconds.

Tools: k6, Locust, JMeter, Gatling, wrk for quick checks.

Stress test

A load test, but on purpose past the system’s known capacity, to see how it degrades.

Catches: the moment your service stops returning 500s and starts hanging forever instead. Graceful degradation is a property; you have to test for it.

Tools: same as load testing; just crank the VUs.

Soak / endurance test

A load test held at moderate load for hours or days.

Catches: memory leaks, file-descriptor leaks, slow connection-pool starvation. Bugs that take 6 hours to appear cannot be caught in a 30-second CI run.

Tools: k6 with duration: '12h', JMeter, custom load harnesses.

Smoke test

A tiny set of tests that answer “did the build even boot?” Run before anything else in CI.

curl -fsS http://localhost:8080/healthz || exit 1

Catches: the catastrophic regression that makes every other test irrelevant. Saves 40 minutes of running the full suite against a binary that segfaults at startup.

Tools: curl in CI, GitHub Actions matrix step, any healthcheck endpoint plus a script.

Regression test

The broad rerun of existing tests on every change, to confirm nothing that used to work has broken. Not really a separate type; more a discipline of “run them all on every PR.”

Catches: the bug where your fix for ticket A re-introduced the bug from ticket B.

Tools: your CI of choice (GitHub Actions, GitLab CI, Buildkite, Jenkins) plus selective-test-running tools like Bazel, Nx, or Turborepo for big monorepos.

Compatibility test

Run the system across browsers, OS versions, devices, locales.

Catches: “works on my MacBook.” Safari on iOS 15 has its own opinions about date parsing, and your Indonesian customer on a 2GB Android phone will find them all.

Tools: BrowserStack, Sauce Labs, LambdaTest, GitHub Actions matrix.

Security test (SAST + DAST + dependency scan)

Three things bundled together:

SAST (static): scan the source code for known-bad patterns.
DAST (dynamic): hit the running app with attacker-style requests.
Dependency scan: check package-lock.json / requirements.txt against CVE databases.

# GitHub Actions
- uses: github/codeql-action/analyze@v3
- uses: snyk/actions/node@master

Catches: Equifax got breached in 2017 via a known-and-patched Apache Struts CVE that was sitting in their lock file. SAST or a CVE scanner running in CI would have flagged it.

Tools: GitHub CodeQL, Snyk, Semgrep, Bandit, OWASP ZAP, Trivy for containers.

Penetration test

A human (or a careful automated red-team) actively trying to break in.

Catches: chained exploits that no automated tool would notice; “your SSO accepts a token from a sibling tenant” class of bug.

Tools: OWASP ZAP, Burp Suite, Metasploit, hire a firm.

Fuzz test

A test that hits a function or service with massive amounts of random or semi-random input until it crashes.

# Atheris fuzzing for Python
import atheris
def TestOneInput(data):
    parse_json(data)
atheris.Setup(sys.argv, TestOneInput)
atheris.Fuzz()

Catches: crashes from inputs no one would have written by hand. Heartbleed (2014) was independently rediscovered by Codenomicon engineers via fuzzing the SSL handshake; the bug had survived roughly two years of conventional unit and integration tests in OpenSSL.

Tools: AFL++, libFuzzer, Atheris (Python), Jazzer (JVM), cargo-fuzz (Rust), Hypothesis with strategies (Python).

Design-principled

Tests whose value comes from the way they are designed, not the layer they sit at.

Property-based test

You declare an invariant (a “property”) and the framework generates random inputs to try to break it.

from hypothesis import given, strategies as st

@given(st.lists(st.integers()))
def test_sort_is_idempotent(xs):
    assert sorted(sorted(xs)) == sorted(xs)

Catches: the edge case you didn’t think of. Example-based tests cover the cases you wrote down. Property-based tests cover the ones you didn’t.

Industry note. A 2025 study (Maaz et al., arXiv 2510.09907, preprint, not peer-reviewed) combined Claude Code with Hypothesis to generate property-based tests automatically. The system found real, confirmed bugs in numpy (PR #29609), requests (issue #6238), python-dateutil (issue #1437), Hugging Face tokenizers (PR #1853), and AWS Powertools (PR #7246). Worth knowing: the technique that catches the most subtle bugs is now itself being run by agents.

Tools: Hypothesis (Python), fast-check (JS/TS), QuickCheck (Haskell), ScalaCheck, jqwik (Java).

Mutation test

A meta-test. It changes (mutates) your production code slightly (== becomes !=, + becomes -) and then runs your test suite. If your tests still pass, your tests are weak.

mutmut run
# 87 / 1320 mutants survived  ← you have 87 places where tests are decorative

Catches: the false confidence of 100% coverage. Coverage says “this line ran during tests.” Mutation testing asks “would your tests notice if this line lied?” Often the answer is no.

Tools: mutmut (Python), Stryker (JS/TS, .NET), PIT (Java), cargo-mutants (Rust).

Contract test

Two services agree on a request/response shape. The consumer writes the contract; the provider’s CI verifies it still holds.

# pact contract from the consumer side
interactions:
  - description: "get user by id"
    request: { method: GET, path: /users/42 }
    response: { status: 200, body: { id: 42, name: "X" } }

Catches: breaking changes in the producer that no test in the producer’s own repo noticed, because the producer doesn’t know who is consuming it.

Tools: Pact, Spring Cloud Contract, schemathesis (OpenAPI-driven).

Approval / golden-master test

You run your code, save the output, and commit it as the expected answer. Future runs diff against it.

result = render_invoice(order)
verify(result)   # compares to invoice.approved.txt; updates on demand

Catches: the small unintended change in a sprawling output (a report, an HTML page, an LLM prompt template). Cheaper than writing 50 assertions about a 500-line file.

Tools: ApprovalTests (cross-language), Jest inline snapshots, syrupy (Python).

Characterization test

A test you write for code that already exists, to lock in its current behaviour before you change it. The starting move in Michael Feathers’ “Working Effectively with Legacy Code.”

# does anyone know what this does?
assert legacy.compute(7, 3) == 21  # apparently, this.

Catches: behavioural drift when refactoring untested legacy code. You are not asserting it is right; you are asserting it has not changed. After the refactor, every diff is suspicious.

Tools: any unit-test framework. The discipline is what matters, not the tool.

Delivery and Ops

Testing that happens at or near production.

Smoke test in CI/CD

The trivial set of checks that run after deploy to confirm the new build is alive. Same idea as the smoke test above, run after release.

curl -fsS https://api.example.com/healthz
curl -fsS https://api.example.com/v1/version | jq -e '.git_sha == env.GIT_SHA'

Catches: the deploy that “succeeded” but routed traffic to the wrong revision.

Tools: your CD platform (Argo, Spinnaker, GitHub Deployments) plus a few curls.

Canary release

You roll the new version to 1% of traffic, watch metrics, then 5%, 25%, 100%. The test is production.

Catches: the bug that no staging environment can find because staging does not have the variety of real traffic.

War story. CrowdStrike, July 19, 2024: a Channel File 291 update for the Windows Falcon Sensor crashed approximately 8.5 million Windows machines worldwide. The configuration update bypassed staged rollout, and customers had no mechanism to delay content-file installation. A canary tier with a kill switch would have contained it to a tiny fraction. The post-mortem (crowdstrike.com Channel File 291 RCA) confirms the absence of staged rollout as the root operational failure.

Tools: Argo Rollouts, Flagger, LaunchDarkly, AWS App Mesh, Linkerd / Istio traffic-splitting.

Shadow / mirroring test

Send a copy of real production traffic to the new version, throw away its responses, compare against the live version’s responses.

Catches: behavioural regressions under real traffic shapes, without exposing users to the new version’s mistakes.

Tools: Envoy mirroring, Istio traffic-mirroring, GoReplay, NGINX mirror.

Chaos engineering

You deliberately inject failures (kill a pod, drop a packet, slow a disk) and confirm the system survives.

War story. Netflix’s Chaos Monkey (chaosmonkey official site): “Chaos Monkey randomly terminates virtual machine instances and containers that run inside of your production environment.” The goal is to force engineers to build services that already assume failure, instead of pretending failure won’t happen.

Tools: Chaos Monkey, Gremlin, LitmusChaos, Chaos Mesh, AWS Fault Injection Simulator.

A/B test

Two versions ship to disjoint user populations; you statistically test which performs better on a chosen metric. Not a correctness test; a product-level test.

Catches: the assumption that the new UI is better. It usually is not. A/B testing is the place where engineering intuition goes to die, in a healthy way.

Tools: GrowthBook, Statsig, LaunchDarkly Experimentation, Optimizely.

Synthetic monitoring

A real test, running every few minutes, against production. Like end-to-end tests, but the SUT is the live system and the goal is detection, not gating.

# checkly check
curl -fsS https://example.com/login \
  && curl -fsS -X POST https://example.com/api/login -d ...

Catches: the outage at 3 AM that no user has hit yet but will in 20 minutes when European traffic wakes up.

Tools: Datadog Synthetics, Checkly, Pingdom, Uptime Kuma (self-hosted).

Observability as test

Logs, metrics, and traces are not testing in the classical sense. But in 2026 they often replace large swaths of staging tests, because real traffic + good observability + fast rollback is sometimes a better signal than 200 e2e tests.

Catches: the bug that only happens under the combination of real data, real concurrency, and real load. None of your tests will reproduce it.

Tools: OpenTelemetry, Honeycomb, Datadog, Grafana + Loki + Tempo + Prometheus.

Data and ML

If your code includes a data pipeline or a model, the testing toolbox is different.

Data validation test

Assertions about data shape, freshness, and value distributions, run before downstream tasks.

import great_expectations as gx
df.expect_column_values_to_not_be_null("user_id")
df.expect_column_values_to_be_between("age", 0, 130)

Catches: the upstream change where someone shipped age in months instead of years and your model now thinks every user is a baby.

Tools: Great Expectations, Pandera, dbt tests, Soda.

Model evaluation

A “test set” with metrics: accuracy, F1, AUC, BLEU, nDCG, whatever fits the task. Not pass/fail; threshold-gated.

metrics = evaluate(model, test_set)
assert metrics["accuracy"] >= 0.92
assert metrics["latency_p95_ms"] <= 150

Catches: the silent quality regression after a “small” retraining.

Tools: MLflow, Weights & Biases, scikit-learn’s metrics, deepchecks, CheckList (for NLP behavioural tests).

LLM / agent eval

The new layer. You curate a golden set of inputs with expected outputs (or expected properties), and you run them on every prompt or model change. LLM-as-judge is one common style; rubric-graded humans is another.

for case in golden_set:
    out = agent.run(case.input)
    assert judge_llm(out, case.rubric).pass_, case.id

Catches: the prompt change that improved one user flow and silently broke five others. Without an eval set, every model bump is a roll of the dice.

Tools: OpenAI Evals, LangSmith, Braintrust, Promptfoo, Inspect (AISI), Anthropic-style internal evals.

Strategy

You will not run all of the above. Choose where to invest.

The test pyramid

Mike Cohn’s 2009 metaphor: many small unit tests at the bottom, fewer integration tests in the middle, very few end-to-end tests at the top. The reason is cost: each layer up the pyramid is slower, flakier, and harder to localize.

Google operationalizes it at roughly 80% unit / 15% integration / 5% end-to-end by test count (Software Engineering at Google, Chapter 11). They also classify tests by size, not scope: a “small” test runs in one process with no I/O, a “medium” test runs across processes on localhost, a “large” test crosses machines. Size, not scope, is what determines speed and flake risk, and is the more honest axis to budget against.

The testing trophy

Kent C. Dodds’ counter-proposal: static analysis (TypeScript, ESLint, mypy) at the base, then unit, then integration (heaviest layer), then end-to-end at the top. The argument is that integration tests give the best ratio of confidence to cost for typical front-end apps. Pyramid or trophy, pick one consciously; do not let the shape happen by accident.

TDD and BDD

TDD (Kent Beck): write the test first, watch it fail, write the code, watch it pass, refactor. The benefit is not the tests; it is the design discipline of “if it is hard to test, the design is wrong.”

BDD: write the acceptance test first, in business language (Gherkin or equivalent), then build the code that makes it pass. Best when product, QA, and engineering need to literally read the same artifact.

You do not need to use TDD all the time. Use it when you are stuck on shape; the act of writing the test forces clarity.

Coverage is a signal, not a target

pytest --cov reports 90%. Looks great. Mutation testing will tell you that 30% of those covered lines are tested by assertions so weak they would not notice a sign flip. Coverage is a useful lower bound. It is a terrible goal; Goodhart’s law applies the moment you put it in a CI gate. If you must gate something, gate mutation score on critical packages and let coverage be advisory.

Flaky tests

Flaky = sometimes pass, sometimes fail, on no real code change. They are worse than no test; they teach the team to ignore CI.

Google’s Testing Blog (2016) documents how Google handles it: run a new test in a loop for a week, and if it shows patterns of unreliability, mark it flaky. Their “Beyoncé Rule” (“if you liked it then you shoulda put a test on it”) was paired with the realization that they spend nontrivial CI compute on flake retries, around 1.5% of test runs. Treat flakes the way you would treat memory leaks: triage hot, quarantine the rest, never normalize them.

Test data management

Bad tests share data, leak fixtures across runs, and depend on the order of execution. A test that requires tests/01_setup.py to have just run is not a test; it is a ritual. Use per-test fixtures, transactional databases, fakes you control, and seed data you can regenerate from a script.

The AI-in-the-loop reality

You are vibecoding. You generate code with an LLM. You probably let it generate tests too. Three things to know in 2026.

LLMs are great at generating more tests. Coverage goes up almost for free. Whether the new tests are good tests is a different question.
LLM test generators that filter out failing tests are dangerous. A 2024 study (Pizzorno et al., arXiv 2412.14137, preprint) found CoverAgent validated buggy behaviour in 59.6% of cases and CoverUp in 68.1%, because both tools discard tests that fail against the current code, baking in whatever bugs already exist. GitHub Copilot’s chat, which does not filter that way, generated bug-revealing tests in 67.6% of cases.
Agent + property-based testing is a force multiplier. The Maaz et al. 2025 work (arXiv 2510.09907, preprint) showed Claude Code + Hypothesis finding real bugs in numpy, requests, dateutil, HF tokenizers, and AWS Powertools. The agent’s leverage was not “write more example-based tests”; it was “ask the right questions about the invariants the code is supposed to preserve.”

Translation for the vibecoder: do not blindly accept generated unit tests, especially against code you also generated. Generate property-based tests instead. Run mutation testing on the package you care about most. Use AI to write the test scaffolding; use your brain to write the invariants.

A minimum survival kit

If you are starting from zero and want to stop getting paged:

Lint and type-check. Ruff, ESLint, mypy, TypeScript strict mode. Free; catches the dumb stuff.
Unit tests on the gnarly logic. Skip the pure-glue functions; test the algorithm, the parser, the calculator.
One smoke test per critical user flow. Playwright. One per: sign up, log in, the main “happy path” of your product, the checkout.
A canary on deploys. Argo Rollouts or LaunchDarkly. Roll to 5% first. Watch error rate. Wait 10 minutes.
Synthetic monitoring on the same flows you smoke-tested. Checkly. Every 5 minutes. PagerDuty if it fails twice.
Mutation testing on the one package whose bug would be most expensive. Run it weekly, not on every PR.

This is more testing than most one-person products have. It is also less than what your future self will wish you had set up the night you got paged.