Markdown-Driven Browser Tests with Claude Code

Built a declarative browser testing system where test cases are plain markdown files and Claude Code executes them against a live site using Chrome DevTools MCP. No test framework, no Playwright config, no TypeScript — just text files with steps and an AI agent that reads them.

The test format is minimal:

---
name: Options Dashboard Smoke Test
tags: [smoke, navigation]
requires_auth: false
---

## Steps
1. Navigate to {{SITE_URL}}/
2. Wait for the page to load completely
3. Verify the main dashboard is visible
4. Click on "Wizard" in the navigation
   > Hint: nav link with text "Wizard", or aria-label="Wizard flow"
5. Verify the wizard panel opens

## Expected Results
- All steps completed without errors
- No JavaScript console errors detected

## Cleanup
- Navigate back to homepage

{{SITE_URL}} gets substituted at runtime so the same tests run against prod, a feature branch deploy, or localhost:3000.

There are two modes. Studio — interactive, human alongside the agent, used to create and debug tests. Runner — non-interactive, spawns a fresh subagent per test file, writes PASS/FAIL to tests/results/RESULTS.md and saves failure screenshots. Fresh subagent per test is important: prevents Chrome state leaking between tests and keeps context windows from bloating as results accumulate.

The runner prompt to each subagent is strict by design:

You are a test executor. Run the following test using chrome-devtools MCP tools.

RULES:
- Use take_snapshot to find elements by accessibility tree uid before clicking
- Use wait_for after navigation
- Use list_console_messages at the end to check for JS errors
- If a step has a "> Hint:" line, use it as fallback if text-based identification fails
- Replace {{SITE_URL}} with the actual URL
- Be STRICT: if a step says "click X" and X does not exist, FAIL the step. Do not improvise.

That last rule matters. Without it, agents start guessing — clicking nearby elements, scrolling around hoping to find something. That produces false PASSes.

The canvas drag problem

The site has a chart where users drag a price line endpoint. Canvas-rendered, no DOM handles. Standard click and drag MCP tools don't work here.

The agent solved this by reading the source code (the charting library is lightweight-charts), then discovering the library expects MouseEvent not PointerEvent. The approach that ended up working:

// 1. scan canvas pixels for the purple drag handle (#8b5cf6 = rgb(139,92,246))
const canvas = document.querySelectorAll('canvas')[0];
const ctx = canvas.getContext('2d');
const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);
const data = imageData.data;
let sumX = 0, sumY = 0, count = 0;
for (let i = 0; i < data.length; i += 4) {
  if (data[i] >= 120 && data[i] <= 160 &&   // r ~139
      data[i+1] >= 75 && data[i+1] <= 110 && // g ~92
      data[i+2] >= 230 && data[i+2] <= 255) { // b ~246
    const px = (i / 4) % canvas.width;
    const py = Math.floor((i / 4) / canvas.width);
    sumX += px; sumY += py; count++;
  }
}
const circleX = sumX / count + canvas.getBoundingClientRect().left;
const circleY = sumY / count + canvas.getBoundingClientRect().top;

// 2. hover first (circle turns orange as confirmation), then drag
const chart = document.querySelector('.tv-lightweight-charts');
const dispatch = (type, x, y) =>
  chart.dispatchEvent(new MouseEvent(type, { bubbles: true, clientX: x, clientY: y, buttons: 1 }));

dispatch('mousemove', circleX, circleY);
dispatch('mousedown', circleX, circleY);
for (let i = 0; i < 12; i++) dispatch('mousemove', circleX, circleY - (i * 5));
dispatch('mouseup', circleX, circleY - 60);

Once working, this gets documented as a > Hint: in the test step so future runners don't have to rediscover it. The knowledge accumulates in the test file itself.

Persistent knowledge base

All site-specific knowledge — navigation quirks, auth tricks, tricky selectors, known JS errors to ignore — lives in one place: tests/CLAUDE.md under a "Site Knowledge Base" section. Both studio and runner agents read it at session start. When the agent discovers something new (like the canvas technique above), it updates that file.

This is what makes the system progressively more reliable — each solved problem gets encoded so no future session re-spends time on it.

Gotchas

  • Sandbox isolation: Claude Code sandboxes each session to its working directory. Runner is in tests/, so results have to go into tests/results/ — writing to ../results/ fails silently.

  • Playwright MCP was tried and reverted: Switched to @playwright/mcp hoping for more reliable real mouse events. Canvas drag was less consistent in Playwright (required multiple retries, still unreliable). Also had unpredictable results file write failures. The JS event dispatch workaround documented in hints turned out more reproducible than Playwright's native drag.

  • No data-testid on the target site: All element identification falls back to visible text → aria-label → element ID. Tests are brittle to label changes. Keep hints pointing at stable identifiers.

  • Agent improvisation is the main failure mode: Without the "do not improvise" rule, the agent will find something close to what the step describes and click it, then report PASS. The test passes but didn't test what you thought it tested. Being explicit about strictness in the runner prompt fixes this.

  • Session debugging: Claude Code writes conversation logs as JSONL to ~/.claude/projects/<encoded-path>/. Useful for watching a stuck agent in real-time from another terminal: tail -f <session>.jsonl | python3 -c "import sys,json; [print(json.loads(l)) for l in sys.stdin]". How the canvas technique was discovered — watched the agent explore source code and try fallback approaches live.