717 user stories, four phases, one spreadsheet

The QA ritual I run on every product I own: inventory every user story from the code, test each one, fix, retest. As a Claude skill.

Where it came from: My four-phase QA ritual, run at 132, 175 and 464 stories, then 717 across the portfolio.

When AI writes most of your code, you lose track of what your app does. Not in a dramatic way. You just stop knowing. Did the password reset still work after last week's auth change? Does a canceled subscription actually lose access? You don't know, and "looks fine" isn't an answer.

So I stopped asking "is it broken?" and started asking "does each of these user stories pass?" That one has an answer.

Four phases, in order

1. Inventory. Read the code, not the marketing site, and write down every user story: routes, API handlers, emails, webhooks, admin screens, billing states, permission checks. Include the unhappy paths too: wrong password, expired link, declined card, someone else's ID pasted into the URL.

2. Test. Run every story against a real local or staging build and record PASS, PARTIAL, FAIL or BLOCKED with evidence. No fixing in this phase. Even an obvious one-line bug gets written down and left alone. Fixing as you go is how you end up with half a test pass and a mess of changes nobody reviewed.

3. Fix. Work through the failures in priority order, one commit each, every one referenced in the tracker. Anything that needs me personally, like a dashboard setting or a real payment, gets marked "deferred: owner" and lands on my to-do list.

4. Retest. Rerun everything that was fixed, plus everything that shares code with a fix.

All of it lives in one CSV in the repo: ID, area, priority, user story, expected behavior, code reference, how to test it, then one column per phase. Each phase also writes a dated ledger.

The receipts

I've run this on my own products at 132, 175 and 464 stories, then across the whole portfolio at 717. On that portfolio run, 659 passed on the first try, 28 failed and were fixed, 26 turned out to need no fix, and 4 were blocked on me or blocked by design. The 464-story pass on my document tools found 198 defects, and 176 got fixed in the same cycle.

The numbers aren't the point. The point is that afterward I could say exactly what was tested and what wasn't.

What went wrong, so you don't repeat it

"Deployed" in a note doesn't mean deployed. One product served a different branch for weeks while my notes said the fixes were live. Now the last step checks the live site and the commit it's actually running.
Two sessions, one repo, chaos. Run one audit per repo at a time.
Rotations fail quietly. A rotated database password that never reached the app took down hundreds of pages. Any credential change now triggers a retest of login, database and webhooks.
Cross-tenant checks go first. Put account A's IDs into account B's requests on every endpoint. That's the bug class worth the whole exercise.

The safety rules are in the skill too: the test suite has to pass before you start, nothing gets sent to real customers, payments run only in test mode, and paid API calls are capped.

Steal it

The skill is feature-audit in my John Lab plugin. Point it at a repo and say "audit this app."

/plugin marketplace add johnarndt/john-lab
/plugin install john-lab@john-lab

Install

/plugin marketplace add johnarndt/john-lab
/plugin install john-lab@john-lab

Download feature-audit.zip View on GitHub

View SKILL.md

---
name: feature-audit
description: "Use when a user wants to know whether their app actually works, end to end, before a launch or after a lot of AI-written changes. Triggers include 'audit my app', 'test every feature', 'is anything broken', 'QA this product', 'find the bugs before launch', 'feature audit', or 'make a test tracker'. Runs four phases with a CSV tracker: inventory every user story from the code, test each one, fix what failed, retest. Writes a dated ledger per phase and never fixes before the testing phase is done."
---

# Feature audit

"Is my app broken?" has no answer. "Does each of these 175 user stories pass?" does. This skill turns the first question into the second, in four phases, with a tracker that says exactly what was tested, what failed, what got fixed and what is still blocked.

## Tracker

`qa/FEATURE_TRACKER.csv` in the repo:

```
ID,Area,Priority,Feature,Role,User Story,Expected Behaviour,Code Ref,Test How,P1_Documented,P2_Test_Result,P3_Fix,P4_Retest,Notes
```

| Column | Values |
|---|---|
| Priority | P0 (money, auth, data loss), P1 (core flow), P2 (everything else) |
| P2_Test_Result | `PASS`, `PARTIAL`, `FAIL`, `BLOCKED-ENV` + a dated note |
| P3_Fix | `FIXED`, `NO-FIX-NEEDED`, `DEFERRED-ENV`, `DEFERRED-OWNER`, `WONTFIX` + commit |
| P4_Retest | `PASS`, `BLOCKED-ENV`, `N/A` |

Each phase also writes `qa/P<n>_<NAME>_<date>.md`: inventory, errors, fixes, retest.

## Phases

**P1 · Inventory.** Read the code, not the marketing. Routes, API handlers, background jobs, emails, webhooks, admin screens, settings, billing states, permission checks. One row per user story: "As a <role>, I can <action> so that <outcome>", with the expected behavior, the file it lives in, and how to test it. Include the unhappy paths: wrong password, expired link, card declined, another tenant's ID in the URL.

**P2 · Test.** Run each story through the real interface or API on a local or staging build and record the result with evidence. No code changes in this phase, even for obvious bugs. Write them down. A `PARTIAL` says what part failed.

**P3 · Fix.** Fix the `FAIL` and `PARTIAL` rows in priority order, one commit per fix or per tight group, each referenced in the tracker. Anything that needs the owner (a dashboard setting, a decision, a real payment) is `DEFERRED-OWNER` and goes on the owner's action list.

**P4 · Retest.** Rerun every fixed row and every row that shares code with a fix. Update the tracker. Summarize: total stories, first-pass passes, fixed, deferred, blocked.

## Safety rules

- Before P2, the existing test suite must pass. Record the baseline.
- No real sends to customers or prospects. No live payments; use the payment provider's test mode. No production feature-flag flips. Cap any paid third-party API calls and state the cap.
- One audit per repo at a time. Two sessions editing the same repo will collide.
- Commit fixes as you go. Uncommitted fixes get lost between sessions.

## Rules learned the hard way

- **Deployed is not proven.** After fixes ship, check the live site's behavior and its deployed commit. Notes that say "deployed" have been wrong for weeks at a time because the live site was serving a different branch.
- **Cross-tenant checks are P0.** Swap one account's IDs into another account's requests for every read and write endpoint. These are the bugs that matter most and the ones people skip.
- **After any credential rotation, retest login, database and webhooks.** A rotated secret that never reached the app fails silently.