A correct total can hide an incorrect AI answer
Review AI output against original source evidence with explicit omissions, false positives, source conflicts, and acceptance boundaries.
Where it came from: Portravo source-review work defined frozen acceptance gates across complete input coverage, schedules, counts, conflicts, scope, mapping, pricing, and independent source review. An independent scorer review reproduced synthetic fail-open paths including impossible citations, duplicate marks, and contradictions. This establishes historical method provenance, not accuracy of the newly authored Claude skill.
Package status: Draft package; structurally checked; not verified in Claude.
You give it
- Authorized original sources and revisions
- Unchanged raw result
- Task scope
- Prospective acceptance criteria
You get back
- Source-linked claim ledger
- Coverage and unsupported-claim findings
- Bounded acceptance verdict
- Unresolved evidence and next checks
An AI result arrives with a neat table, a confidence score, and the number you expected. The temptation is to approve it and move on. That becomes expensive when the table contains the right total for the wrong reasons.
In our construction takeoff review work, we had to evaluate quantities, specifications, missing scope, and conflicting plan evidence together. The useful method was a claim ledger tied to the original source. The same approach applies to invoice extraction, research summaries, and reports assembled from several documents.
The evaluate-ai-output skill packages that method for Claude. The historical review process supplies the foundation. The new Claude package remains an untested draft.
Read the full article on soxoa.com
Use evaluate-ai-output to review this extraction against the original document. Preserve the raw result. Build a claim ledger covering required fields, omissions, unsupported additions, and source conflicts. Explain the denominator for any accuracy metric. State which conclusions the available evidence cannot support.
/plugin marketplace add johnarndt/soxoa-claude-plugins /plugin install soxoa-ops@soxoa-pluginsDownload evaluate-ai-output.zipView on GitHub
View SKILL.md
--- name: evaluate-ai-output description: Review an AI result against its original sources and declared acceptance criteria, recording omissions, false positives, conflicts, and limits of the evidence. --- # Evaluate AI output Produce a source-grounded assessment of an existing AI result. This skill reviews supplied work; it does not run paid inference, change a benchmark, or edit the underlying product. ## Inputs Use the authorized original sources, the unchanged raw result, the intended task and scope, and any acceptance criteria fixed before the result was seen. Record source revisions and result identity with available filenames, versions, dates, or hashes. If a required input is missing, review the supported portion and name the missing input and its effect on the verdict. When criteria are absent, propose explicit criteria before scoring. Label them as a new prospective definition. Do not present a newly chosen definition as the historical benchmark. ## Review 1. Inventory the complete source set and requested scope. Separate sources actually inspected from unavailable or unreadable material. A selected page or trade cannot establish whole-document coverage. 2. Preserve the raw result. Put normalization and reviewer corrections in separate records. Converting equivalent units or labels is acceptable; inserting a missing answer is a correction and must remain visible. 3. Build a claim ledger: claim, source revision and location, expected basis, observed result, disposition, and short evidence note. Use supported, contradicted, omitted, unresolved, or outside scope. Keep source conflicts distinct from model mistakes. 4. Verify material claims against the source itself. A plausible citation or matching total is insufficient. Check that cited pages and regions exist and contain the asserted evidence. For counted instances, inspect uniqueness and ownership; repeated marks cannot count as different objects. Preserve distinctions such as openings versus components and quantities versus specifications. 5. Audit coverage in both directions: every required source fact should have a disposition, and every material output claim should have support. Identify unsupported additions, substituted items, omitted groups, duplicate counts, and false claims that evidence is absent. 6. Keep unknown values unknown. A missing price is not zero; an unresolved specification is not permission to substitute a familiar value. Report incomplete totals as incomplete. 7. Apply the declared criteria without averaging away a mandatory failure. Report any narrow subpass with its exact scope. If quantitative metrics are appropriate, define numerator, denominator, exclusions, and adjudication of ambiguous cases. Record automated delivery cost and latency only when measured records exist, distinguishing them from review labor. If a scorer is involved, inspect whether wrong citations, contradictory duplicate claims, or repeated instances can pass. Use separate synthetic copies for any authorized negative checks. Preserve earlier results and explain every criteria revision. Evaluator answers and annotated gold material must stay out of any subsequent inference inputs. ## Deliverable and stopping condition Return a bounded verdict, the claim ledger, the highest-impact omissions and false positives, unresolved source conflicts, criteria/version notes, and the next evidence needed. State who or what performed the review and any independence limitations. Stop when the declared scope has a supported disposition or a specific evidence gap prevents one. Do not manufacture a percentage from incomplete ground truth, call a development example held-out, or claim expert certification. Package status: this Claude skill draft has not been behaviorally tested in Claude. Its method was derived from historical source-review work; that provenance does not validate this package.