The central scientific contribution
Validation design
The key question is whether AI-assisted coding can validly measure human judgment from decision traces. The answer may be yes, no, or only under human audit.
| Stage | Purpose |
|---|---|
| Trace collection | Capture initial decision, AI exposure, acceptance/rejection, verification, final decision, and confidence change. |
| Human reference coding | Train coders, establish construct-level agreement, and create a gold-standard benchmark. |
| AI-assisted coding | Run fixed prompts and documented model settings on the same traces. |
| Agreement analysis | Compare AI-human agreement by construct, not as one global score. |
| Failure-mode audit | Identify fluency bias, concision penalty, over-caution bias, verification hallucination, and model drift. |
Pre-committed success criteria
Thresholds are fixed before data collection, per construct, so the design is falsifiable rather than a slogan. Agreement is judged against an adjudicated human reference on a locked validation set that is never used to tune the AI prompt.
| Classification | Condition |
|---|---|
| AI-codeable | human–human κ ≥ 0.60 AND AI–human within-one ≥ 0.80 AND weighted κ ≥ 0.60 |
| Human-audit-required (hybrid) | meets human–human bar, but within-one 0.60–0.79 or weighted κ 0.40–0.59 |
| Not AI-codeable | weighted κ < 0.40 or within-one < 0.60 |
| Not reliably codeable by anyone | human–human κ < 0.60 — a boundary finding about the construct, not a pipeline failure |
We also pre-register a separability test for whether AI overreliance and cognitive outsourcing are empirically distinct or collapse into one factor. On synthetic pipeline-testing data they already correlate ≈ 0.84, so the question is registered rather than assumed.
A useful outcome can be a validated instrument, a hybrid AI-human coding protocol, or a boundary-setting map of where AI measurement fails. Because the thresholds are committed in advance, "we learn either way" is a stated decision rule with defined ways to win and to lose.