Synthetic data versus validation data

Benchmark logic

Synthetic traces

Used to show the schema, prototype, scoring interface, and audit logic. They are designed examples, not evidence of validity.

Gold-standard traces

Real anonymized traces, independently coded by trained human coders, will be needed for validation.

Locked validation set

AI prompts should not be tuned on the final validation set. Otherwise, agreement estimates become inflated.

Circularity risk: if we write synthetic traces with known patterns and AI detects them, that only proves the pattern was written clearly. It does not prove the instrument measures real judgment.