Evaluation report (synthetic corpus)
Every deterministic claim ships with a reproduction path: the annotated
corpus lives in the repository and these numbers regenerate with
pnpm eval. No real PHI was used, anywhere.
Scope statement: these numbers describe the published
synthetic corpus - English text written to exercise every detector rule. They do not measure
performance on real charts, other languages, or OCR noise.
95.9%micro recall
100%micro precision
100%detection rate
0false alarms
De-identification detector
9 synthetic clinical notes, 49 gold identifier spans across 18 categories.
| Result | Value |
|---|---|
| Micro-average recall | 95.9% |
| Micro-average precision | 100% |
| Hardest kept misses | bare names directly after headings; letter-prefixed reference codes with no structural signature |
| Low-confidence handling | flagged for human review inside every result |
Completeness scanning
| Metric | Value |
|---|---|
| Synthetic notes with known omitted sections | 5 |
| Genuinely absent required fields detected | 6 of 6 (100%) |
| False "not found" verdicts on present fields | 0 |
CI gates
- De-id micro recall ≥ 0.85; precision ≥ 0.80
- Structured identifiers (SSN, email, URL, IP, MRN, phone, fax): 100% recall on labeled forms
- Completeness detection rate ≥ 75%; false-alarm rate ≤ 50%
Reproduce it
# from a clone of https://github.com/Mr-Neutr0n/dsh-medseek pnpm install pnpm eval # rewrites eval/REPORT.md from eval/corpus pnpm verify # runs the gates above in tests/eval.spec.ts
Readability scoring uses published formulas (Flesch-Kincaid, SMOG) asserted
directly in unit tests. The full per-category table lives in
eval/REPORT.md in the repository.