Benchmark
Measured, not promised.
Kaviar’s detection performance, measured on a public test set the engine has never seen. Data, annotations and script are published: anyone can rerun the test on their own computer, offline, with no account and no need to trust us.
Recall
94.5%
Share of sensitive details actually protected. A missed detail is a detail sent to the AI.
Documents without any leak
67%
Share of documents where every sensitive detail was protected.
Precision
98.1%
Share of replacements that were justified. Too many wrong replacements degrade the AI’s answers.
By level
The three protection levels
Each level protects more categories. Figures cover the categories each level must protect.
| Recall | Documents without leak | Precision | |
|---|---|---|---|
| Standard | 95.3% | 84.3% | 98.7% |
| Enhanced | 94.5% | 67% | 98.1% |
| Strict | 94.4% | 61.6% | 97.7% |
By category
Category by category, none hidden
Measured at the Strict level, where every category is protected. The weakest ones top our work list.
| Protected from | Recall | Values | |
|---|---|---|---|
| People | Enhanced | 94% | 1159 |
| Companies | Enhanced | 86% | 400 |
| Emails | Standard | 99.6% | 279 |
| Phone numbers | Standard | 98% | 343 |
| Postal addresses | Standard | 76.1% | 310 |
| IBANs | Standard | 98.5% | 137 |
| Payment cards | Standard | 100% | 80 |
| National ID numbers | Standard | 100% | 144 |
| Passports | Standard | 99% | 496 |
| API keys | Standard | 100% | 80 |
| IP addresses | Standard | 100% | 80 |
| Dates | Enhanced | 98% | 537 |
| Cities | Strict | 94.5% | 640 |
| Amounts | Strict | 93% | 256 |
By language
Five languages
Enhanced level. This set only contains Latin scripts: non-Latin scripts will get a dedicated set.
| Recall | Values | |
|---|---|---|
| French | 95.1% | 801 |
| English | 97.1% | 837 |
| German | 95.5% | 804 |
| Spanish | 89.9% | 802 |
| Italian | 94.5% | 801 |
By document type
Easy documents, and hard cases
Enhanced level. PDF and OCR texts reproduce hard line breaks and character-recognition errors.
| Recall | Values | |
|---|---|---|
| Emails | 94.5% | 656 |
| Contracts | 94.5% | 640 |
| Medical reports | 93% | 560 |
| Tables (CSV) | 100% | 480 |
| Support tickets | 99.4% | 480 |
| Text extracted from PDF | 92.6% | 621 |
| Text from OCR | 89.3% | 608 |
Comparison
Against the open-source reference
Microsoft Presidio is the open-source reference in the field. We ran it in its default configuration (en_core_web_lg model) on the English documents, the only language it supports without configuration, and scored it with exactly the same criteria. Kaviar is measured at the Strict level. The comparison script is included.
| Kaviar | Presidio | |
|---|---|---|
| Recall | 97% | 68.6% |
| Documents without leak | 75.9% | 4.5% |
| Precision | 97% | 100% |
| People | 99.2% | 89.5% |
| Companies | 98.8% | Not covered |
| Emails | 98.4% | 100% |
| Phone numbers | 98.6% | 73.9% |
| Postal addresses | 78.8% | Not covered |
| IBANs | 100% | 74.1% |
| Payment cards | 100% | 100% |
| ID numbers | 100% | 58.3% |
| Passports | 100% | 100% |
| API keys | 100% | 0% |
| IP addresses | 100% | 100% |
| Dates | 98.1% | 87.9% |
| Cities | 94.5% | 87.5% |
| Amounts | 100% | Not covered |
“Not covered”: Presidio has no recognizer for this category in its default configuration. We only publish comparisons with tools anyone can test for free, using the published method.
Network proof
What leaves your computer, verifiable by you
The test runs offline.
The script blocks all network access before starting the engine, and counts every attempt. Published result: zero.
The engine is the app’s.
The kit contains the engine of the stated version, bundled from the app source, with the same local entity-recognition model.
The app’s traffic is observable.
Put a proxy such as mitmproxy between Kaviar and the internet: only the pseudonymised message goes to the AI. Your sensitive details appear in no request.
Limits
What this benchmark does not say
The documents are synthetic. They reproduce real formats, but real documents are messier.
Our current weak spots: addresses, especially in PDF and OCR text; companies with unusual names; towns whose name is also an ordinary word (“Sedan”, “Tarifa”, “Wells”) when nothing around them points to a place; a few needless replacements of column headers or of the word “passport” itself. They top our work list.
Indirect identifiers are not measured. A rare job and a small town are sometimes enough to recognise someone. For these cases, the Vault and manual selection remain necessary.
Text only. Images, and scans before character recognition, are not part of this test.
Answer usefulness is not measured yet in this version: checking that the AI’s answer stays right once the originals are restored will get its own section.
Pseudonymisation is not anonymisation in the GDPR sense. The mapping between originals and stand-ins exists, and it stays on your computer. Kaviar reduces the exposure of your data; it replaces neither your impact assessment nor your DPO.
Method
How the figures are obtained
A separate test set. The 560 documents were generated after the engine was frozen and have never been used to tune it. When a weakness found here gets fixed, the next version is measured on a new set: the published figure never comes from data used for fixes. Fixes are made on a separate internal set, also included in the kit.
A public annotation rule. Each sensitive detail is annotated with its category and the level that must protect it. A detail counts as protected if the engine covers at least 95 % of it, and if none of its identifying parts appear in the text sent to the AI: a first name left next to a replaced surname counts as a leak.
Valid identifiers. IBANs, cards, NIR, DNI, Steuer-ID and codice fiscale carry correct check digits, as in real documents.
A kept history. Every Kaviar version is measured and dated. Showing progress strikes us as more honest than a perfect, frozen figure.
History
Results by version
Each test set is measured with the previous version and then with the new one: progress is read on data the engine has never seen.
| Date | Test set | Recall | Documents without leak | Precision | |
|---|---|---|---|---|---|
| 1.8.2 | 2026-11-15 | holdout-v3.jsonl | 94.5% | 67% | 98.1% |
| 1.8.1 | 2026-11-15 | holdout-v3.jsonl | 89.1% | 39.1% | 98% |
| 1.8.1 | 2026-11-15 | holdout-v2.jsonl | 94.3% | 68.2% | 98.1% |
| 1.8.0 | 2026-11-15 | holdout-v2.jsonl | 91.7% | 56.8% | 98.1% |
| 1.8.0 | 2026-11-13 | holdout-v1.jsonl | 92.4% | 54.1% | 93.3% |
Replay
Four commands, on your computer
You need Node.js 20 or later. The entity-recognition model is downloaded from Hugging Face, at the same revision as in the app, with its checksum verified.
unzip kaviar-benchmark-1.8.2.zip && cd kaviar-benchmark
npm install
npm run setup # downloads the local model (SHA-256 checked)
npm run bench # results in results/latest.json, offlineRelated