Benchmark

Measured, not promised.

Kaviar’s detection performance, measured on a public test set the engine has never seen. Data, annotations and script are published: anyone can rerun the test on their own computer, offline, with no account and no need to trust us.

Recall

94.5%

Share of sensitive details actually protected. A missed detail is a detail sent to the AI.

Documents without any leak

67%

Share of documents where every sensitive detail was protected.

Precision

98.1%

Share of replacements that were justified. Too many wrong replacements degrade the AI’s answers.

Kaviar 1.8.2 · Enhanced level, the default · 560 documents · 4,045 annotated details · 2026-11-15 · 0 network connection attempts

By level

The three protection levels

Each level protects more categories. Figures cover the categories each level must protect.

RecallDocuments without leakPrecision
Standard95.3%84.3%98.7%
Enhanced94.5%67%98.1%
Strict94.4%61.6%97.7%

By category

Category by category, none hidden

Measured at the Strict level, where every category is protected. The weakest ones top our work list.

Protected fromRecallValues
PeopleEnhanced94%1159
CompaniesEnhanced86%400
EmailsStandard99.6%279
Phone numbersStandard98%343
Postal addressesStandard76.1%310
IBANsStandard98.5%137
Payment cardsStandard100%80
National ID numbersStandard100%144
PassportsStandard99%496
API keysStandard100%80
IP addressesStandard100%80
DatesEnhanced98%537
CitiesStrict94.5%640
AmountsStrict93%256

By language

Five languages

Enhanced level. This set only contains Latin scripts: non-Latin scripts will get a dedicated set.

RecallValues
French95.1%801
English97.1%837
German95.5%804
Spanish89.9%802
Italian94.5%801

By document type

Easy documents, and hard cases

Enhanced level. PDF and OCR texts reproduce hard line breaks and character-recognition errors.

RecallValues
Emails94.5%656
Contracts94.5%640
Medical reports93%560
Tables (CSV)100%480
Support tickets99.4%480
Text extracted from PDF92.6%621
Text from OCR89.3%608

Comparison

Against the open-source reference

Microsoft Presidio is the open-source reference in the field. We ran it in its default configuration (en_core_web_lg model) on the English documents, the only language it supports without configuration, and scored it with exactly the same criteria. Kaviar is measured at the Strict level. The comparison script is included.

KaviarPresidio
Recall97%68.6%
Documents without leak75.9%4.5%
Precision97%100%
People99.2%89.5%
Companies98.8%Not covered
Emails98.4%100%
Phone numbers98.6%73.9%
Postal addresses78.8%Not covered
IBANs100%74.1%
Payment cards100%100%
ID numbers100%58.3%
Passports100%100%
API keys100%0%
IP addresses100%100%
Dates98.1%87.9%
Cities94.5%87.5%
Amounts100%Not covered

“Not covered”: Presidio has no recognizer for this category in its default configuration. We only publish comparisons with tools anyone can test for free, using the published method.

Network proof

What leaves your computer, verifiable by you

The test runs offline.

The script blocks all network access before starting the engine, and counts every attempt. Published result: zero.

The engine is the app’s.

The kit contains the engine of the stated version, bundled from the app source, with the same local entity-recognition model.

The app’s traffic is observable.

Put a proxy such as mitmproxy between Kaviar and the internet: only the pseudonymised message goes to the AI. Your sensitive details appear in no request.

Limits

What this benchmark does not say

The documents are synthetic. They reproduce real formats, but real documents are messier.

Our current weak spots: addresses, especially in PDF and OCR text; companies with unusual names; towns whose name is also an ordinary word (“Sedan”, “Tarifa”, “Wells”) when nothing around them points to a place; a few needless replacements of column headers or of the word “passport” itself. They top our work list.

Indirect identifiers are not measured. A rare job and a small town are sometimes enough to recognise someone. For these cases, the Vault and manual selection remain necessary.

Text only. Images, and scans before character recognition, are not part of this test.

Answer usefulness is not measured yet in this version: checking that the AI’s answer stays right once the originals are restored will get its own section.

Pseudonymisation is not anonymisation in the GDPR sense. The mapping between originals and stand-ins exists, and it stays on your computer. Kaviar reduces the exposure of your data; it replaces neither your impact assessment nor your DPO.

Method

How the figures are obtained

A separate test set. The 560 documents were generated after the engine was frozen and have never been used to tune it. When a weakness found here gets fixed, the next version is measured on a new set: the published figure never comes from data used for fixes. Fixes are made on a separate internal set, also included in the kit.

A public annotation rule. Each sensitive detail is annotated with its category and the level that must protect it. A detail counts as protected if the engine covers at least 95 % of it, and if none of its identifying parts appear in the text sent to the AI: a first name left next to a replaced surname counts as a leak.

Valid identifiers. IBANs, cards, NIR, DNI, Steuer-ID and codice fiscale carry correct check digits, as in real documents.

A kept history. Every Kaviar version is measured and dated. Showing progress strikes us as more honest than a perfect, frozen figure.

History

Results by version

Each test set is measured with the previous version and then with the new one: progress is read on data the engine has never seen.

DateTest setRecallDocuments without leakPrecision
1.8.22026-11-15holdout-v3.jsonl94.5%67%98.1%
1.8.12026-11-15holdout-v3.jsonl89.1%39.1%98%
1.8.12026-11-15holdout-v2.jsonl94.3%68.2%98.1%
1.8.02026-11-15holdout-v2.jsonl91.7%56.8%98.1%
1.8.02026-11-13holdout-v1.jsonl92.4%54.1%93.3%

Replay

Four commands, on your computer

You need Node.js 20 or later. The entity-recognition model is downloaded from Hugging Face, at the same revision as in the app, with its checksum verified.

unzip kaviar-benchmark-1.8.2.zip && cd kaviar-benchmark
npm install
npm run setup      # downloads the local model (SHA-256 checked)
npm run bench      # results in results/latest.json, offline