Accuracy: how the named people are checked

Whodar names individuals. An assessment that names the wrong person is worse than no assessment, so this page says how the naming is measured, on what, and what the numbers were. Everything here is reproducible from a public repository with the commands given.

The problem with measuring yourself

Two earlier ways of judging whodar were not good enough to publish.

The simulation gauntlet scores against a synthetic company whose answers whodar's own generator planted. It is useful for catching regressions, which is why it runs in CI, and it is worthless as evidence to somebody outside: the answer key and the thing being tested come from the same code.

Tuning on the target is worse. A threshold chosen until it agreed with a repository, and then reported as agreement with that repository, is a measurement of nothing.

So the checks below use targets whodar was never tuned against and judges written independently of it: git itself for the sanity check, and the ownership files project maintainers wrote for their own governance for the real one. There is no single public ground truth for organizational expertise, so separate observable properties are measured separately rather than pretending one benchmark proves the whole system.

The method

Pick a public repository. Run an assessment. For each of the heaviest subjects, ask whodar who holds it, then ask git log who touched that directory most, and see whether whodar's name is among git's top three.

The two measures are deliberately different. Whodar counts a subject once per commit, saturates repetition, decays by recency, and joins one human's several identities. Git's file-touch count does none of that. They should broadly agree and should not agree exactly: perfect agreement would mean whodar is a slower git shortlog.

Run it yourself:

git clone https://github.com/prometheus/prometheus.git
whodar assess --repo-path ./prometheus --codeowners ./prometheus \
  --out prometheus-assessment

git -C prometheus log --since="2 years ago" --no-merges \
  --format='@@%aN' --name-only

The second command is the judge. Group its output by author, count directory segments, and compare the top three per directory against the experts field in findings.json.

Measurement one: agreement with git, on prometheus/prometheus

Measured 2026-09-01 against the tip of main, a two-year window, 2,759 commits, 428 people, 190 subjects scored.

The named person appeared in git's own top three for 17 of 19 subjects heavy enough to judge. Read this as a sanity check rather than a validation: the judge is commit frequency, which shares its biases with the thing being judged. The measurement below is the one that does not.

One correction was applied to git's side, and it matters. Prometheus has no .mailmap, so its history holds "György Krajcsovits" and "George Krajcsovits", and "Bartlomiej Plotka" and "Bartek Plotka", as four people where there are two. Without folding those, git's own ranking disagrees with itself, and whodar scores 3 of 19 rather than 17. The identity join is not a detail on top of the ranking; on a repository without a mailmap it is most of what makes the ranking mean anything.

Measurement two: recovering human-written ownership

The stronger test uses ground truth humans wrote for their own reasons: the per-directory OWNERS files projects in the Kubernetes ecosystem maintain for real governance, aliases expanded to individuals. The harness ships in the repository as whodar eval owners; the index is built from git history alone, with the CODEOWNERS connector off, so declared ownership never leaks into the input. The naive baseline is git's top three committers per directory, and the directories where that baseline misses are scored as their own cohort, because a tool that only finds the maintainers who are also the top committers is a slower git shortlog.

Measured 2026-09-02, two-year windows, place-scoped ownership model:

Three readings.

Ownership is asked about places, and scoring it by place is what wins. whodar's subjects pool every path sharing a name; the place model reads the same walk by directory, with the identity join, bot filtering, trailer credit, and a breadth discount layered on. From git history alone that beats commit counting by thirteen points on test-infra.

Where a review happened matters more than how much review data there is. Crediting reviewers with the words of a pull request's title, which is what a forge connector naturally gives you, adds nothing an ownership question can use: it doubled the population we could judge and left the ranking flat. The same reviews credited against the directories the pull request actually changed take test-infra to sixty percent against a twenty-nine percent baseline, and take the cohort that commit counting cannot reach from twenty-eight percent to forty-eight. The link costs no extra request: a merge commit names its pull request, so the history already knows which change touched which places.

Both repositories now beat the baseline, kubernetes included. Kubernetes is the harder case, because its approvers approve through review rather than authorship, and with git history alone whodar trailed commit counting there. With review placed properly it leads by ten points. The absolute numbers stay lower than test-infra's because two thousand pull requests is a few months of kubernetes velocity; the gap to close now is depth of read, not method.

Read the cohort C column as the real one. Every judged directory where the designated approver is also a top committer is a directory commit counting already gets right, and a tool that only wins those is a slower git shortlog.

Commit trailers (Reviewed-by, Acked-by, Co-authored-by) are also credited at ingest, which moved test-infra's git-only score two points; the mechanism matters more than the size here, since the Kubernetes ecosystem barely uses trailers and trailer-heavy ecosystems will see more.

Measurement three: prediction, and what it told us

A temporal holdout builds the index from history up to a cutoff, asks who each place rests on knowing only that, and then reads the work that came after. Nothing past the cutoff reaches the index, so the answer cannot leak into the prediction. It ships as whodar eval holdout.

It does not favor us, and that changed the product rather than the page. Measured on prometheus and on kubernetes/test-infra, the naive prediction, the past window's top committers, forecasts future activity better than whodar does. The reason is coherent: the breadth discount that makes whodar good at finding designated owners deliberately down-weights the people who touch everything, and those are exactly the people who go on committing most. Finding who owns an area and forecasting who will be busy in it are different questions, and one formula was answering both.

So the wording changed. Concentration findings now say what the record holds, that an area has one person in it and nobody behind them, which is a checkable present fact. They no longer say what will happen when that person leaves, which is a forecast this measurement does not support.

Measurement four: does concentration predict fragility

The holdout above asks who will be busy, which whodar does not claim to know. The claim it does make is narrower: that an area resting on one person is fragile. That is testable in the same windows. Take every area busy before the cutoff, note whether whodar called it single-held, then read the future for the areas whose leading person stopped working there, and see which ones fell silent.

Measured 2026-09-02 across five repositories, a three-year window with a one-year cutoff, quiet meaning activity fell by ninety percent:

When whodar says an area rests on one person and that person stops working there, the area goes quiet about three and a half times as often as an area it did not flag whose leading person also left. Two repositories contribute a single flagged area each and are noise on their own; they are in the table because leaving them out after seeing them would be choosing the sample to fit the answer.

One honest qualification. Part of this is definitional: an area with one person in the record has, by construction, nobody else already there. What is not definitional, and is the whole of the risk being described, is that nobody new arrives either. A single-held area could plausibly attract a successor and mostly does not; a covered area could plausibly collapse when its leading person goes and mostly does not.

This is the forward-looking measurement the concentration finding rests on, and unlike the activity holdout it supports the claim. It ships as part of whodar eval holdout.

The assessment itself

The report this was measured on is published as it came out of the command, unedited, at whodar.dev/sample-assessment.html, alongside the seal that certifies it at sample-assessment.loomseal. It carries an evaluation mark, because it was produced by an unlicensed install, and the mark cannot be removed without breaking the signature.

What this does and does not show

It shows that on a large repository nobody tuned against, whodar's heaviest subjects name people git agrees are doing that work.

It does not show the ranking is right where git is a poor judge, which is most of what the product claims. Commit counts say nothing about the person who reviews every change, answers every question in Slack, and commits rarely. Whodar reads those sources precisely because the commit log does not hold them, and this check cannot reach that.

It also does not measure concentration findings, only who is named. On prometheus the concentration result was that no significant subject is critically concentrated: the core systems carry bus factors above twenty. A tool that reports a healthy project as healthy is the same tool that can be believed when it reports otherwise.

Rerunning this

The numbers above will drift as the repository moves. Rerun both commands and recompute rather than trusting this page: a published accuracy figure that is never rechecked becomes a claim rather than a measurement.