# Retrieval benchmark data — Mounaji Studio

The saved runs behind every retrieval number on our blog, and the code to
recompute them. Explained in
the retrieval posts on [mounajistudio.com/blog](https://www.mounajistudio.com/blog).

```
node verify.mjs                                        # every published number, ok / FAIL
node verify.mjs 2026-09-28-bm25-only 2026-09-28-hybrid # any two runs, paired, 95% CI
node verify.mjs 2026-09-10-substring 2026-09-10-bm25 graph+boosts   # a row other than core
```

Node 18 or later, no dependencies. Download the whole folder first:
`README.md`, `claims.json`, `verify.mjs`, `evalstats.js` and every file listed
in `claims.json` under `runs/`.

## What is here

| file | what it is |
|---|---|
| `runs/*.json` | One benchmark run: its conditions, the averages per row, and the score of **every query** per row. |
| `evalstats.js` | The statistics, copied unchanged from our benchmark (comments are in Spanish). Paired differences, 95% interval with 1.96·SE. |
| `claims.json` | Each number we published, the post it appears in, and the two runs and row that produce it. |
| `verify.mjs` | Recomputes `claims.json` and exits non-zero if a published number does not match. |

## What is not here, and why

- **The query text and the documents.** Queries are real tasks from our own
  work and some name clients. The documents are our internal notes. Without
  them you cannot rerun the search. You can check every statistic we derived
  from it. That covers the claims we make, but not the corpus we made them on.
- **Query ids are opaque.** Each is the first 12 hex characters of a SHA-256
  of our internal id: stable across runs so that pairing works, and meaningless
  outside.
- **Three published numbers** have no per-query data behind them. They are
  listed in `claims.json` under `notCovered`.

## Rows

`core` has no graph expansion and no boosts, and is the only row allowed to
decide. `graph` adds the graph hop, `graph+boosts` is what production serves
(and reads the labels it is scored on), and `graph+boosts-honest` removes that
query's own labels from the boosts.

## Metrics

`recall@k` is the share of useful documents in the first k results. `mrr` is
mean reciprocal rank of the first useful one. `ndcg10` is normalized DCG at 10.
`noise10` is the share of the top 10 that someone marked as noise (lower is
better).

If a number here disagrees with one in a post, the post is wrong. Tell us:
mounaji.studio@gmail.com.