Retrieval Evaluation

Measure retrieval quality with golden question sets — hit@k and MRR per retrieval leg.

Retrieval tuning (chunk size, FTS language, fusion parameters, reranking) should be measured, not guessed. HyperSaaS ships an eval harness that runs a golden question set through each retrieval leg independently and reports where the first correct chunk lands.

Running an evaluation

python manage.py eval_retrieval golden.json --workspace <uuid>
python manage.py eval_retrieval golden.json --knowledge-base <uuid> --modes keyword
python manage.py eval_retrieval golden.json --documents <uuid>,<uuid> \
    --modes hybrid,hybrid_rerank        # the rerank go/no-go comparison

Scope with --workspace, --knowledge-base, or --documents; select legs with --modes; control result depth with --top-k (default 10).

Golden-set format

{
  "cases": [
    {
      "description": "optional note for humans",
      "query": "What did Q3 revenue do?",
      "expect": {
        "document": "report.pdf",
        "content_contains": ["revenue grew", "gelirleri"]
      }
    }
  ]
}

A retrieved chunk matches when the optional document name substring matches and any content_contains substring appears (case-insensitive). Two deliberate design choices:

Content substrings, not chunk IDs — golden sets survive re-ingestion and chunking changes.
content_contains accepts a list — any match counts, so mixed-language corpora can accept either phrasing.

A format template ships at backend/documents/evals/golden.example.json.

Modes

Mode	What it measures
`semantic`	The embedding leg alone (pgvector cosine)
`keyword`	The FTS leg alone — works offline, no OpenAI key needed
`hybrid`	RRF fusion, exactly as `search_documents` runs it
`hybrid_rerank`	Hybrid pool + LLM rerank, forced regardless of `DOCUMENT_RERANKER` — measures lift before you enable anything

Output

Per-case table (rank of the first correct chunk per mode) plus aggregates:

#   query                                     semantic   keyword    hybrid  hybrid_rerank
1   What did revenue do in the third quarter?        1      miss         1              1
2   How profitable was the company recently?         1      miss         1              1
3   hava durumu raporu                                1         1         1              1

Aggregate metrics (rank of first correct chunk):
mode            hit@1   hit@3   hit@5     MRR  evaluated
semantic         1.00    1.00    1.00   1.000          3
keyword          0.33    0.33    0.33   0.333          3
hybrid           1.00    1.00    1.00   1.000          3
hybrid_rerank    1.00    1.00    1.00   1.000          3

This sample output also illustrates the documented simple FTS trade-off: the keyword leg nails exact-phrase queries but misses natural-language questions — which the semantic leg answers at rank 1. Hybrid fusion gets the best of both.

The harness degrades gracefully: if query embedding fails (no OpenAI key), semantic/hybrid report n/a and the keyword leg still runs; if the rerank call fails, hybrid_rerank reports n/a without poisoning the other columns.

Decision workflow

Ingest a real document; write ~10–20 golden cases from it.
Run all four modes. hybrid is your baseline.
If hybrid_rerank MRR meaningfully beats hybrid → set DOCUMENT_RERANKER=llm.
If keyword scores near zero on a single-language corpus → set DOCUMENT_FTS_LANGUAGE to that language, re-ingest, re-run.
Re-run the eval after any chunking or fusion change — the golden set is your regression suite for retrieval quality.