Otrobonita AI Labs

LLM-as-a-Judge

Traditional software testing asserts one output against one expected value. Generative systems break that contract — not because they are unpredictable, but because correctness is one-to-many. A single input has a large set of acceptable answers, and two very different responses may both be right.

So we evaluate instead of assert. One model performs the task, a second reviews the output against explicit written criteria, and in a RAG system the reviewer also sees the evidence the answer was built from.

LLM-as-a-Judge

Subject for Evaluation
mark-twain (RAG)
Judging LLM
claude-haiku-4-5 · anthropic
Vector Cluster
twain_test

This tool uses the same style of Python evaluation functions as a full RAG eval suite. The rubric has eight dimensions:

  • persona_integrity
  • groundedness
  • attribution_precision
  • embroidery
  • source_fidelity
  • tone_adherence
  • synthesis
  • voice

In a normal lab run we score all eight dimensions on every eval. Here we split them so each pair can be measured and exemplified without 32 repetitive judgments. Question → retrieval → unedited answer → judging LLM → verdict. No composite score.

  1. Eval 1 — first two dimensions, measured and exemplified: persona_integrity, groundedness
  2. Eval 2 — next two dimensions, measured and exemplified: attribution_precision, embroidery
  3. Eval 3 — another pair of dimensions measured: source_fidelity, tone_adherence
  4. Eval 4 — final pair of dimensions: synthesis, voice

Required model claude-haiku-4-5 is not available. No substitute was selected.