CAFE — Compound AI Factorial Evaluation

Design. Run. Attribute.

You bring your pipeline as a black box; CAFE builds the experiment around it and tells you which components actually drive quality — and which just add cost.

architecture

01 // Design

Factorial experiments

Declare categorical, ordinal, or continuous factors, and CAFE expands the full factorial design automatically — or a fractional 2^k-p design when the full grid is too large, with resolution and alias reporting.

play_arrow

02 // Run

Replicated & resumable

Every configuration runs multiple times to separate real effects from run-to-run LLM noise. Checkpoints are written as it goes, so an interrupted study resumes instead of restarting — and one failed item never sinks the whole run.

biotech

03 // Attribute

Statistics matched to your rubric

Ordinal rubrics get a cumulative-link mixed model, binary pass/fail a logistic GLMM, and numeric scores a linear one. On top: partial η² attribution, Cohen's d, Pareto frontiers, and Krippendorff's α for judge↔human reliability.

The systematic pipeline

Declare

Wrap your system as run(config, item) and list the factors to vary.

Design

Choose full or fractional factorial. Preview the plan and cost before spending tokens.

Replicate

Run every cell × input × replication. Resume from checkpoints automatically.

Judge

LLM judge with transparent prompts, or collect human ratings for calibration.

Attribute

Get per-factor effects, significance, interactions, and a quality/cost Pareto frontier.

Developer native. Scientifically rigorous.

Use it from Python, a notebook, or the web platform.

check_box pip-installable cafe-core with zero core dependencies
check_box Works with any LLM provider via LiteLLM
check_box Human rating workflow + inter-rater reliability built in
check_box Self-hostable web platform via Docker Compose

          
          study.py
        

          import cafe

study = cafe.Study(
    name="rag-reranker",
    system=my_rag,
    factors=[
        cafe.Factor("retrieve", ["none", "dense", "dense_rerank"]),
        cafe.Factor("model", ["small", "large"]),
    ],
    dataset=questions,
    rubric=cafe.rubrics.CORRECTNESS_0_3,
    judge=cafe.LLMJudge(model="ollama_cloud/gpt-oss:20b"),
    replications=3,
)

result = study.evaluate()  # answers → judge → attribution
print(result.report())
        

Explore three real studies

Browse three real CAFE evaluations: the HotpotQA RAG study from the paper (re-run), a customer-support chatbot, and invoice processing. Below, the first one: 18 configurations, 900 judged answers on a 0–2 scale.

check_box Every verdict is auditable: prompt + raw response stored
check_box Effect sizes separate real drivers from noise
check_box Pareto trade-offs show quality vs. cost/latency

open_in_new Open the live demo

          
          factorial_report
        
                Factor
                η²p
                p
              
                retrievalbest: with reranker · 1.73
                0.35
                <0.001
              
                modelbest: gpt-oss-120b · 1.53
                0.06
                <0.001
              
                retrieval × modelretrieval helps the small model most
                0.03
                <0.001
              
                placeboa step that does nothing · no effect
                0.00
                0.91
              
                best config: dense_with_reranker · gpt-oss-20b → 1.80 / 2

Factor	η²_p	p
retrievalbest: with reranker · 1.73	0.35	<0.001
modelbest: gpt-oss-120b · 1.53	0.06	<0.001
retrieval × modelretrieval helps the small model most	0.03	<0.001
placeboa step that does nothing · no effect	0.00	0.91
best config: dense_with_reranker · gpt-oss-20b → 1.80 / 2

Open source. Self-hostable. Reproducible.

Join researchers and engineers who want their evaluation to be as rigorous as their model training. Star the repo, read the docs, or cite the paper.

star Star on GitHub description Read the docs school Read the paper

Built in the open — contributions are welcome. Add your own techniques, open an issue, or send a pull request; see the contributing guide to get started.