CAFE — Compound AI Factorial Evaluation

Design. Run. Attribute.

CAFE measures compound systems — it does not implement them. You bring your pipeline as a black box; CAFE runs the experiment around it and tells you what matters.

architecture

01 // Design

Factorial experiments

Declare categorical, ordinal, or continuous factors. CAFE expands full factorial designs automatically, and fractional 2^k-p designs when the combinatorial cost gets real — with resolution and alias reporting.

play_arrow

02 // Run

Replicated & resumable

Every configuration runs multiple times to separate real effects from LLM noise. Checkpoints write incrementally: crash at hour 5 of 6 and you resume, not restart. Per-item error isolation keeps one bad cell from killing the study.

biotech

03 // Analyze

Correct models per scale

Ordinal rubrics get a cumulative-link mixed model (CLMM). Binary pass/fail gets a logistic GLMM. Numeric gets a Gaussian mixed model. Plus partial η² effect attribution, Cohen's d, Pareto frontiers, and Krippendorff's α for judge↔human reliability.

The systematic pipeline

Declare

Wrap your system as run(config, item) and list the factors to vary.

Design

Choose full or fractional factorial. Preview the plan and cost before spending tokens.

Replicate

Run every cell × input × replication. Resume from checkpoints automatically.

Judge

LLM judge with transparent prompts, or collect human ratings for calibration.

Attribute

Get per-factor effects, significance, interactions, and a quality/cost Pareto frontier.

Developer native. Scientifically rigorous.

Use it from Python, a notebook, the CLI, or the web platform. The same engine everywhere.

check_box pip-installable cafe-core with zero core dependencies
check_box Works with any LLM provider via LiteLLM
check_box Human rating workflow + inter-rater reliability built in
check_box Self-hostable web platform via Docker Compose

          
          study.py
        

          import cafe

study = cafe.Study(
    name="rag-reranker",
    system=my_rag,
    factors=[
        cafe.Factor("retrieve", ["none", "dense", "dense_rerank"]),
        cafe.Factor("model", ["small", "large"]),
    ],
    dataset=questions,
    rubric=cafe.rubrics.CORRECTNESS_0_3,
    judge=cafe.LLMJudge(model="ollama_cloud/gpt-oss:20b"),
    replications=3,
)

result = study.evaluate()  # answers → judge → attribution
print(result.report())
        

Explore a real study

Browse an actual CAFE evaluation — a HotpotQA RAG study over 18 configurations — in the full web UI. Read-only, no sign-up, backed by real data.

check_box Every verdict is auditable: prompt + raw response stored
check_box Effect sizes separate real drivers from noise
check_box Pareto trade-offs show quality vs. cost/latency

open_in_new Open the live demo

          
          factorial_report.csv
        
                Config
                Mean
                η²
                p
              
                dense_rerank · large
                2.71
                0.23
                <0.001
              
                dense · large
                2.45
                0.05
                <0.001
              
                none · large
                1.98
                -
                -
              
                dense_rerank · small
                1.62
                -
                -

Config	Mean	η²	p
dense_rerank · large	2.71	0.23	<0.001
dense · large	2.45	0.05	<0.001
none · large	1.98	-	-
dense_rerank · small	1.62	-	-

Open source. Self-hostable. Reproducible.

Join researchers and engineers who want their evaluation to be as rigorous as their model training. Star the repo, read the docs, or cite the paper.

star Star on GitHub description Read the docs school Read the paper (arXiv)