CAFE is open-source and under active development. This page tracks where we are taking it — the capabilities we are building next across statistics, judging, experimental design, and the platform.
Deepening the part that makes CAFE more than a leaderboard: honest, scale-correct variance attribution.
A fair credit-split across factors that works for the ordinal model and for unbalanced or fractional designs — exactly where a single-model partial η² is weakest.
Treat each human rater as a random effect, then nest raters within their group or organization — revealing how different expert pools systematically score the same system, and separating that rater-side variation from the factor effect you are actually measuring.
Full posterior effect sizes and credible intervals, not just point estimates and p-values.
Use a handful of human labels to debias the LLM judge's effect estimates, with statistical guarantees.
Turn "not significant" into "underpowered — add N questions to detect a gap this size."
More ways to turn answers into trustworthy scores.
"Is A better than B?" per item (Bradley–Terry / Elo), alongside the existing absolute-rubric mode.
Several named scores per answer — e.g. correctness, faithfulness, tone — each with its own attribution.
Run a panel of judges and aggregate their verdicts to reduce single-judge bias.
Surface position, verbosity, and self-preference checks as a first-class result, not just a reliability number.
Making large factorial studies affordable — and even smarter.
Memoize shared sub-paths across configurations so a factorial only recomputes what changed — the lever that makes big studies financially survivable.
Cost-aware designs that exploit hard-to-change versus cheap-to-change factors, plus D-optimal and response-surface methods for continuous knobs.
Bayesian optimization and bandits to reach the best configuration fastest — complementing the "which factor matters?" spine rather than replacing it.
Lowering the barrier from "write Python" to "click and read the verdict."
An add/remove-a-component wizard that returns a one-line verdict: "web search: +0.4, d = 0.6, p = 0.03 — worth it."
A/B two structurally different pipelines head-to-head, not just levels within one system.
The effects table, forest plot, and comparison table as paper-ready LaTeX and figures.
Ship public benchmarks out of the box, and evaluate an untouched production system straight from its OpenTelemetry traces.