CAFE is in active development toward an open-source release and an EMNLP 2026 System Demonstration. This page tracks what is shipped, what is being built next, and the research directions on the horizon.
The pip-installable evaluation engine: Study / Factor,
full-factorial design generation, async resumable executor, checkpointing, and the
cafe run CLI. The defining reframe: black-box + declared factors,
library-first.
Configurable LLM judge with structured-output probing, presets, rubrics, and transparent prompts. Three statistical layers: descriptive marginal means, Gaussian mixed-effects model (F / p / partial η² / Cohen's d), ordinal CLMM via R, and a binary logistic GLMM.
Regular 2k-p fractional factorial designs with resolution/alias reporting. Cost-quality-latency Pareto frontiers. CSV answer sheets, human rater ingestion, and Krippendorff's α for judge↔human calibration.
A FastAPI + React platform to define studies over a discovered pipeline, launch runs, watch live SSE progress, explore results, and collect human ratings by name. Ships with a HotpotQA RAG seed system. No auth, self-hostable via Docker Compose.
6-page paper + live deployed demo + screencast + real evaluation. Deadline 2026-07-10. Evaluation plan: recovery experiment, judge↔human reliability, and holdout/CV on public data.
Sequential / Bayesian-optimization / bandit mode to find the best configuration fastest. Kept conceptually distinct from the Explain/DoE spine — it answers "what is the best config?" rather than "which factor matters?".
Memoize shared upstream sub-paths across cells so factorial designs only recompute what changed. This is what makes large factorial studies financially survivable, especially paired with split-plot designs.
Split-plot designs that exploit hard-to-change factors (e.g., re-indexing a corpus) and cheap-to-change factors (e.g., temperature). D-/I-optimal designs with infeasible-combo constraints. Response-surface methods for continuous knobs.
Ingest OpenTelemetry-style traces from an untouched production system and run factorial attribution on observed internals, without surrendering execution.