Benchmarking & Evaluation

In science and ambitious engineering, changing a model, a Specialist, a skill, or a memory architecture should be measured, not guessed. Sandwalk includes a built-in evaluation harness (Dashboard → Benchmarking and sandwalk eval in the CLI) that runs real agent sessions end-to-end.

Shipped Benchmark Suites

Bio-Capybara-full

Biomedical Research · Full Harness

Multi-step biomedical workflows evaluating literature synthesis, target/compound lookup, skills, and MCP orchestration.

Sandwalk · Bio-Capybara ↗

BixBench-Verified-50

50 CapsuleData Tasks · CompBio

50 verified computational biology and bioinformatics agent capsules graded with string, numeric range, and LLM verifiers.

Phylo · BixBench ↗

MMLU-Pro

12,032 Questions · 14 Subjects

Challenging 10-option reasoning benchmark across 14 STEM and professional subjects executed through real Sandwalk agent sessions.

TIGER-Lab · MMLU-Pro ↗

MMLU

14,042 Questions · 57 Subjects

Broad 57-subject multiple-choice knowledge evaluation with fixed seed-42 stratified sampling for fair model-to-model comparison.

Hendrycks et al. · MMLU ↗

ARC-C (AI2 Challenge)

1,172 Questions · Science Reasoning

Grade-school and scientific reasoning challenge questions from the Allen Institute for AI evaluated under the agent harness.

Allen Institute for AI ↗

MBPP

500 Tasks · Python Code Exec

Mostly Basic Python Problems graded by executing generated code against assert-based test suites in the sandbox.

Google Research · MBPP ↗

LiveCodeBench

1,055 Tasks · Competitive Programming

Contamination-free code generation benchmark graded against stdin/stdout public and hidden test cases in the sandbox.

livecodebench.github.io ↗

Running Evaluations

From Dashboard → Benchmarking, pick one or more suites, select a model (or queue multiple models/Specialists for an A/B comparison), set concurrency and sample size, and click Launch.

Or run headlessly from the CLI:

sandwalk eval

You can also ask Capy in any session: "Run a 20-question BixBench comparison between Claude Sonnet and GPT-5 with the Single-cell Analyst specialist."

Benchmarking Roadmap (TODO)

We are actively expanding the evaluation harness to benchmark long-horizon memory, multi-agent collaboration, and skill crystallization:

• Long-Horizon Memory & Omnigraph Recall Bench (TODO) — Measure multi-session fact retention, Supports vs. Contradicts claim resolution, and KV-cache token efficiency across Cookbook modes 01–06.
• Skill Crystallization Lift Eval (TODO) — Quantify accuracy and step-count improvements before vs. after crystallizing domain sessions into reusable ~/.sandwalk/skills/<slug>/SKILL.md skills.
• Multi-Specialist Grid vs. Single-Agent Ablations (TODO) — Benchmark parallel Specialist cohorts (e.g., Nova + Soren + Otium) orchestrated by Capy against single-session baselines.
• SWE-bench Verified & Terminal-Bench Suites (TODO) — First-class software engineering and PTY terminal benchmark suites alongside BixBench and Bio-Capybara.
• Automated CI Harness Regression Gates (TODO) — Per-Specialist accuracy, latency, and token-budget gates enforced via sandwalk eval in CI.