Evidence and evaluation
How to evaluate Sakana Fugu beyond the benchmark chart
The official scores are useful vendor evidence. They are not a purchase order. This playbook turns the chart into a controlled comparison that measures correctness, cost, latency, failure recovery, observability, and governance on work your team actually performs.
1. Read the published claims as a map of strengths
Sakana AI's current benchmark summary reports the results below. They are reproduced here to compare the models, not to imply independent verification by this site.
| Model (version) | What Sakana AI reports |
|---|---|
| Fugu Ultra (v2.0) | Best or joint-best score on five of eight benchmarks: GDP.pdf, Chartography, DeepSWE, Toolathon, and SWEFish; top-2 on seven of eight. |
| Fugu Max (v1.0) | Best overall score on six benchmarks: Terminal Bench 2.1, GPQAD, AA-LCR, GDP.pdf, AutomationBench, and SWEFish; expands the cost-performance frontier on seven of ten; output pricing 40–60% below Sonnet 5, GPT 5.6 Terra, and Kimi K3. |
| Fugu Cyber (v1.0) | 86.9% on CyberGym and 72.1% on CTI-REALM, compared with GPT-5.5-Cyber and Mythos-Preview. |
Read this as positioning, not a ranking. Ultra is presented as the peak-quality option and Max as the cost-performance option, and the two overlap on several suites (GDP.pdf and SWEFish appear in both lists). Different models are scored on different benchmark sets, so "always use the top one" is not something these summaries support. Every figure here is a vendor claim; use the blind method below to test it on your own tasks.
2. Ask what each benchmark is standing in for
Repository issue resolution
SWE Bench style tasks can inform coding-agent evaluation, but your language mix, repository size, test quality, and tool permissions may be different.
Terminal operation
Terminal benchmarks probe multi-step action and recovery. They do not automatically cover your deployment permissions, network limits, or destructive-action safeguards.
Standalone code generation
LiveCodeBench is useful for code reasoning under a defined harness. It says less about navigating a large private codebase or maintaining a long editing session.
Scientific reasoning
GPQA and SciCode can indicate difficult reasoning ability. They do not prove the model will cite internal evidence correctly or respect domain-specific review standards.
Long context
Long-context results are a starting signal. Real retrieval quality depends on where evidence appears, distractors, tool calls, and whether the output can point back to the source.
Agentic orchestration
A benchmark can reward a final answer while hiding extra latency, retries, route instability, or token use. Production evaluation must add those missing dimensions.
3. Keep the evidence boundary visible
The official product page says some baseline values are provider reported. The technical report describes the authors' training and evaluation setup. Those are valuable primary sources, but the producer and evaluator are still closely related.
Evidence you have
- Named benchmarks and scores
- A technical report with method detail
- Documented model behavior and API fields
- Qualitative vendor case studies
Evidence you still need
- Your exact prompts and tools
- Repeated runs, not one showcase
- Blind human or deterministic scoring
- Total time and billable usage
- Failure and recovery behavior
- Security and governance fit
It is a reason to lower confidence and run a controlled trial, not a reason to declare the vendor result false.
4. Build a ten-task test set that can reveal a bad fit
A useful test set contains tasks you would actually pay to complete. Keep it small enough to score deeply and varied enough that one model cannot win through a narrow trick.
| Task group | Example | Hidden failure to include |
|---|---|---|
| Routine x 2 | Small bug diagnosis, focused explanation | Simple work where orchestration overhead may not pay off |
| Repository x 2 | Cross-file regression, migration review | Misleading naming or an incomplete test suite |
| Research x 2 | Compare sources, reproduce a method plan | Conflicting evidence or a missing primary source |
| Long context x 2 | Contract analysis, design review | Relevant clause buried among plausible distractors |
| Adversarial x 2 | Underspecified request, impossible demand | The correct answer should ask, refuse, or state uncertainty |
Do not add a task after seeing which model failed it. Keep the test set frozen for the comparison, then create a new version for the next evaluation cycle.
5. Freeze the run contract
- Same task text and attachments for every model
- Same tool access, network access, and repository snapshot
- Same maximum wall time and output constraints
- Same retry count and retryable error policy
- Same system or developer instructions where supported
- At least three runs for tasks where nondeterminism could change the conclusion
- Raw response, status, duration, and usage retained for every attempt
If a field is accepted but ignored by one provider, do not pretend the run conditions are identical. Record the difference in the evaluation notes and focus on the user-visible contract.
6. Score the answer without showing the model name
Rename outputs A, B, and C before review. A simple 0 to 4 rubric is easier to apply consistently than a vague “felt better” judgment.
| Dimension | 0 | 2 | 4 |
|---|---|---|---|
| Correctness | Unsafe or fundamentally wrong | Partly right with material errors | Correct under the stated test oracle |
| Evidence | Unsupported or invented | Some traceable support | Claims map cleanly to sources or code |
| Completeness | Misses the main task | Covers core path, misses edge cases | Covers required behavior and key failure modes |
| Uncertainty | Confidently fabricates | Some caveats, not well calibrated | States limits and asks only necessary questions |
| Actionability | Cannot be used | Needs substantial interpretation | Clear next actions with verifiable outcomes |
Define the oracle before scoring. For code, it may be tests plus expert review. For research, it may be a source checklist. For a decision memo, it may be required questions and traceable evidence.
7. Add the dimensions a benchmark chart leaves out
Latency
Record median and 90th percentile wall time. A quality win can still fail an interactive service target.
Cost
Capture full usage, including orchestration fields. Report cost per successful task, not only cost per request.
Reliability
Count timeouts, tool failures, malformed outputs, and tasks requiring manual rescue.
Recoverability
Measure whether a failed run can resume safely or forces the whole task to repeat.
Observability
Ask whether logs and response metadata let an operator understand what happened without exposing sensitive content.
Governance
Check agent opt-outs, data handling, regional availability, audit needs, and whether a fixed pool is acceptable.
8. Make a narrow decision, not a universal ranking
A useful conclusion has the form “Use model X for task class Y under constraint Z.” It does not need one global winner.
Adopt Fugu when it meets the quality bar and its latency or control advantages matter.
Escalate to Ultra only where repeated blind scoring shows a material quality gain worth the extra time and usage.
Use Cyber only for authorized defensive security tasks after access approval, with scope and human review.
Keep your current model where orchestration adds no measurable outcome.
Keep self-managed orchestration when route visibility and deterministic governance are mandatory.
Re-run the test set after material model updates or workload changes. Sakana AI says it aims to update orchestration after new frontier model releases, so a decision can expire even if your application code does not change.
When a task needs agent behavior rather than a score, read Fugu AI for what the agent does, how the orchestration is billed, and the selection table between Fugu, Ultra, Max, and Cyber.
Primary sources
- Sakana AI Models: current score table, supported models, and behavior
- Sakana Fugu Technical Report: architecture, training, and reported evaluation methodology
- Sakana AI Fugu product page: benchmark notes and qualitative examples
Scores and product behavior are time sensitive. The evaluation process is designed to remain useful when the exact leaderboard changes.