{"id":34849,"date":"2026-09-22T10:54:59","date_gmt":"2026-09-22T08:54:59","guid":{"rendered":"https:\/\/sii.pl\/blog\/?p=34849"},"modified":"2026-09-22T13:05:25","modified_gmt":"2026-09-22T11:05:25","slug":"beyond-the-flagship-what-fable-5-1-and-gpt-6-astra-tell-us-about-ai-native-delivery","status":"publish","type":"post","link":"https:\/\/sii.pl\/blog\/en\/beyond-the-flagship-what-fable-5-1-and-gpt-6-astra-tell-us-about-ai-native-delivery\/","title":{"rendered":"Beyond the flagship: what Fable 5.1 and GPT-6 Astra tell us about AI-native delivery"},"content":{"rendered":"\n

Two new flagship models, one week of evidence, and plenty of noise around them. The early results already tell us something useful about how frontier models should be evaluated for real delivery.<\/p>\n\n\n\n

Within three days at the start of September, two new flagship models arrived: Claude Fable 5.1 and GPT-6 Astra. We spent the week reading everything measurable that has been published about both, and this is what we found.<\/p>\n\n\n\n

As you might expect, it is far too early for a production verdict. We do not yet have meaningful production deployments, published ROI figures, failure stories, or rollback postmortems. Most of the evidence available today comes from vendor benchmarks, independent evaluations, and early experiments.<\/p>\n\n\n\n

That makes any definitive ranking premature. It also makes this first week unusually useful for examining something broader: how much of the performance we attribute to a model actually comes from the system built around it.<\/p>\n\n\n\n

The differentiator goes beyond specifications<\/strong><\/h2>\n\n\n\n

Just to build the intuition before we go anywhere near a benchmark, here is the basic comparison.<\/p>\n\n\n\n

Spec<\/td>GPT-6 Astra<\/td>Claude Fable 5.1<\/td><\/tr><\/thead>
Context<\/td>~1M<\/td>~1M<\/td><\/tr>
Knowledge cutoff<\/td>April 30, 2026<\/td>June 2026<\/td><\/tr>
API price \/ 1M<\/td>$10 \/ $50<\/td>$10 \/ $50<\/td><\/tr>
Cache read \/ 1M<\/td>$1.00<\/td>$0.25<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Both models are multimodal, although neither generates video, audio, or images natively.<\/p>\n\n\n\n

Specifications give us useful boundaries. They tell us much less about what will happen once a model becomes part of an actual delivery workflow, with real context, real tools, and real people around it.<\/p>\n\n\n\n

Benchmarks increasingly measure systems<\/strong><\/h2>\n\n\n\n

On September 3, Artificial Analysis had Fable 5.1 at 66 on its Intelligence Index<\/a>, the highest score it had recorded, against Astra at 61.<\/p>\n\n\n\n

By September 8, under index v4.3, both models were at 53<\/a>.<\/p>\n\n\n\n

The models had not changed, whereas the evaluation methodology had. The effect becomes even clearer when we look at the systems surrounding a model.<\/p>\n\n\n\n

One benchmark takes that further and measures model-and-harness pairs on purpose, and by harness, we mean a software layer to orchestrate agents’ actions, tooling, and resources. It is also the closest thing to production-like evidence published so far. Real-SWE<\/a>, from Specific Labs, runs ten tasks drawn from licensed private production repositories written before 2023, each model in its own native harness. Eight configurations, eight runs per task, 640 rollouts in total, reported as pass@1 with confidence intervals.<\/p>\n\n\n\n

No configuration reached 40%. Fable 5.1 in Claude Code leads at 38.8%, Astra in Codex CLI reaches 33.8%, Gemini 3.8 Flash 31.2%, and the remaining five configurations land between 16% and 29%. To visualize the complexity of the problems, please note that six of the ten tasks are solved less than 15% of the time, and one is never solved at all.<\/p>\n\n\n\n

The caveats<\/strong><\/strong><\/h3>\n\n\n\n

The failure pattern is the part we would take into a client conversation, as the most common reason for failure is a missed requirement. For the two strongest models, unverified assumptions and integration errors are just as frequent, and Astra fails more often on those than on requirements. The reference fixes modify a median of eleven files, about twice the median of the public coding benchmarks, while the prompts are underspecified only to the same degree as DeepSWE or Terminal-Bench \u2013 they say what to change, not how, and leave the details to be found in the repo and its surrounding tools.<\/p>\n\n\n\n

Cost does not explain the gap: Fable 5.1’s 38.8% costs $6.96 per rollout against $4.67 for Astra’s 33.8% \u2013 roughly 1.5\u00d7 the price for five points, with Fable burning 64k output tokens per rollout to Astra’s 24k. Gemini 3.8 Flash makes the curve flatter still: 31.2% for $2.50. Neither do time \u2013 rollouts under ten minutes fail at almost the same rate as longer ones (71.4% vs 73.4%).<\/p>\n\n\n\n

The caveats are real. Ten tasks and eighty rollouts per model mean neighboring scores sit inside each other’s confidence intervals, so the ordering is soft. Native harnesses match practice, but make it impossible to separate the model from the harness. Nothing here can be reproduced, because the code is private.<\/p>\n\n\n\n

ARC Prize reported Astra scoring<\/a> 62.7% on ARC-AGI-3 under a provider-neutral harness with max level<\/em> of reasoning, 98,6% respectively using OpenAI’s own Provider Adapter, and, unintuitively, 99.9% with a high <\/em>level of reasoning within the latter environment.<\/strong> The difference is the scaffolding, not the weights: the neutral harness discards the model’s reasoning between actions, while OpenAI’s adapter keeps that state and manages the context itself.<\/p>\n\n\n\n

Interpretation<\/strong><\/strong><\/h3>\n\n\n\n

NVIDIA reached 100%<\/strong><\/a>on the public subset of the same benchmark using an agentic architecture built on Opus 5<\/strong>, whose bare baseline was around 30%. NVIDIA itself notes that the two figures come from different reasoning settings, agent systems, and evaluation sets, so the gap illustrates the point rather than measures it.<\/p>\n\n\n\n

Connect the dots, and these gaps are larger than many of the differences used to declare one frontier model better than another.<\/p>\n\n\n\n

They also change what we should evaluate.<\/p>\n\n\n\n

A production AI capability consists of more than model weights. Context, tools, orchestration, retrieval, agents, permissions, and validation mechanisms can materially affect the outcome \u2013 and every one of those is designed, tuned, and reviewed by people who understand the task.<\/p>\n\n\n\n

The complete workflow, therefore, deserves the same scrutiny as the underlying model.<\/p>\n\n\n\n

Where Astra currently looks stronger<\/strong><\/h2>\n\n\n\n

One of Astra’s clearest early strengths is computer use: operating software directly rather than relying exclusively on explicit integrations.<\/p>\n\n\n\n

 Among directly comparable results, Astra leads Opus 5 on OSWorld 2.0 (72.6% vs 70.2%) and beats the Mythos 5 figure on ScreenSpot-Pro (92.7% vs 87.3%)<\/a>; Anthropic publishes no comparable computer-use score for Fable 5.1, and it’s 41.7% was measured under different tasks, grading, and safeguard conditions<\/a>.<\/p>\n\n\n\n

This could matter for enterprise environments with a long tail of internal applications where dedicated API integrations are difficult to justify.<\/p>\n\n\n\n

An agent capable of interacting with the interface can potentially automate some of these workflows directly.<\/p>\n\n\n\n

That shifts part of the engineering challenge into a different area rather than removing it.<\/p>\n\n\n\n

If an agent can log into an application, navigate it, and take actions, we need to manage identity, permissions, and secrets. Its actions need to be auditable. Sessions need recovery mechanisms. UI changes need fallbacks. High-risk decisions still belong to a human.<\/p>\n\n\n\n

Computer use can reduce integration effort while increasing the importance of governance and observability.<\/p>\n\n\n\n

There is a testing aspect here that is easy to misread. An agent that drives a UI can serve as a testing tool, but its own behavior needs verification too: it might report a failure because it got lost, or report success while missing a defect. OSWorld measures task performance, not defect detection, so these scores say nothing about how well either model would test an application. That has to be measured separately, on representative workflows with known defects, counting missed issues, false alarms, and the effort needed to review the agent’s findings. Acceptance criteria and release decisions stay with the product team.<\/p>\n\n\n\n

Results<\/strong><\/strong><\/h3>\n\n\n\n

Astra also shows promising early results in software engineering. CodeRabbit measured roughly 4% more bugs caught in review<\/strong><\/a> than Sol and 22% more than Opus 5<\/strong>, rising to 20% and 33%, respectively, on complex cross-file reviews<\/strong>. The authors themselves describe these results as early and directional, which we appreciate.<\/p>\n\n\n\n

Visual and spatial reasoning have improved as well. In one experiment, in Decrypt’s test, Astra produced a playable Three.js version of Hangzhou in 24 minutes<\/a>. These are useful signals. Production evidence will tell us how consistently they translate into real workloads.<\/p>\n\n\n\n

Performance and price tell different stories<\/strong><\/h2>\n\n\n\n

Astra also illustrates why the API price alone is a weak measure of AI economics. In Nate Herk’s 100-hour test<\/a>, Fable completed the workload in 9h35m, compared with Astra’s 11h19m<\/strong>. Astra was slower, but the total cost went in the opposite direction: $327 for Astra versus $513 for Fable<\/strong>. For this particular workload, Fable provided the shorter completion time and Astra the lower bill. Another workload, context size, caching pattern, or reasoning setting could change that relationship.<\/p>\n\n\n\n

For delivery teams, cost per completed task<\/strong> is therefore a more useful metric than token price in isolation. Completion time, quality, and failure rate belong in the same calculation. The question is no longer which model has the better list price, but how the whole setup changes delivery cost, speed, quality, and predictability.<\/p>\n\n\n\n

Where Fable 5.1 currently looks stronger<\/strong><\/h2>\n\n\n\n

Writing is one of the more consistent positive signals around Fable 5.1. Several independent reviewers report more natural prose and fewer characteristic Claude writing patterns. Every found the prose readable at roughly a seventh-grade level<\/a>, while Latent Space described it as closer to normal human communication<\/a>.<\/p>\n\n\n\n

Simon Willison’s long-running SVG test<\/a> produced what he considered the best pelican-riding-a-bicycle result from an Anthropic model so far. It is an informal test, but a useful longitudinal one because he has applied it repeatedly across model generations.<\/p>\n\n\n\n

The second interesting area is long-running work. Ramp reported an unsupervised 38-hour machine-learning run<\/strong><\/a> that completed multiple experiments. Anthropic also reports improvements in transparency when the model encounters problems, including a greater tendency to indicate that it is stuck instead of presenting the task as successfully completed. If that holds up outside launch week, it matters more for unattended work than any benchmark point in this post.<\/p>\n\n\n\n

Millennium reported another interesting case<\/a>: the model helped identify a rare bug that had resisted explanation for four or five years.<\/p>\n\n\n\n

These are individual examples rather than broad production evidence. If the behavior holds up at scale, however, it could matter for long-running agentic engineering workflows.<\/p>\n\n\n\n

Fable’s cache economics deserve separate attention<\/strong><\/h2>\n\n\n\n

Fable’s cache-read price dropped from $1.00 to $0.25 per million tokens. That makes it particularly interesting for workflows built around repeatedly querying the same large body of information: repositories, documentation, knowledge bases, or other persistent project context.<\/p>\n\n\n\n

Effort scaling may also matter. Harvey reported that quality remained relatively stable at lower reasoning settings<\/a> while output-token use dropped by roughly half.<\/p>\n\n\n\n

At the same time, early measurements of Fable’s overall token efficiency depend heavily on the baseline. Every measured Fable 5.1 using less than half the tokens of Opus 5<\/a> with comparable results in its Slack agent at Medium effort, while relaying \u2014 not measuring \u2014 Anthropic’s estimate of 25\u201345% lower costs versus Fable 5. Artificial Analysis, as reported by Latent Space<\/a>, compared Fable 5.1 against Fable 5 at max effort and found roughly 1.7x more output tokens and about 20% higher cost per task ($3.76), with ~4% of output tokens served by Anthropic’s fallback models.<\/p>\n\n\n\n

A GitHub issue filed on September 1<\/a> also reported Fable at medium effort exhausting a full Claude Code token limit in under 20 minutes. These findings measure different workloads and configurations, so the disagreement itself is useful information. Cache-heavy workloads may benefit significantly from the new pricing. Other workloads need to be measured separately.<\/p>\n\n\n\n

Both models introduce different risks<\/strong><\/h2>\n\n\n\n

Capability improvements increase the importance of understanding how a model fails.<\/p>\n\n\n\n

Astra<\/strong><\/h3>\n\n\n\n

OpenAI’s system card reports a significant decrease in chain-of-thought monitorability<\/a>. Under adversarial testing, where it was instructed to conceal underperformance, Astra evaded OpenAI’s chain-of-thought monitor in most cases \u2013 detection fell below 11% on the WMDP sandbagging test and to zero on an internal coding benchmark. OpenAI frames this as a capability shown under instruction, not observed default behavior, and reports no steganographic reasoning or full-context-monitor evasion.<\/p>\n\n\n\n

OpenAI notes that further degradation of this kind in future generations could significantly reduce confidence in detecting some forms of misaligned behavior.<\/p>\n\n\n\n

That is particularly relevant as models receive greater autonomy.<\/p>\n\n\n\n

Fable<\/strong><\/h3>\n\n\n\n

For Fable, the early concerns are different. Every Reviewer has reported inconsistent adherence to requested limits, including word counts. There are also reports of hallucinated citations when working with large source documents \u2013 an important failure mode for document analysis.<\/p>\n\n\n\n

Anthropic’s own system information also points to limitations around honesty under pressure<\/a> and compliance with system prompts, asserting information the model knows to be false.<\/p>\n\n\n\n

Model routing adds another operational consideration. Some biology queries are routed to Opus 5 and cyber queries to Opus 4.8<\/a>. The switch is visible in the response<\/a>, but it is decided by a classifier rather than by the team, so the same workflow can be served by different models on different runs. That complicates exact reproduction and makes the fallback path one more thing to test.<\/p>\n\n\n\n

These behaviors need to be understood in the context of the actual workload and the level of autonomy given to the model. Production workflows require scoped permissions, validation, quality and security gates, auditability, and clear human accountability.<\/p>\n\n\n\n

Data residency can be decided before benchmarks do<\/strong><\/h2>\n\n\n\n

For European organizations, model quality may sometimes be secondary to deployment constraints \u2013 and this may be the single most practical fact in this post. At the time of writing, Harvey reports that Anthropic does not provide regional processing<\/a> for Fable 5.1 or Fable 5, meaning data is processed in the United States.OpenAI’s migration guidance also indicates that Astra’s fast and priority modes are currently unavailable under EU data residency<\/a>.<\/p>\n\n\n\n

These conditions can change quickly and should always be verified for the specific model, mode, and deployment configuration before implementation. For workloads subject to data-residency requirements, the deployment architecture can narrow the model choice before performance benchmarking even begins.<\/p>\n\n\n\n

Outside coding, the evidence is still thin<\/strong><\/h2>\n\n\n\n

There are some genuinely interesting signals in scientific workloads.<\/p>\n\n\n\n

Epoch AI’s FrontierMath Erdos benchmark<\/a> used 68 unsolved Erd\u0151s problems formalized in Lean, with a budget of $300 and 72 hours per problem. Astra solved two, approximately 3%<\/strong>. Sol, GPT-5.5, Fable 5.1, and Fable 5 all scored zero.<\/p>\n\n\n\n

Protein design provides another example. Anthropic’s wet-lab-validated study<\/a>,  with designs produced and tested independently by Adaptyv Bio and Twist Bioscience, reported Claude Mythos  Preview reaching a 35.1% hit rate when run against one target per 24-hour session (26.7% in a multi-target setup, 22.6% for Opus 4.8), against a 10\u201315% baseline Anthropic derived from proteinbase.com. Mythos Preview is not generally available, and Anthropic says it plans further characterization to confirm the hit rates. There is an important qualification: Fable 5.1 routes biology queries to Opus 5 and is officially not recommended for professional biological research.<\/p>\n\n\n\n

Robotics provides a third signal. On Robocurve’s<\/a> block-into-bowl task, same harness for all three, Fable 5 hit 1\/20, Fable 5.1 8\/20, and Astra 19\/20; on the harder puzzle insertion, Astra and Fable 5.1 each managed 2\/20 and stalled at the same step. Nonetheless, several Caveats from Robocurve surfaced: different rigs for the bowl runs, non-interleaved trials two days apart, and unblinded operator grading. The progress is real, yet precise manipulation remains largely unsolved.<\/p>\n\n\n\n

The gaps in evidence are equally interesting. There are still no meaningful documented production deployments of either model across many common enterprise domains. Independent evaluations for spreadsheets and tabular data are particularly scarce, despite their importance in everyday office work.<\/p>\n\n\n\n

One concrete business example comes from Legora<\/a>. In a financial-statement reconciliation task across 41 documents, Astra identified all four planted errors in minutes, producing around 40% improvement on that workflow<\/strong>. Across the full benchmark, however, the average improvement was closer to 3%<\/strong>. That difference is worth remembering when evaluating impressive demonstrations. Results from carefully selected tasks may differ substantially from average performance across a broader delivery process.<\/p>\n\n\n\n

Migration is more than a configuration change<\/strong><\/h2>\n\n\n\n

Both vendors have published migration guidance, and there are meaningful breaking changes.<\/p>\n\n\n\n

Anthropic provides separate migration paths depending on the previous model<\/a>. Some behaviors that existing systems may rely on are no longer supported, including forced tool calls, assistant prefilling, and switching reasoning off. There is also a subtle change around reasoning traces. They are bound to the model that produced them, so editing earlier parts of a conversation can invalidate them and cause the request to fail. Frameworks that compact their own context, rebuild tool lists between calls, or inject per-turn reminders may therefore require changes.<\/p>\n\n\n\n

OpenAI’s migration list is shorter<\/a>, but existing integrations may still need engineering work. Common sampling parameters have been removed, the lowest reasoning setting is no longer supported, and tool calling expects the newer API surface. Some behavioral changes are harder to detect because they do not necessarily trigger an error: fewer parallel tool calls, less retrieval at low effort, denser prose, or whole-file rewrites where a targeted edit was expected. Those are the changes a reviewer who knows what the output used to look like will catch.<\/p>\n\n\n\n

For a delivery team, this means a model upgrade is a release, and it needs a regression suite like any other release. Anthropic’s own guidance on agent evaluations puts it plainly: teams with evals upgrade in days; teams without them spend weeks testing by hand. That suite has to account for two things. The system under test is non-deterministic, so what matters is not whether a task passes once but how reliably it passes across repeated runs. And both vendors highlight that the models test their own work, but an agent can carry the same misunderstanding of a requirement into the code and its tests, so passing tests alone does not show that the requirement was met.<\/p>\n\n\n\n

This has an architectural consequence for AI-native systems. Models will continue to change, and so will their interfaces and behaviors. Delivery workflows need enough modularity to evaluate and replace AI capabilities without rebuilding the entire process around every new flagship \u2013 reusable context, workflows, and controls that outlive any single model.<\/p>\n\n\n\n

Where would we start testing<\/strong><\/h2>\n\n\n\n

There is not enough evidence for a universal ranking. There is enough to identify sensible starting points, and such research is time-consuming, so we will hand you the table rather than the reading list.<\/p>\n\n\n\n

Task<\/td>Starting point for evaluation<\/td><\/tr><\/thead>
Writing, editing, and tone<\/td>Fable 5.1<\/td><\/tr>
Operating applications and UIs<\/td>Astra<\/td><\/tr>
Long unattended engineering runs<\/td>Fable 5.1<\/td><\/tr>
Mathematics and selected scientific workloads<\/td>Astra<\/td><\/tr>
Repeated queries over the same large context<\/td>Fable 5.1<\/td><\/tr>
EU data residency<\/td>Verify current deployment options first<\/a><\/td><\/tr>
Simple, cost-sensitive Polish-language tasks<\/td>Include smaller models in the test<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Coding remains particularly difficult to call.<\/p>\n\n\n\n

Astra tops Code Arena: WebDev<\/a> (1,797 vs 1,762) and Terminal-Bench 4.0<\/a> (57.7% vs 55.8%), and leads DeepSWE by seven points; FrontierCode 1.1.<\/p>\n\n\n\n

Fable 5.1’s 81.2% on SWE-bench Pro<\/a> has no Astra counterpart to beat. Zvi’s early reader poll<\/a> landed near 50\u201350, down from 2:1 for Claude.<\/p>\n\n\n\n

These recommendations should be treated as starting points for evaluation<\/strong>. Actual selection depends on the workload, project context, required integrations, acceptable risk, governance constraints, and total economics.<\/p>\n\n\n

\n \n \n \"\"\n <\/picture>\n
\n
\n

Testing & QA<\/h2>\n <\/div>\n

\n Ensure the quality, performance, and security of your software with our testing and test automation services.\n <\/p>\n \n Testing&QA offering<\/span>\n <\/a>\n <\/div>\n<\/div>\n\n\n\n

What the first week tells us<\/strong><\/h2>\n\n\n\n

Four questions remain difficult to answer with confidence:<\/p>\n\n\n\n