Sponsored Content material
TL;DR:
- Smaug Flash posts the road’s largest jumps: +14.3 pts in agentic coding, +13.7 on AutomationBench, +19 on NL2Repo-Bench – instantly concentrating on DeepSeek V4 Flash’s failure mode in lengthy tool-use loops.
- Smaug Mini (28B) beats Claude Sonnet 5, Claude Opus 4.6, and GPT-5.6 Luna on IFBench, AutomationBench, and JobBench – regardless of a slight dip in LiveBench agentic coding.
- Smaug Agentic exhibits the smallest rating deltas (+0.3 general) however ran 113 duties over 7+ hours with a median of 78 agent steps and 0 infrastructure errors – behavioral stability is the story, not the numbers.
In all three circumstances, the good points focus the place the bottom mannequin’s gaps have been widest. Identical recipe, completely different flooring.
Significance of AI Brokers’ Efficiency
An AI agent may present an output with out finishing the duty. In real-world eventualities, there’s a want for the agent to grasp the consumer question, determine an acceptable software, analyze output, and take subsequent actions. After some variety of cycles, the mannequin would veer away from the preliminary aim, thus compromising on the credibility of the end result.
It presents an issue for groups creating dependable brokers. Whereas a chatbot offers a response to a single request, an agent has to retain its context and make choices primarily based on associated actions. With the extension of workflow, there can be further value implications in addition to possibilities of errors.
Abacus.AI skilled related shortcomings in its personal operations, with agentic duties taking a number of hours to finish, sustaining a considerable amount of context from stage to stage, and incessantly invoking instruments from different paperwork, APIs, and databases. These operations may run out of steam or not be accomplished efficiently in any respect. Abacus.AI describes its findings on its analysis web page within the following means: “Making a state-of-the-art mannequin work like a very good agent is a coaching problem, not a prompting problem.” Normal language fashions, regardless of their skills, weren’t at all times prepared for prolonged utilization. Immediate caching finally grew to become much less efficient, and context grew larger.
The fundamental method was to enhance the fashions unaided as an alternative of attempting to compensate for his or her shortcomings through the use of prompting or weight changes that will repair the recognized issues. This manner of working has resulted in creating the household of Smaug fashions. It’s important not solely to ask how efficient these fashions are but additionally the place the fine-tuning works the perfect. The efficiency of the fashions relies upon enormously on the mannequin and activity sort.
What Is the Smaug Line?
Smaug is a household of three open-weight fashions fine-tuned by Abacus.AI for agentic workloads. Every mannequin is constructed on a distinct open-weight base, chosen for a definite level on the aptitude–effectivity curve. All three are publicly accessible on Hugging Face.
A standard methodology involving human-curated, precise agentic trails and synthetic knowledge primarily based on onerous circumstances was used for all of the three bases, which is why the acquire patterns are comparable and the variations are significant.
| Mannequin | Base Mannequin | Optimized For | Dimension |
|---|---|---|---|
| Smaug Flash | DeepSeek V4 Flash 0731 | All the time-on enterprise brokers, pace + value | 304B |
| Smaug Mini | Qwen3.8-27B | Multimodal, compact, one-off duties | 28B |
| Smaug Agentic | Kimi K3 | Frontier-scale, long-horizon coding | 2.8T (MoE) |
Making use of the identical recipe to a few very completely different bases is itself a analysis declare: that the methodology is normal sufficient to provide constant, measurable enhancements no matter structure or scale. What follows examines every mannequin’s outcomes to find out the place that declare holds strongest — and the place trade-offs emerge.
Smaug Flash: The Largest Features within the Line
Smaug Flash is the place the Smaug methodology delivers its largest benchmark jumps. It’s an agentic fine-tune of DeepSeek V4 Flash, typically often called the workhorse of the road, constructed for enterprise self-improving brokers that run constantly. Its base, DeepSeek V4 Flash, is quick and cost-effective however vulnerable to spins and confusion in long-context software use. The fine-tuning focused that particular failure mode.
The outcomes on LiveBench replicate this precision. The +14.3 level acquire in agentic coding starting from 46.8 to 61.1 is the most important single-category delta in the whole Smaug line. On AutomationBench, the rating improved by 13.7 factors (25.1 to 38.83), and on NL2Repo-Bench the acquire was 19 factors (54.2 to 73.3). These should not marginal enhancements; they symbolize the mannequin closing a real functionality hole in precisely the workflows it was designed for.
Smaug Flash vs DeepSeek V4 Flash vs Claude Sonnet 5:
¹ = vendor-reported rating, not run beneath Abacus.AI’s harness.
LiveBench Class-by-Class Breakdown:
These modest reductions in Information Evaluation (-0.2) and Reasoning (-0.4) may very well be seen as a trade-off since it’s extra essential to hone these expertise to be helpful for manufacturing brokers, for instance, the power to make use of instruments accurately and course of longer contexts, than to retain full efficiency in reasoning. When it comes to the benchmarks evaluating these expertise, Smaug Flash outperforms Claude Sonnet 5.
Smaug Mini: The Stunning Story
Smaug Mini is a complicated iteration of the Qwen3.8-27B mannequin and the small-scale model of the Smaug sequence with 27 billion parameters. The mannequin is meant for multimodal duties resembling picture/video interpretation, instruction following, and power use.
With its comparatively smaller measurement, deployment may probably turn into simpler beneath sure situations of {hardware} and inference configuration. You will need to be aware that Smaug Mini has achieved greater efficiency than a number of superior closed fashions in sure benchmarks the place the latter often excel.
Smaug Mini vs Base and Frontier Closed Fashions:
¹ = vendor-reported rating, not run beneath Abacus.AI’s harness.
LiveBench Class Profile for Smaug Mini:
Mainly, Smaug Flash has the largest LiveBench acquire in agentic coding, however as a result of Qwen3.8-27B, the bottom of Smaug Mini is already sturdy at coding-style duties, it’s largest good points are in Arithmetic (+3.5) and Reasoning (+2.8) which leaves no room for enchancment. The fine-tuning discovered the gaps that have been truly open. On NL2Repo-Bench, Mini additionally gained 13.5 factors over its base (42.3 to 55.8), a acquire value noting for code-related workflows.
The twist is essential: Smaug Mini’s agentic coding rating (-0.6) is definitely barely decrease than its base, but it leads Claude Sonnet 5, Claude Opus 4.6, and GPT-5.6 Luna on IFBench (instruction following), AutomationBench, and JobBench. JobBench particularly exhibits a 17.1 level acquire over the bottom (33.4 to 50.5). Herein lies the excellence between what a benchmark class is ready to measure and what the mannequin is able to attaining in actuality — since completely different benchmarks measure completely different skills, and a drop in a single space doesn’t imply a scarcity in others.
Smaug Agentic: When Conduct Issues Extra Than Numbers
Smaug Agentic is the most important one on this line-up. It is a fine-tuning of Kimi K3, which is the state-of-the-art Combination of Consultants mannequin from Moonshot AI, with 2.8 trillion parameters. Its benchmark deltas over the bottom are the smallest of the three Smaug fashions. The analysis web page states this instantly: “the behavioral change is greater than the deltas.”
Smaug Agentic vs Kimi K3 Base:
LiveBench Class Profile for Smaug Agentic:
Two benchmarks the place the bottom mannequin wins needs to be said clearly: Terminal-Bench 2.1 (86.5 vs 88.3) and MMMU-Professional (81.0 vs 81.6). These should not hidden in footnotes within the official knowledge, they usually belong in any trustworthy evaluation.
What the numbers don’t seize is what Abacus.AI describes because the mannequin’s major behavioral enchancment: suppressing runaway reasoning whereas preserving regular deliberation. Throughout 113 agentic coding duties and greater than seven hours of steady operation, Smaug Agentic ran a median of 78 agent steps per activity with no infrastructure errors and no timeouts. At frontier scale, the fine-tuning didn’t produce headline rating jumps, slightly it produced a mannequin that doesn’t break down within the lengthy loops the place frontier-scale fashions are literally deployed.
The Coaching Methodology Behind the Features
The identical methodology was utilized to all three Smaug fashions. It combines human-curated traces from actual agentic work with artificial knowledge grounded in onerous, difficult examples. Coaching knowledge is filtered to take away situations that exploit the scoring standards of a activity slightly than truly fixing it. The aim is to reward real problem-solving, not format compliance.
What differs throughout the three fashions is the precise fine-tuning method:
- Smaug Flash: The eye issue matrices are alone modified with the assistance of three LoRA adapters, two of which have been educated by supervised studying and one with choice coaching. The remainder of the parts keep unchanged and byte an identical to that of the official DeepSeek V4 Flash launch. This ensures that the mannequin is totally appropriate with the bottom mannequin on any inference stack, with the power to have full 1-million-token context window.
- Smaug Agentic: Supervised fine-tune of Kimi K3 with reasoning tokens masked from the loss throughout coaching. This preserves the mannequin’s native considering course of whereas instructing higher tool-use habits. Each architectural parameter is unchanged from the bottom, making it a drop-in substitute for any inference stack that serves Kimi K3.
- Smaug Mini: The identical normal methodology utilized to a 27B dense base, producing constant patterns: good points within the classes the place the bottom had essentially the most headroom, and trade-offs in areas already well-covered.
“The core philosophy of this technique is captured in a single quote on the Abacus.AI’s analysis web page: Most fine-tuning silently overwrites the reasoning of a mannequin for the sake of activity compliance – you get the format, however you lose the thoughts.” Smaug coaching recipes attempt to get round this pitfall through the use of the coaching finances on knowledge which improves the mannequin, not knowledge which leverages benchmark scores. This is likely to be an element explaining the numerous improve in agentic coding (+14.3 factors) seen in Smaug Flash – its fine-tuning addressed the weak point of the bottom mannequin.”
What the Benchmarks Are Truly Measuring
Understanding the place Smaug improves most requires understanding what every benchmark is designed to measure.
LiveBench is an ICLR Highlight paper from Abacus.AI. It goals to forestall knowledge contamination by producing questions primarily based on essentially the most up-to-date sources resembling math contests, LeetCode questions, arXiv abstracts, information articles, and Kaggle datasets. Questions are refreshed each month to make it more durable for fashions to have memorized the solutions earlier than coaching. Scoring is completed primarily based on floor fact, not LLM as a choose scoring methodology, which might be misguided on difficult prompts. There are seven classes in LiveBench: Reasoning, Coding, Agentic Coding, Instruction Following, Arithmetic, Language, and Information Evaluation.
The opposite benchmarks every measure narrower expertise:
- AutomationBench — real-world tool-use automation (600 public duties, strict cross)
- DeepSWE 1.1 — software program engineering executed by coding brokers
- NL2Repo-Bench — translating pure language to repository-level code
- IFBench — instruction following precision (300 duties, prompt-level free)
- JobBench — multi-step job-level activity completion (65 duties, official protocol)
- GPQA Diamond — graduate-level science reasoning
- SciCode — scientific coding
- AA-LCR — long-context reasoning
- MMMU-Professional — multimodal understanding
As a result of these benchmarks measure completely different, although associated, expertise, a excessive rating on one doesn’t assure a excessive rating on one other. That’s the reason Smaug Mini leads all in contrast closed fashions on AutomationBench and JobBench whereas displaying a -0.6 delta in LiveBench’s Agentic Coding class. They don’t seem to be the identical measure. It’s also why some competitor scores marked ¹ within the tables above that are the vendor-reported figures have to be interpreted fastidiously, not run beneath Abacus.AI’s personal harness.
The place Features Are Largest: Comparability Throughout All Three Fashions
| Class / Benchmark | Smaug Flash | Smaug Mini | Smaug Agentic |
|---|---|---|---|
| LiveBench Agentic Coding (delta vs base) | +14.3 ★ | -0.6 | +2.4 |
| LiveBench Instruction Following | +3.8 | +1.2 | -0.4 |
| LiveBench Arithmetic | +3.3 | +3.5 ★ | -0.5 |
| LiveBench Reasoning | -0.4 | +2.8 | -0.4 |
| AutomationBench (delta vs base) | +13.7 ★ | +4.5 | +0.2 |
| NL2Repo-Bench (delta vs base) | +19.0 ★ | +13.5 | — |
| Behavioral endurance at scale | Steady enterprise loops | Compact, one-off duties | 78 steps/activity, 7+ hours, zero timeouts ★ |
The sample throughout all three is in keeping with one clarification: the good points focus the place the bottom mannequin’s gaps have been widest and the place the coaching knowledge was designed to push. Flash’s base had the most important failure mode in agentic coding and therefore it was liable to spinning in lengthy tool-use loops in return, the fine-tuning produced the most important acquire there (+14.3). Mini’s base was already sturdy at coding, so the good points unfold throughout math, reasoning, and language as an alternative. Agentic’s base was already at frontier functionality, leaving little room for benchmark rating enchancment; the fine-tuning centered on behavioral reliability slightly than uncooked functionality.
Implications for Sensible Use
It’s extra useful when evaluating the usage of Smaug fashions to begin with the workload that the mannequin is designed to deal with.
For functions that demand sustained, long-context software use, doc processing, or repeated API calls throughout hours-long workflows, Smaug Flash’s +14.3 agentic coding and +13.7 AutomationBench good points over its base are essentially the most instantly related knowledge. The place multimodality and instruction-following duties are involved, and the place a mannequin that matches on a single GPU at 28B mannequin measurement is desired, the related comparisons come from Smaug Mini’s IFBench rating (82.0), AutomationBench (41.8), and JobBench (50.5), all beating Claude Sonnet 5 and GPT-5.6 Luna. The place longevity in a whole lot of steps per agent is required with out infrastructural failure, the behavioral stability data from Smaug Agentic (median: 78 steps, 7+ hours, zero timeouts from 113 duties) carries much more weight than any benchmark delta.
Different helpful metrics in addition to the benchmarks are: activity completion ratio, latency at context size as in manufacturing, value per thousand agent steps, and reproducibility.
Open weights present entry and deployment flexibility, however don’t assure effectivity or ease of integration with out testing towards the precise workload. {Hardware} configuration, context size, and activity issue all have an effect on inference habits in ways in which benchmark scores can not predict.
The benchmark outcomes are a place to begin for figuring out which fashions are value evaluating — not an alternative to testing on the precise activity.
Conclusion
Every enchancment seen within the Smaug sequence will depend on what specific weak point this model’s fine-tuning aimed to resolve. For instance, Smaug Flash reveals the largest enhancements in its personal class, rising scores by 14.3 factors for LiveBench agentic coding, 13.7 factors for AutomationBench, and 19 factors for NL2Repo-Bench. That is due to the weaknesses that the unique mannequin had with the prolonged multi-step agent workflow. Smaug Mini, alternatively, demonstrates enhancements in arithmetic (+3.5), reasoning (+2.8), and language (+2.7). It beats the talked about frontier closed fashions in instruction following and job-level automation regardless of being a 28 billion parameter mannequin. Then again, Smaug Mini exhibits decline in agentic coding rating (by 0.6 factors) as compared with its authentic mannequin. There may be additionally a novel case of Smaug Agentic, which exhibits a modest enchancment of +0.3 factors however has excessive operational efficiency that’s outlined by 78 agent steps per activity, seven plus hours of run time, and 0 infrastructure errors.
The outcomes stand out by advantage of their divergence from one mannequin to a different. The identical coaching routine is mirrored otherwise primarily based on the precise mannequin that’s used and the constraints of that specific mannequin. The outcomes additionally embrace details about these elements of the testing process the place there is no such thing as a enchancment in efficiency. As an example, the Kimi K3 base mannequin outperforms Smaug Agentic in Terminal-Bench 2.1 and MMMU-Professional whereas Smaug Mini exhibits some discount in efficiency in LiveBench agentic coding. Together with such limitations makes the outcomes of this analysis train extra real looking.
Builders in search of a efficiency analysis of varied open-weight fashions to be used in agent-based functions would discover the outcomes of Smaug household very helpful. As an alternative of figuring out the perfect mannequin for any software, the efficiency outcomes would assist perceive which fashions is likely to be higher beneath sure circumstances.
