BusinessSlopBench: Which Models Write Documents You Could Actually Send?
I ran 33 models through eight business briefs. Muse Wonder is almost as good as GPT-6 Astra with my anti-slop rules, after fixing an ambiguous grading point. Every draft and correction is open source.
TL;DR
Muse Wonder 1.0 is almost as good as GPT-6 Astra on my house-rule writing panel. Both passed 54/54 content checks. Muse produced 7/8 drafts ready without edits, versus Astra's original 8/8. Both full Astra repeats returned 6/8. Muse now ranks second, ahead of GPT-6.1 Sol, and its eight house-rule drafts cost about 21 cents versus Astra's 36 cents.
Those are corrected scores. The vendor brief had an ambiguous signing-authority point that the grader applied inconsistently. I removed that inference from the historical scores for every model and replaced it with explicit authority facts for future runs.
This is a one-person pilot, not a benchmark. Everything is open source at github.com/mattlgroff/business-slop-bench, and I am asking someone with the time and the standing to make it a real one.
October 13 Update: Muse Wonder Is Close to Astra
Muse Wonder was the sleeper hit I was looking for. With the anti-slop instructions, it passed every scored content check and needed one repetition edit across eight drafts.
| Model / attempt | Content checks | Content ready | Ready without edits | Input / output per 1M tokens | Cost for 8 house drafts |
|---|---|---|---|---|---|
| GPT-6 Astra, original | 54/54 | 8/8 | 8/8 | $10 / $50 | $0.358 |
| Muse Wonder 1.0 | 54/54 | 8/8 | 7/8 | $4 / $20 | $0.207 |
| GPT-6.1 Sol | 54/54 | 7/8 | 4/8 | $2 / $10 | $0.073 |
| GPT-6 Astra, latest repeat | 54/54 | 8/8 | 6/8 | $10 / $50 | $0.359 |
I would call that almost as good as Astra on this panel. Muse's remaining edit is a summary bullet that repeats the three bullets above it. Astra still has the best primary result, but its two full repeats both needed two edits. I have only one Muse attempt per brief, so I cannot turn a one-draft gap into a reliability claim.
The plain-prompt result is weaker: Muse passed 53/54 content checks and produced 3/8 drafts ready without edits. Its strategy deck called a pilot approved and ready when the source only said the internal documents were approved. That is still a content failure. Astra's original plain result was 54/54 and 4/8 ready without edits.
All sixteen Muse calls cost $0.3785. Its house-rule run cost about 42% less than Astra's original house-rule run. Its plain run cost more than Astra's plain run, so the lower token price did not make every document cheaper.
The Signing Check Was a Benchmark Flaw
The source said Priya owned procurement and needed CFO approval before signing. It never identified the authorized signatory. Muse wrote, “Please approve Alpha so Priya can proceed with signing.” The grader withheld a point for that implication, while passing similar Astra wording.
That was an inconsistent test. I retired the signatory inference across all 33 models. Seventeen grounding verdicts changed, including findings against Claude, Qwen, Kimi, Google and Mistral. Other factual errors, missed requirements and style findings stayed in place. Muse moved from 53/54 checks and 6/8 ready without edits to 54/54 and 7/8. The saved output did not change.
This does not establish that Priya was allowed to sign. It means the old brief could not support that grading distinction consistently. The replacement states that Priya coordinates procurement but cannot sign, Alex is the sole authorized signatory, and CFO approval must come first without delegating authority. A draft must communicate those roles and that order. No model has run that new version yet; the tables here are the corrected historical panel.
I also tested Gemini 4 Argon and Nemotron 3.5 Lightning Free. With the same correction, Argon passed 51/54 house checks with 4/8 ready without edits. Nemotron passed 46/54 with 2/8 ready without edits. The saved Gemini 3.8 Flash run passed 51/54 with 3/8 ready without edits; I did not rerun it. Muse came closest to Astra among these additions.
Every correction and its original quotation. Muse comparison, prices and saved outputs.
October 7 Update: Haiku 5.5 and the Claude Family Against Astra
Haiku 5.5 finished all sixteen drafts for about one cent. It used zero em dashes and met every word limit, but still invented team progress, commitments and ownership. It passed 50/54 content checks with the plain brief and 49/54 with my house rules. Two drafts in each condition were ready without edits.
Here is where the latest version I have tested from each Claude family sits against Astra. This table ranks these five by writing performance: content checks first, then drafts ready without edits. All five used low reasoning and the same eight briefs with my anti-slop instructions. The table includes the October 13 grading correction; all models are now ranked on writing performance.
| Rank | Model | Content checks | Ready without edits | Input / output per 1M tokens | Cost for 8 drafts |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra | 54/54 | 8/8 | $10 / $50 | $0.358 |
| 2 | Claude Opus 5.5 | 52/54 | 3/8 | $4 / $20 | $0.253 |
| 3 | Claude Fable 5.1 | 51/54 | 3/8 | $10 / $50 | $0.570 |
| 4 | Claude Sonnet 5.5 | 51/54 | 2/8 | $2 / $10 | $0.115 |
| 5 | Claude Haiku 5.5 | 49/54 | 2/8 | $0.1 / $0.5 | $0.008 |
If I had to stay on Anthropic for this kind of business writing, I would choose Opus 5.5 based on these results. Opus and Fable both produced 3/8 house-rule drafts ready without edits, but Opus passed one more content check and cost less than half as much. With plain prompts, Opus produced two sendable drafts and Fable produced none. Paying more for Fable did not buy better writing on this panel.
Haiku matched Sonnet's two sendable house-rule drafts for about fifteen times less generation cost. Sonnet still passed two more content checks. Haiku's low price makes it interesting, but it does not remove the need to review the substance.
Astra's 8/8 is its original house-rule result. Both full repeats returned 6/8 ready without edits and still passed all 54 content checks. Each row above retains its primary attempt. These are provisional assistant grades on eight short briefs, not a general ranking of everything these models can do. Performance, prices and evidence.
October 6 Update: Mistral Large 4, Un Échec
The French word for failure is échec. Mistral Large 4 performed poorly on this writing panel. After the October 13 correction and additions, it sits 31st of 33 with the plain brief and 26th with my anti-slop rules.
| Condition | Checks | Content ready | Ready without edits |
|---|---|---|---|
| Plain | 46/54 | 1/8 | 0/8 |
| House rules | 49/54 | 3/8 | 2/8 |
It invented prior discussions, claimed the team was ready, and said no pending approvals blocked launch. The earlier signing-authority deduction is removed under the October 13 correction. The discovery proposal promised to schedule kickoff within five business days, a commitment the source never supplied. Even with the explicit ban, the house-rule drafts contained three em dashes. The change order and vendor memo were ready without edits under the corrected scoring.
Some cheaper Chinese models did better. GLM 5.3 Flash passed 48/54 house checks versus Mistral's corrected 49/54. Both produced two drafts ready without edits, for about $0.0046 across eight drafts versus Mistral's $0.0197. DeepSeek V4.1 Flash passed more checks in both conditions, though all its drafts still needed editing. Qwen 3.8 Max beat Mistral on both checks and drafts ready without edits. This was Gateway inference; I did not test laptop deployment or quantized versions.
There is a configuration limitation. Gateway advertised low reasoning for Mistral, but rejected that request with a provider-options error. The successful requests used none, and Gateway warned that it ignored reasoning configuration. These are provider-default reasoning results, not a verified low-reasoning comparison with Astra or Sol.
All sixteen completed Mistral drafts cost about 2.5 cents. They met their word limits, but cheap generation did not make the documents reliable. The failed request remains recorded separately, and these are still provisional assistant grades on eight short briefs. Comparison, drafts and findings.
September 29 Update: GPT-6.1 Sol Takes Second Place
GPT-6.1 Sol is close to Astra on this panel, but Astra still produces cleaner documents. Sol passed 53/54 checks with the plain brief and 54/54 with my anti-slop rules. That put it second in both rankings at the time of this September update. Muse now ranks second in the corrected house-rule table, with Sol third.
I also reran Astra on all eight briefs in both conditions. Both models used low reasoning, identical prompts and no output-token cap. These are the fresh results:
| Model | Condition | Checks | Content ready | Ready without edits |
|---|---|---|---|---|
| GPT-6.1 Sol | Plain | 53/54 | 7/8 | 2/8 |
| GPT-6 Astra, fresh run | Plain | 54/54 | 6/8 | 2/8 |
| GPT-6.1 Sol | House rules | 54/54 | 7/8 | 4/8 |
| GPT-6 Astra, fresh run | House rules | 54/54 | 8/8 | 6/8 |
Content ready means the draft passes every content check, fits the word limit and has no drafting residue. Ready without edits also requires it to pass the style review. Sol's house-rule drafts needed three repetition edits and removal of a reference to "the source pack" in the handoff deck. Astra needed two editorial edits. Both used zero em dashes with the house rules.
The sixteen Sol drafts cost $0.0946. The sixteen fresh Astra drafts cost $0.4712. At roughly one-fifth the cost, Sol was a strong second-place result at the time of this comparison. It still needs an editing pass.
The ranking tables below retain Astra's original attempt, which had 8/8 house-rule drafts ready without edits. Both full repeats returned 6/8. I keep those attempts separate instead of choosing the best output for each brief. These remain unblinded assistant grades on eight short briefs. Fresh comparison and every draft.
September 28 Update: Sonnet 5.5 Is a Big Improvement
Sonnet 5.5 is a big improvement over Sonnet 5 in these runs. With my anti-slop rules, content-ready drafts increased from the corrected 3/8 to 4/8. Under the October 13 correction, drafts ready to send without edits went from Sonnet 5's 1/8 to Sonnet 5.5's 2/8. Sonnet 5.5 passed 51/54 checks versus Sonnet 5's corrected 49/54, at the same listed token price.
Content ready means the draft passes its content checks and word limit and contains no drafting residue. It can still need style edits. Two of Sonnet 5.5's four content-ready drafts needed repetition removed, leaving two ready to send as is.
Opus 5.5 passed 52/54 checks with the house rules after the signing correction. It had three content-ready drafts, and all three passed the style review too. So Opus 5.5 produced more drafts ready to send as is: 3/8 versus Sonnet 5.5's 2/8. Sonnet 5.5's big improvement is over Sonnet 5; it does not beat Opus 5.5 on that measure.
It also used zero em dashes across all sixteen drafts, even without the house rules. Sonnet 5 used sixteen in its plain-prompt drafts. The discovery proposals and change orders improved, though Sonnet 5.5 still invented some facts and referred to the source pack in finished documents. Three plain-prompt drafts exceeded their word limits.
Astra still leads this small panel. But this is a meaningful improvement in the writing I care about. I used the same prompts, rules and low reasoning setting for both Sonnet versions. Sonnet 5's saved sample is from September 22; Sonnet 5.5's is from September 28. Full comparison and draft evidence.
Why I Built It
Most of what I use language models for at work is not code. It is the proposal, the change order, the readout for a client, the four-slide deck for a steering committee. Every coding benchmark will tell you which model writes the best Rust. None will tell you which model will invent a delivery date you never agreed to.
When I said as much on X, Dex Horthy replied:
lemme save you some time on that one - if a model wrote it, probably slop. wordcels vindicated etc
He was quoting Pedram:
pretty wild that the hardest job for even the most frontier models is not shape rotation but writing, wordcels absolutely vindicated
He is right. But some are more slop than others, and I wanted numbers.
What It Does
Eight synthetic business briefs, each with a small pack of source facts. Every model writes each brief once in two conditions: the plain brief, and the plain brief plus my Zero Defect anti-slop rules. Each draft is graded against six or seven checks per brief (the fee is right, no invented start date, the right person owns the decision), including a grounding check: every material claim must trace to the source or be labeled a proposal. That is 54 checks per condition. The historical vendor grounding check now excludes the under-specified signatory inference described above; all other grounding requirements remain. A draft is ready to send as is if it passes every check, fits the word limit, contains no drafting residue, and has no em dashes, no rhetorical negation, and no slop phrasing.
Reasoning effort was set to low, except for the DeepSeek models, which used none, and Mistral Large 4, whose reasoning configuration was ignored as described above. Prices are Vercel AI Gateway list rates per million tokens from the saved October 13 catalog snapshot. Historical route restrictions no longer affect the writing ranking.
Results
These tables use the October 13 corrected v2 grades. All 33 models are eligible on writing performance. Ranks follow content checks, then ready-without-edits counts, content-ready counts and fewer confirmed failures. Equal scores on those measures do not establish an ordering in quality.
Plain brief, no instructions about style:
| Rank | Model | Checks (of 54) | Ready as is (of 8) | In / Out $/M |
|---|---|---|---|---|
| 1 | GPT-6 Astra | 54 | 4 | 10.00 / 50.00 |
| 2 | Muse Spark 1.3 | 53 | 4 | 1.25 / 4.25 |
| 3 | GPT-5.6 Terra | 53 | 3 | 2.00 / 12.00 |
| 4 | GPT-6 Luna | 53 | 3 | 0.10 / 0.50 |
| 5 | Muse Wonder 1.0 | 53 | 3 | 4.00 / 20.00 |
| 6 | GPT-6.1 Sol | 53 | 2 | 2.00 / 10.00 |
| 7 | Claude Opus 5.5 | 53 | 2 | 4.00 / 20.00 |
| 8 | GPT-6 Sol | 52 | 3 | 2.00 / 10.00 |
| 9 | Grok 4.7 | 52 | 3 | 2.00 / 6.00 |
| 12 | DeepSeek V4.1 Flash | 52 | 0 | 0.30 / 1.20 |
| 13 | GPT-5.6 Luna | 51 | 2 | 0.20 / 1.20 |
| 14 | Claude Sonnet 5.5 | 51 | 2 | 2.00 / 10.00 |
| 15 | Gemini 4 Argon | 51 | 1 | 2.00 / 10.00 |
| 17 | Gemini 3.8 Flash | 51 | 0 | 0.75 / 3.75 |
| 19 | Claude Fable 5.1 | 51 | 0 | 10.00 / 50.00 |
| 20 | Claude Haiku 5.5 | 50 | 2 | 0.10 / 0.50 |
| 25 | Claude Opus 5 | 50 | 0 | 5.00 / 25.00 |
| 26 | GLM 5.3 Flash | 49 | 1 | 0.15 / 0.50 |
| 30 | Claude Sonnet 5 | 48 | 0 | 2.00 / 10.00 |
| 31 | Mistral Large 4 (provider-default reasoning) | 46 | 0 | 0.68 / 2.09 |
| 32 | Qwen 3.8 Flash | 46 | 0 | 0.15 / 0.47 |
| 33 | Nemotron 3.5 Lightning Free | 45 | 1 | 0.00 / 0.00 |
Same brief plus the anti-slop rules:
| Rank | Model | Checks (of 54) | Ready as is (of 8) | In / Out $/M |
|---|---|---|---|---|
| 1 | GPT-6 Astra | 54 | 8 | 10.00 / 50.00 |
| 2 | Muse Wonder 1.0 | 54 | 7 | 4.00 / 20.00 |
| 3 | GPT-6.1 Sol | 54 | 4 | 2.00 / 10.00 |
| 4 | GPT-6 Sol | 53 | 7 | 2.00 / 10.00 |
| 5 | GPT-6 Luna | 53 | 6 | 0.10 / 0.50 |
| 6 | Muse Spark 1.3 | 53 | 5 | 1.25 / 4.25 |
| 9 | GPT-5.6 Luna | 52 | 4 | 0.20 / 1.20 |
| 10 | GPT-5.6 Terra | 52 | 3 | 2.00 / 12.00 |
| 12 | Claude Opus 5.5 | 52 | 3 | 4.00 / 20.00 |
| 14 | Gemini 4 Argon | 51 | 4 | 2.00 / 10.00 |
| 15 | Gemini 3.8 Flash | 51 | 3 | 0.75 / 3.75 |
| 16 | Grok 4.7 | 51 | 3 | 2.00 / 6.00 |
| 18 | Claude Fable 5.1 | 51 | 3 | 10.00 / 50.00 |
| 20 | Claude Sonnet 5.5 | 51 | 2 | 2.00 / 10.00 |
| 22 | DeepSeek V4.1 Flash | 50 | 0 | 0.30 / 1.20 |
| 23 | Claude Opus 5 | 49 | 3 | 5.00 / 25.00 |
| 26 | Mistral Large 4 (provider-default reasoning) | 49 | 2 | 0.68 / 2.09 |
| 27 | Claude Haiku 5.5 | 49 | 2 | 0.10 / 0.50 |
| 28 | Claude Sonnet 5 | 49 | 1 | 2.00 / 10.00 |
| 30 | GLM 5.3 Flash | 48 | 2 | 0.15 / 0.50 |
| 31 | Nemotron 3.5 Lightning Free | 46 | 2 | 0.00 / 0.00 |
| 33 | Qwen 3.8 Flash | 42 | 1 | 0.15 / 0.47 |
Selected rows are shown here. All 33 models, content readiness, prices and charges.
Three things stood out:
- Style rules fix the surface, not the substance. House rules removed em dashes from many drafts, but repetition and other editorial findings remained. They did not stop the invented commitments: "the team is executing well", "confirm by Friday", "ready to launch immediately", "no legal blockers".
- Same price, very different output. Claude Fable 5.1 costs what Astra costs and invented a prior discussion, a Friday deadline and a ready team on briefs where Astra invented nothing.
- Even the winner needs a style pass zero-shot. Half of Astra's original plain drafts were not ready as is. No remaining scored factual defect was identified. They had em dashes.
Please Make This a Real Benchmark
The ranking uses one primary attempt per model and brief; repeat runs are reported separately. The grader was an AI assistant that could see the model names, with every verdict anchored to a quoted line so it can be checked. I wrote the briefs, the rubric and the house rules. Eight short briefs. A gap of one or two checks between models is noise.
A real benchmark needs many more briefs, blind grading by people who do not work for a model vendor, several generations per brief, and published agreement between graders. Business writing is where most people meet these models, and there is nothing pressuring the labs to get better at it. The code is MIT and the rubric is CC BY-SA. Take whatever is useful.
As always feel free to connect with me on LinkedIn or follow me on X.