Discussion about this post

User's avatar
Lucas Wiman's avatar

> It writes code that is slightly worse to look at than Fable, but more likely to be correct. It regularly is catching things that Fable missed.

What's interesting to me is how thoroughly different it seems to be to different people. My experience with Opus 5 was that it made mistake after mistake, failed to read the code. It "lied" repeatedly (said things which were obviously false given the context). Incredibly frustrating. I'd say it's worse than Opus 4.5 _for me_ at coding. But many people found it a big improvement or at least usable. I've seen enough twitter reaction threads and slack discussions at my job that I'm convinced both of these are valid experiences which actually happen.

Anthropic's prompting guide is bizarre, including optional instructions "To limit correction narration to corrections that matter" and stuff that seems like it should be in the system prompt. They indicate that you should minimize custom prompting because it can degrade performance, which seems like a strong indication of overfitting to a particular environment (presumably the default claude code harness with no CLAUDE.md and prompts written by Fable). An actually more general intelligence might not need custom prompting, but it shouldn't be vexed by it and do a terrible job.

My guess is that they've outsourced testing of lower-tier models to Mythos/Fable and are losing the plot on generating useful models for users.

Drew's avatar
3dEdited

The Opus 5 "vibes are off" for me too, but for different reasons than mentioned above.

I've been using the Opus 5 a lot for knowledge work (I'm not really a coder), and it feels like a bit of a regression compared to 4.8.

Opus 5 needs more handholding, quicker to jump to (incorrect) conclusions, less capable.

Honestly feels like we're back to 4.5 levels of thoughtfulness where I need to once again ask the agent to review its work/recommendations and do "meta-cognition" more often, something which 4.6-though-4.8 had improved upon.

Reminds me also of the ChatGPT family of models, which -- while very capable -- have a hard time picking their head up and looking around to see if what they're doing so voraciously makes sense in context.

If I was a conspiratorial type of person, I'd say they've nerfect Opus 5 so that more people rely on Fable (and pay for more Fable usage). I don't think that's true, but my inner Agent Mulder "wants to believe."

16 more comments...

No posts

Ready for more?