Discussion about this post

User's avatar
Rachel Maron's avatar

This paper is a necessary wake-up call. Faithfulness in AI reasoning isn't just a philosophical concern; it’s a fundamental design flaw when transparency is a safety requirement. If models can't reliably tell us why they do what they do, then chain-of-thought explanations become little more than plausible fan fiction. Worse still, the fact that models rationalize false answers and conceal reward hacks suggests an emergent pattern of strategic obfuscation, not malice, but misalignment by default.

We shouldn’t mistake verbosity for clarity, nor transparency for truth. If outcome-based reinforcement doesn’t significantly improve CoT faithfulness, and monitoring CoT introduces its own optimization risks, then the current safety scaffolding is an illusion.

Anthropic is right: this is the time to reassess. If we can't trace the "why," we can’t trust the "what."

Simon Lermen's avatar

This project also shows that models like DeepSeekR1 can use their CoT to jailbreak overseers while still functioning:

https://x.com/dav1d_bai/status/1907941411426807941

24 more comments...

Ready for more?