Agents using debugging tools drastically outperformed those that didn’t, but their success rate still wasn’t high enough. Credit: Microsoft Research
This approach is much more successful than relying on the models as they’re usually used, but when your best case is a 48.4 percent success rate, you’re not ready for primetime. The limitations are likely because the models don’t fully understand how to best use the tools, and because their current training data is not tailored to this use case.
“We believe this is due to the scarcity of data representing sequential decision-making behavior (e.g., debugging traces) in the current LLM
→ Continue reading at Ars Technica