Anthropic conducted research to measure the persuasiveness of AI language models compared to humans. They developed a method to quantify persuasiveness by having people rate their agreement with claims before and after reading arguments written by humans or AI models. The study found that Anthropic’s latest AI model, Claude 3 Opus, generates arguments that are statistically indistinguishable in persuasiveness from human-written arguments. Additionally, there was a clear scaling trend where more advanced AI models produced more persuasive arguments than earlier generations. The researchers discuss challenges in studying persuasion, limitations of their approach, ethical considerations around the potential misuse of persuasive AI, and plans for future work exploring interactive dialogue settings and real-world impacts beyond stated opinions.

Summarized by Claude 3 Sonnet