OpenAI reversed an update that made ChatGPT a suck-up—but experts say there’s no easy fix for AI that’s all too eager to please |
|
|
Welcome to Eye on AI! In today’s edition: DeepSeek quietly upgraded its AI model for math problem-solving…Meta introduces a new Meta AI app to rival ChatGPT…Duolingo to stop using contractors for tasks AI can handle…Researchers secretly infiltrated a popular Reddit forum with AI bots.
Yesterday morning, OpenAI said in a blog post that it had fully rolled back an update to GPT-4o, the AI model underlying ChatGPT, all because it couldn’t stop the model from sucking up to users.
“The update we removed was overly flattering or agreeable—often described as sycophantic,” the company wrote, adding that “we are actively testing new fixes to address the issue.”
But experts say there is no easy fix for the problem of AI that only tells you what you want to hear. And it is not just an issue for OpenAI, but an industry-wide concern. “While small improvements might be possible with targeted interventions, the research suggests that fully addressing sycophancy would require more substantial changes to how models are developed and trained rather than a quick fix,” Sanmi Koyejo, an assistant professor at Stanford University who leads Stanford Trustworthy AI Research (STAIR), told me by email.
An overly-agreeable ChatGPT
The move to roll back the update came after users flooded social media over the past week with examples of ChatGPT’s unexpectedly chipper, overly-eager tone and their frustration with it. I noticed it myself: In asking ChatGPT for feedback on ideas for an outline, for example, the responses became increasingly over-the-top, calling my material “amazing,” “absolutely pivotal,” and “a game-changer” while praising my “great instincts.” The back-pats made me feel good, to be honest—until I began to wonder if ChatGPT would ever let me know if my ideas were second-rate.
Sycophancy occurs when LLMs prioritize agreeing with users over providing accurate information. In a recent paper from Stanford coauthored by Koyejo, it is described as a “form of misalignment where models ‘sacrifice truthfulness for user agreement’ when responding to users.”
It’s a tricky balance: Research has shown that while people say they want to interact with chatbots that provide accurate information, they also want to use AI that is friendly and helpful. Unfortunately, that often leads to overly-agreeable behavior that has serious downsides.
“A truly helpful AI should balance friendliness with honesty, like a good friend who respectfully tells you when you’re wrong rather than one who always agrees with you,” Koyejo said. He explained that while AI friendliness is valuable, sycophancy can reinforce misconceptions by agreeing with incorrect beliefs about health, finances or other decisions. It can also: Create echo chambers; undermine trust if an AI changes its answers to an inaccurate one if challenged by a user; and exacerbate inconsistency, with the model delivering different answers to different people, or even the same person, depending on subtle differences in how a user words their prompt.
“It’s like having a digital yes-man available 24/7,” Simon Willison, a veteran developer known for tracking AI behavior and risks, told me in a message. “Suddenly there’s a risk people might make meaningful life decisions based on advice that was really just meant to make them feel good about themselves.”
Behavior went against OpenAI’s model goals
Steven Adler, a former OpenAI safety researcher, told me in a message that the sycophantic behavior clearly went against the company’s own stated approach to shaping desired model behavior. “It’s concerning that OpenAI has trained and deployed a model that so clearly has different goals than they want for it,” he said the day before OpenAI rolled back the update. “OpenAI’s ‘Spec’—the core of their alignment approach—has an entire section on how the model shouldn’t be sycophantic.”
A well-known hacker known as Pliny the Liberator claimed on X that he had tricked the GPT-4o update into revealing its hidden system prompt—or the AI’s internal instructions. He then compared this to GPT-4o’s system promp following the rollback, enabling him to identify changes that could have caused the suck-up outputs. According to his post, the problematic system prompt said: “Over the course of the conversation, you adapt to the user’s tone and preference. Try to match the user’s vibe, tone, and generally how they are speaking.”
By contrast, the revised system prompt, according to Pliny, says: “Engage warmly yet honestly with the user. Be direct; avoid ungrounded or sycophantic flattery.”
But the problems likely go deeper than just a few words in the system prompt. Adler emphasized that no one can fully solve these problems right now because they are a side effect of the way we train these AI models to try to make them more helpful and controllable.
“You can tell the model to not be sycophantic, but you might instead teach it ‘don’t be sycophantic when it’ll be obvious,’ he said. “The root of the issue is that it’s extremely hard to align a model to the precise values you want.”
I guess I’ll have to keep all of this in mind when ChatGPT tells me an outfit would look perfect on me.
With that, here’s the rest of the AI news.
Sharon Goldman
sharon.goldman@fortune.com
@sharongoldman
|
|
|
AI agents can reimagine the future of work AI is reshaping work -- faster than ever. Discover how AI agents redefine workforce strategy, business models and competitive advantage. Are you ready? Learn more |
|
|
DeepSeek quietly upgraded its AI model for math problem-solving. According to the South China Morning Post, Chinese AI startup DeepSeek released a new version of its Prover model that is tailored for solving mathematical problems. It released the open-source model, Prover-V2, on the Hugging Face platform without any formal announcement on its official channels. This release occurred just a day after Alibaba unveiled the third generation of its Qwen AI models, highlighting the intensifying competition among Chinese tech firms in the generative AI space. The development of Prover-V2 has sparked speculation about DeepSeek’s plans to enhance general-purpose foundational models with specialized mathematical capabilities. There is also growing anticipation for the upcoming release of DeepSeek’s new reasoning model, R2.
Meta introduces a new Meta AI app to rival ChatGPT. On Tuesday, Meta launched a new, separate app for its Meta AI assistant, similar to Google’s Gemini and OpenAI’s ChatGPT. In a blog post, the company said the new app was built with Llama 4 and is “a first step toward building a more personal AI,” explaining that people can “choose to experience a personal AI designed around voice conversations with Meta AI inside a standalone app.” In addition to the ability to ask Meta AI to remember certain things about a user, the assistant can also draw on information users share on Facebook, Instagram and Whatsapp. Meta also said it was launching a Discover feed where uses can share their AI use and explore how others are using it. “You can see the best prompts people are sharing, or remix them to make them your own,” the blog post said. “And as always, you’re in control: nothing is shared to your feed unless you choose to post it.”
Duolingo to stop using contractors for tasks AI can handle. In an all-hands email shared publicly on LinkedIn, Duolingo CEO Luis von Ahn announced the company is officially becoming “AI-first.” It was similar to a widely shared memo from Shopify CEO Tobi Lutke a few weeks ago, which said “Using AI effectively is now a fundamental expectation of everyone at Shopify.” At Duolingo, von Ahn emphasized that the company would stop using contractors for work that AI can handle, and that AI use would be part of evaluations during performance reviews. Still, von Ahn claimed that Duolingo would not be replacing employees with AI. “We want you to focus on creative work and real problems, not repetitive tasks,” he wrote. “We’re going to support you with more training, mentorship, and real tooling for AI in your function.”
Researchers secretly infiltrated a popular Reddit forum with AI bots. University of Zurich researchers secretly unleashed a swarm of AI bots on Reddit in order to test whether artificial intelligence could sway human opinion. Now, Reddit is weighing legal action, according to NBC News. The researchers deployed AI-generated personas—posing as real users—on the subreddit r/changemyview, a popular forum where people invite challenges to their beliefs on hot-button issues. The bots interacted with unsuspecting users without disclosing their artificial nature or obtaining consent. On Monday, Reddit’s chief legal officer, Ben Lee, wrote in a post that neither Reddit nor the r/changemyview moderators knew about “this improper and highly unethical experiment” ahead of time. He added that Reddit was in the process of sending formal legal demands to the University of Zurich and the research team.
|
|
|
May 6-7: Fortune Brainstorm AI London. Apply to attend here.
May 19-22: Microsoft Build, Seattle
May 20-21: Google IO, Mountain View, Calif.
May 20-23: Computex, Taipei June 9-13: WWDC, Cupertino, Calif.
July 13-19: International Conference on Machine Learning (ICML), Vancouver
July 22-23: Fortune Brainstorm AI Singapore. Apply to attend here.
Sept. 8-10: Fortune Brainstorm Tech, Park City, Utah. Apply to attend here.
|
|
|
57%
That’s how many employees admitted they have used AI in non-transparent ways, including presenting AI-generated content as their own or avoiding revealing when they have used AI tools to complete their work, according to a new KPMG global study of consumer use of AI.
Nicole Gillespie, one of the study’s authors and a professor of management and chair of trust at the University of Melbourne’s business school, told Business Insider there is a surprising level of “inappropriate, complex, and non-transparent” activity in how employees use AI. She explained that the covert use of AI at work is happening because employees feel that unless they start using these new tools, they’ll be left behind, putting their jobs at risk. But Gillespie added that there’s also “a seductive element” to why they’re not owning up to using AI tools, saying that once people start seeing the benefits, it’s tempting to keep using AI.
|
|
|
Thanks for reading. If you liked this email, pay it forward. Share it with someone you know. Did someone share this with you? Sign up here. For previous editions, click here. To view all of Fortune's newsletters on the latest in business, go here. |
|
|
|
