From strategy to dispute resolution, the next generation of AI will be defined by its ability to challenge us, not charm us, says Resolutiion founder Fayola-Maria Jack.
As many will know, human feedback is quite often used to fine-tune and train AI assistants and large language models (LLMs) such as ChatGPT. Otherwise known as reinforcement learning from human feedback (RLHF), it’s a common method used to adjust the AI machine’s ‘policy’, which is essentially the way the AI decides what to output. Over time, the idea is that it learns to prefer responses that reflect human judgement, making it more useful and aligned with expectations.
However, this method comes with downsides, one of the main ones being sycophancy – where human feedback can encourage model responses that match user beliefs over truthful ones. In fact, according to recent research, there’s a growing realisation among enterprises that AI tells you what it believes you want to hear, with convincingly written sycophantic responses outperforming correct ones some of the time.
This particular downside has attracted much of the early criticism of AI tools – from leading businesses into strategic blind spots and masking operational and financial risks, to a workforce and strategy shaped more by reassurance than reality.
Take the example of two parties in a disagreement. Each tends to have a self-consistent but conflicting narrative. If the AI affirms both sides without challenge, it effectively validates incompatible truths. This can create an illusion of fairness (‘the AI hears me’), but in reality it cements division, since neither party is nudged towards recognising the other’s perspective or underlying shared interests. This kind of ‘both-sidesing’ may feel safe for the AI but can reproduce systemic inequities. Neutrality does not mean equidistance; true neutrality means objectivity and shared understanding.
Sycophancy also isn’t always about agreement or accuracy, but tone. In this case, a model may echo a user’s sense of justification (‘You’re right to feel that way’), which can be read as siding even if it introduces no factual errors. Likewise, to appear balanced, AI might validate both parties equally in a disagreement (‘You both make strong points’), but this creates a false equivalence when one side’s position may be factually incorrect. The simple act of over-validating one party’s emotions, or adopting a more sympathetic tone, can make the system feel partial.
Retraining AI as a critical tool
However, rather than avoiding AI completely with the above challenges in mind, many businesses are instead recognising that managing disagreement constructively is far more valuable.
What’s more, the same mechanisms that cause AI to flatter can, if retrained, make it extraordinarily good at structured disagreement and critical evaluation, leading to a move towards next-gen AI tools engineered to resist sycophancy.
Conflict resolution offers the clearest proof point of this shift. Rather than being where sycophancy is most dangerous, it’s where AI’s potential to support fairness and neutrality can be most transformative. With the latest advances in AI, specialist models can in fact be fine-tuned for particular contexts, data sources and goals, allowing them to be optimised for:
- Neutral stance-taking rather than affirmation. Instead of overfitting to user preferences, the model learns to prioritise professional norms (eg impartiality, fairness and progress).
- Structured dialogue navigation, where clarifying differing viewpoints, reframing narratives and surfacing shared ground are the priority.
- Domain-specific ethical alignment with mediation best practices. So, if a model does lean towards one side (eg correcting a factual inaccuracy), it is designed to explain why. This prevents the perception of hidden bias and frames any correction as part of the resolution process.
- Resolution progress – a very different metric to general models – which sees the system rewarded not for making the user feel validated, but for moving the dispute forward in a fair and balanced way.
- Undergoing continuous evaluation in live or simulated disputes. This ensures sycophancy is not only curbed at training but also monitored in deployment, since flattery tendencies can re-emerge under real-world emotional pressure.
Predicted shift in AI model use
The way businesses respond to the growing presence of AI sycophancy is still playing out. However, far from an outright rejection of AI or a slowdown in adoption, a shift from general-purpose chatbots which are optimised for ‘helpfulness’ in casual Q&A to specialist models is the best way for businesses to retain the benefits of AI.
Many organisations may still use the general consumer LLMs. That is absurd if you understand sycophancy. Forward-thinking leaders will lean towards a stronger segmentation, with specialist AI for sensitive domains, adopted precisely because they are engineered to avoid the pitfalls of sycophancy.
Fayola-Maria Jack is a specialist in complex commercial transactions and dispute resolution, and the founder of Resolutiion, an AI-powered system for managing conflict. An accomplished thought leader with an MBA from UCL and doctoral research in dispute resolution and behavioural science, she now drives Resolutiion’s success as a solo female founder.
Don’t miss out on the knowledge you need to succeed. Sign up for the Daily Brief, Silicon Republic’s digest of need-to-know sci-tech news.
