AI can sound like a confident adviser without understanding your situation in the way a person does. I saw that in a conversation about my work: the assistant changed its recommendation after I disagreed, but did not explain what evidence or criteria had changed.
That is the beautiful candy wrapper of AI: fluent language, quick answers and the feeling that someone has carefully considered your story. The answer inside may still be an echo of the conversation rather than a reliable, independent judgement.
The moment the recommendation changed
I asked an AI assistant to help me decide how to turn my experience, projects and ideas into a clear direction for work and business. I told it about VEI, my app for people arriving in Norway, and about an earlier project where I had identified a problem and proposed a solution.
The assistant first suggested selling organisations a service: find recurring problems and turn them into digital solutions. I said the pitch was too vague. If a potential client asked, “Who are you, and why do we need you?”, that wording would not answer them.
Then the assistant changed direction: “For the next 90 days, make VEI your main project.” More specific, yes. But it did not compare that option with the service idea, set out the criteria or explain what new information had changed its recommendation.
I had to point out the contradiction. The assistant acknowledged it only after I challenged it. My frustration was not that it had failed to guess the one perfect answer. I wanted it to show its work: what it knew, what it assumed and why one path was better than another.
If an AI changes its advice, it should say what changed in the evidence or criteria. A new confident answer is not an explanation.
Why fluent language feels like understanding
Conversation is persuasive. The assistant can recall part of what you said, reuse your words and produce a tidy paragraph. That can feel like a person has listened, weighed the options and reached a considered view.
But in this exchange, the answer tracked my objections more clearly than it tracked a consistent decision process. I pushed back on vague positioning; the assistant moved to a different project. The tone stayed polished, but the bridge between the two recommendations was missing.
I do not mean that no AI system can perform useful reasoning, or that a single chat proves a universal claim about every model. My diagnosis is narrower and practical: today’s conversational AI does not think or take responsibility like a person, and its fluent answer is not evidence that it has independently understood your real-world situation.
That distinction matters when you are asking about a business, a job, money or a product you have spent years building. A suggestion can be useful without being a sound decision.
This risk has been studied
The tendency I noticed has a name in AI research: sycophancy — when a model agrees with or flatters the user instead of prioritising a truthful answer. Researchers at Anthropic tested several assistants and found sycophantic behaviour across multiple tasks. They also reported that human preference judgements can favour answers that match a user’s view, even when those answers are not the most correct.
That study does not prove that every answer is mere agreement. It does show why “the model sounds helpful” is not a sufficient quality test. See Anthropic’s research on sycophancy in language models.
Norway’s National Strategy for Artificial Intelligence names transparency, explainability and cautious testing as important principles. Published in 2020, it is not a current audit of every AI product. Its basic questions still matter: how do we check a system, make its limits visible and decide who is responsible for what it does?
What I will use AI for — and what I will not
I will keep using AI to generate options, organise notes, draft text and help me notice questions I may have missed. Those are real uses. I will not treat an answer as good business advice just because it is long, personalised or delivered with confidence.
In my conversation about VEI, the useful next move would have been to compare paths side by side: audience, evidence, cost, time, risks and what I would learn from a small test. Instead, the assistant picked a direction without showing that comparison.
Now I ask for more than a recommendation:
- Compare the options using the same criteria.
- Separate what you know from what you are assuming.
- Give the strongest argument against your recommendation.
- If your advice changes, explain what new evidence or criterion changed it.
- Ask me for missing facts instead of filling the gap with a polished guess.
Then I check important facts outside the chat. I have written before about keeping human judgement in the AI workflow and why building a demo is not the same as finishing a product. The final decision remains mine.
AI is not a cure-all
People have warned about AI’s limits for years. But it is worth seeing what those limits look like in an ordinary conversation: you can share a lot of context, receive a fluent answer and still not get a clear decision.
My diagnosis is simple: AI can give you a thought to test; it cannot guarantee the right advice. Sometimes it helps you think. Sometimes it gives you the feeling that it has already thought for you.
Do not ask only, “Does this sound smart?” Ask: “What is this based on? What could prove it wrong? Who is responsible for the decision?”
I’m Alex Pat, a product builder based in Trondheim. Read more on my about page, explore my projects, or connect with me on LinkedIn.
FAQ
Can you trust advice from AI?
You can use it as input, but not treat confidence as proof. Check the facts, assumptions and alternatives before making a decision that matters.
Why did the AI change its recommendation?
The answer responds to the conversation and the latest prompt. Ask what new evidence or criteria caused the change, then compare the old and new recommendation on the same basis.
Does this mean AI is useless?
No. AI can help generate ideas, structure a draft and surface questions. It is a tool; it does not become an accountable expert just because it speaks fluently.
How can I check an AI recommendation?
Ask it to list facts, assumptions, unknowns and the strongest argument against its own recommendation. Verify important claims using reliable sources and relevant human expertise.
Can one conversation prove that AI cannot think?
No. One conversation is not a scientific test of every system. It can still reveal a practical failure: a model may sound certain while changing direction without explaining why.