There is a quiet risk emerging in businesses that are embracing AI with enthusiasm and speed. The evolution of AI Joyriding you could say.
False Confidence is not data security, nor is it job displacement or even hallucinations.
Across marketing teams, operational departments and leadership groups, AI tools are drafting strategies, writing proposals, analysing markets and shaping corporate messaging. The outputs are often impressive. They are polished, well-structured and persuasive, and they sound intelligent.
And that is precisely where the risk lies. When AI produces coherent and well-argued content in seconds, it can create the illusion of depth and critical thinking. Teams begin to feel that the thinking is theirs because they wrote the prompt. The language feels familiar because they refined it. The strategy appears robust because it reads convincingly. But reading well is not the same as thinking well and the ability to shape strategy with knowledge, human-first knowledge.
AI is exceptionally good at recognising patterns and amalgamating information. It draws on vast amounts of material that is publicly available in order to generate plausible responses. What it does not hold is accountability, commercial responsibility or lived experience in the way humans do. That’s our superpower.
If teams rely heavily on AI to generate reports, insights or recommendations without properly interrogating them, something subtle happens, and we are seeing this in abundance. Outputs are accepted more quickly. Assumptions are left unchallenged, and in turn, the nuance of a business that makes them unique (and there isn’t much uniqueness anymore) is reduced or even destroyed. Context is lightly skimmed rather than deeply understood, and this is what being an expert in your field is about. It’s why your customers trust you, both clients and candidates alike.
Over time, this creates a confidence gap, and it’s very interesting to observe. Individuals begin to feel more strategically capable because the work they are presenting sounds strategic. Yet the cognitive strain that develops genuine expertise may not have taken place internally, and it certainly becomes clear in meetings where these gaps and cracks appear. It is a little like using autocorrect for complex reasoning. The sentence appears refined, but the intellectual effort has been borrowed.
I am firmly pro AI, but with a huge dose of caution and robust guardrails that protect your business and your company values. At Marmalade, we use it deliberately and transparently for ideation and to improve efficiency, such as our AI Assistant that transcribes all client calls to capture notes and actions. But we are careful never to confuse generation with judgement and we are a human-first business that has expanded with the onset of AI, we have certainly not replaced humans with ‘the machine’.
The real danger of false confidence becomes visible in decision-making. Leadership teams can begin to make commercial choices based on AI-generated analysis that feels data-informed. In reality, it may simply be an output from ChatGPT.
There is a huge difference here. AI can summarise a market beautifully. It can surface trends that you didn’t see. It can suggest positioning opportunities, who wouldn’t love that? But what it cannot do is understand the subtleties of your client relationships, the history of your brand reputation, or the political dynamics within your organisation. It cannot sense risk in the way a seasoned leader can. A seasoned leader is spotting this change in behaviour a mile off, and I personally am having many conversations with senior leaders about the challenges this is presenting.
Without rigorous human examination layered on top, businesses may act on outputs that are coherent but not contextual. The organisation feels confident because the document sounds confident. Yet confidence built on untested assumptions cannot be trusted.
There is also a longer-term concern that is less visible but equally important. Capability can quietly erode.
If teams consistently outsource first draft thinking to AI, their own strategic development weakens. The ability to structure an argument from scratch, to work through ambiguity, to interrogate data deeply and to challenge a brief thoroughly just doesn’t happen. The more seamless the tool becomes, the easier it is to bypass the intellectual discomfort that builds expertise. Intellectual discomfort is what makes good leaders great. Every day is a school day and all that.
The irony is that AI should expand intellectual capability, not diminish it. Used well, it acts as a business coach, but that’s it. It offers alternative perspectives which can be valuable and it really does help with the headache of research, but it exposes blind spots and the human must still do the heavy lifting of judgement to refine solutions for customers. This is your IP, and why you do what you do.
We love AI as a tool for accelerated learning because it increases exposure to information. It makes it easier to find stuff out, but of course, every fact and stat must be sanity checked for validity.
The distinction that matters is intent. Overuse of AI masks the absence of critical thinking, and this is vital in any leadership team.
In our experience working with some of the most progressive brands in the industry, is not to retreat from AI. It is to govern its use properly. Outputs must be interrogated, and it is vital that assumptions are questioned. Context must be layered in by experts within your own business, people who hold responsibility for strategic decisions, because these are the people who are accountable.
If you’re not comfortable sharing with your customers where thinking originates in your business, there lies the problem. If AI has shaped a recommendation or created a strategy, acknowledge it because this safeguards your business.
True confidence in business comes from disciplined reasoning, commercial judgement and experience tested over time. AI can support that process, but in our experience, it cannot substitute it.
The goal is not to sound intelligent. The goal is to be intelligent and this is why clients and candidates trust your business. Borrowed intelligence (is what we’ve started calling it) can feel cool in the moment. Owned understanding is what sustains performance in the long term and it’s your IP!
If we want AI to elevate our organisations rather than quietly weaken them, we must treat it as a collaborator. It should stretch our critical thinking, not replace it.
Confidence should be earned, not generated.