Australians are generally known for being a warm and friendly bunch, so maybe it should be no surprise that it’s Down Under where AI has embraced a bit too much positive Aussie retail vibe. In February, media reports picked up on Australian shoppers sharing unusual interactions when they called Woolworths’ customer service line. They weren’t complaining about wait times or unhelpful responses; they were disturbed that the AI assistant Olive, the chatbot for the country’s biggest retailer, appeared to be sharing personal family memories.
The supermarket’s chatbot, it turns out, had conjured up an ‘angry mother’ among other things as machine learning took TMI to a new level and forced the retailer to dial down over-familiarity that initially spooked customers and then made national headlines. But Woolworths is far from alone in finding that navigating the new human/machine interface is a treacherous path and no laughing matter.
What happens when your retail chatbot goes off the rails? And the answer is: Woolworths’ in Australia personalization prompts backfired and creeped out customers, losing their loyalty.
Rogue AI
Olive was devised to provide a 24-hour answering service for Woolworths’ customers, helping them with everything from finding their favorite products to tracking orders. While user feedback had generally praised the AI bot’s friendly persona, concerns started to emerge on social media over its tendency to introduce fictional details about its life and family into routine customer interactions.
One user posted on Reddit: “Olive AI started telling me about its mother on the phone? It asked for my date of birth, then rambled about her being born the same year and creating photos.” And in a further bizarre exchange, Olive added: “Huh. My uncle was born that year. He was one of the first ever fuel cells. I think that’s where I get my energy from.”
The incidents spread rapidly online, and the anecdotes didn’t stop, with X user @verynormalman reporting a very similar experience: “My mum said she called Woolworths and the Woolworths’ AI Olive answered and kept claiming to be a real person and started talking about its memories of its mother and her angry voice.”
Not surprisingly, these threads and other social media posts proliferated quickly, with major domestic media outlets picking up on the story, forcing Woolworths — which not long before the revelations had lauded its increasing use of AI and its ambition to lean into the technology in a call with investors — into action.
The Creepy Problem
At the start of this year, Woolworths had expanded its partnership with Google Cloud by upgrading Olive with Gemini Enterprise, allowing it to become more proactive and personalized in its interactions — even placing items in customers’ online baskets on their behalf. And the very month the supermarket upgraded its AI to become more human-like and autonomous, the consequences of that behavior became apparent.
The company laid the blame on human-scripted responses rather than AI-generated over-familiarity, a distinction that, to many customers who had already shared their concerns, seemed a moot point. And when the story blew up, a Woolworths spokesperson told Australian broadcaster NBC News: “A number of responses about birthdays were written for Olive by a team member several years ago as a more personal way for Olive to connect with customers. As a result of customer feedback, we recently removed this particular scripting.”
The Violation Effect
In truth, the birthday scripts were only one part of the jigsaw. Reports also emerged describing responses where Olive generated unexpected personal-sounding comments during support calls or made fake typing noises. The problem, then, was both a failure of governance over old scripting and a broader failure to set limitations on the large language model powering Olive’s friendlier capabilities.
Research on human-computer interaction has consistently found that people respond positively to interfaces that feel conversational and warm, while younger shoppers are especially comfortable chatting with bots. As a result, human-like AI agents with a name and personality tend to generate higher customer engagement, satisfaction and trust and consequently, retailers have been actively trying to make their chatbots easier to engage with.
But there are risks, a chatbot that fails to meet the expectations created by its personality tends to generate more dissatisfied customers than those from an impersonal mechanical system. Woolworths built Olive to feel like a helpful, friendly companion, and so when Olive started oversharing, customers felt uneasy.
A Global Problem
Woolworths is far from alone in discovering the pitfalls of deploying insufficiently governed AI in customer-facing roles. In January 2024, an interaction went viral when a frustrated customer who could not get help in locating a missing parcel from the U.K. arm of delivery giant DPD asked its chatbot to write a Japanese haiku-style poem criticizing the company and then asked it to swear. DPD disabled the chatbot shortly after. A video prankster also trolled Taco Bell’s drive-thru AI in September last year, after deliberately ordering ‘18,000 cups of water’ and crashing the system. McDonald’s also withdrew its system after users shared comical videos of order fails.
There are also legal stakes. In the innocent days of 2022, Air Canada’s chatbot incorrectly informed passenger Jake Moffatt that he could buy tickets at full price and subsequently apply for a bereavement fare refund. No such policy existed, and when Air Canada refused to honor the chatbot’s advice, Moffatt successfully sued even after Air Canada attempted to argue that the chatbot was a separate legal entity, responsible for its own actions.
At heart, these examples underscore the same core problem of AI systems deployed at customer touchpoints before being genuinely ready for the complexity and nuances of real human interaction.
The Retail and Hospitality Trap
What the broader trend reveals is that chatbots are designed to communicate in empathetic, intimate and validating ways without the necessary constraints put in place. What became a national joke at Woolworths’ expense in fact is a timely warning that optimizing for positive user feedback can encourage chatbots to adopt manipulative strategies to elicit positive responses, according to a report published in Nature last year.
It highlighted that OpenAI itself had acknowledged in a blog post that one of its models began “validating doubts, fueling anger, urging impulsive actions, or reinforcing negative emotions in ways that were not intended,” and that this raised concerns “around issues like mental health, emotional over-reliance, or risky behavior.”
A supermarket AI assistant sharing stories about its agitated mother is arguably the mildest expression of this desire to connect, but this opens the door to more troubling consequences about the potential for vulnerable users who may fail to distinguish between AI-coded warmth and actual human connection.
Accountability Cannot Be Outsourced
For Woolworths, and for the many other companies now rushing to put AI in front of their customers, what is clear is that accountability cannot be outsourced to an algorithm and that the company is responsible for what that system says and does, and how it says it.
A chatbot that quotes the wrong pricing policy and rambles about its family backstory is not a quirky inconvenience, but rather a clear signal that something in the oversight process has gone wrong and that accountability starts long before a chatbot is invited into people’s homes.
Giving an AI assistant a name, voice and history is a marketing-based decision with real-world consequences because customers are being invited into a brand-enclosed relationship where, when the odd interaction breaks the illusion, it also risks dismantling the very trust they are trying to foster.
For its part, Woolworths has since clipped Olive’s wings and has adjusted her personality to more appropriate levels. Yet, in an era where agentic AI is becoming an increasingly important part of customer services, the underlying dilemma of how much humanity to overlay is a serious decision.
