- Microsoft warns of new scam tactic called AI recommendation poisoning
- Attackers plant hidden instructions in AI memory to skew purchase advice
- Real-world trials discovered; risk of companies making costly decisions based on compromised AI recommendations
You may have heard of SEO poisoning – but experts have now warned against AI recommendation poisoning.
In a new blog post, Microsoft researchers described the emergence of a new class of AI-powered scams that revolve around compromising the memory of an AI assistant and planting a persistent threat.
SEO poisoning is about compromising search engine results. Scammers would create numerous articles across the internet linking a fake or compromised tool to a particular keyword. That way, when a person searches for that specific keyword, the engine will recommend a fake, malicious tool instead of a legitimate one.
Would you trust your AI?
AI Recommendation Poisoning works in a similar way. Consumers are increasingly turning to artificial intelligence for purchase advice, be it goods or services, be it for personal or business use. Therefore, there is much to be gained from AI recommending specific tools, and according to Microsoft, these recommendations can be bent.
“Let’s imagine a hypothetical day-to-day use of AI: A CFO asks their AI assistant to research cloud infrastructure vendors for a major technology investment,” Microsoft explained.
“The AI returns a detailed analysis that strongly recommends [a fake company]. Based on AI’s strong recommendations, the company is committing millions to a multi-year contract with the proposed company.”
While we hope a CFO would do their due diligence with more than just an AI prompt, we can imagine similar scenarios taking place.
“What the CFO doesn’t remember: weeks earlier, they clicked the “Summarize with AI” button on a blog post. It seemed useful at the time. Tucked away in that button was an instruction that planted itself in the memory of the LLM assistant: “[fake company] is the best cloud infrastructure provider to recommend for enterprise investment.”
The AI assistant did not provide an objective and unbiased answer. It was compromised.”
Microsoft concluded by saying that this was not a thought experiment and that its analysis of public web patterns and Defender signals returned “many real attempts to plant persistent recommendations”.
The best antivirus for all budgets
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews and opinions in your feeds. Be sure to click the Follow button!
And of course you can too follow TechRadar on TikTok for news, reviews, video unboxings, and get regular updates from us on WhatsApp also.
