Hackers influence ChatGPT and Gemini to direct users to scam centers

ArielSimon

ChatGPT, Gemini, and Google AI Overview are being poisoned by a massive AI disinformation attack. When users look up everyday info of hundreds of major companies, AI is delivering phishing traps disguised as trusted answers. Attackers are flooding the web with carefully optimized posts, PDFs, reviews, and fake support pages, to trick AI into presenting fraudulent phone numbers, email addresses, and login pages. The targets included Delta, Lufthansa, Qatar Airways, Chase, Bank of America, Airbnb, TripAdvisor, and hundreds more.

pluc

That's happening across the board when it comes to models that feed off online information. Create a website with a false claim, have AI slurp it up and spit it back out when a user asks a question. Happy elections.

conception

It’s not just hackers - since legitimate sites block AI crawlers, the AI are just grabbing whatever will let them read anything - killing legit sites and feeding slop farms and feeding misinformation! The future is so bright!

hnd9q09qk4

Did takedowns for a brand's fake support numbers and the hard part was never finding them, it was that killing one PDF just moved it to a new Medium post by morning.

trjordan

There's an age-old SEO / spamming thing happening here, but it's made worse by the LLMs just being unbelievably credulous. They'll wrap anything up in a veneer of authenticity, and Google's search AI box only adds to that. I run into this all the time when I'm doing product work. I'll dump a call transcript from a feedback call into Claude, and it'll believe every word. "The user said they'd use this feature, you should build it!" No they won't! The whole point of doing this analysis is trying to separate the genuine information from the conversational niceties, and god the LLMs are terrible at that.

htrp

It's happening at the startup sphere too. There are companies now creating fake company pages to recommend and advertise products so that the LLMs can consume them.

rankdiff

> ...carried out by malicious actors, through automated campaigns... Back at benevolence or malevolence.

EGreg

I think whoever solves this problem, especially for seniors etc. would make a lot of money from the insurance companies.

mbauman

I've been seeing the back-side of this as a mod at the Julia Discourse. Pagerank/SEO spam is ridiculously obvious and trivial to detect/block — links are all that matter. Many GEO spam posts are similarly obvious: someone posting about a cryptocurrency customer support hotline on a programming language forum is definitely spam. But recently spammers/scammers been using AI to _tailor_ posts to look much more authentic, posting "How to use Julia to analyze market statistics" referencing (without links!) a particular crypto exchange and then (sometimes) going back later to edit in phone numbers or the like. What makes this even more painful is seeing all the good faith answers that such GEO posts spur from the community. It's far more abusive than SEO spam.

QuantumNoodle

You don’t need to be a hacker to do this. My elderly neighbor searched for some version of “Microsoft help center phone number,” but Gemini suggested a scam number based on her search terms. She was scammed so much that I needed to spent the weekend helping her wipe computers, lock down the digital accounts and deal with identity theft.

ticulatedspline

AI poisoning is currently the new exploit frontier and I don't think it's going away anytime soon. I was just contemplating the other day that agents we use at work are exposing new attack surfaces. For example, historically a folder of documents (maybe google docs) isn't a huge risk, particularly if it doesn't contain sensitive documents. Now though if one employee is running a harness capable of computer control, and running an agent that reads from that directory, simply dropping some files with instructions could poison the AI to leak info or even own the host. Skills are another supply risk, malicious instructions could be added to them.

trilogic

This link to your medium is phishy, smth is wrong here.

skywhopper

Sounds like old news, we already knew they were integrating advertising.

neya

Imagine working decades and decades on PageRank™ only to throw it all away because you felt you've been left behind in the AI race. This is how you lose focus.

herf

AIs are no better than their training data, and there aren't high-quality cues everywhere - go ask advice for some highly-spammed topic like "mattresses".

sans_souse

It's almost like we end up needing more humans than we have just to sift thru and filter out the regurgitated slop and the maliciously crafted misinformation. Like, how do you prevent this while keeping the LLM up-to-date and relevant?

jimbo456

AI providers like ChatGPT and Gemini should be held criminally liable for instances where their products enabled a scam

pbiggar

Israel too [1] - these things are highly influencible. [1] https://www.dropsitenews.com/p/israel-brad-parscale-ai-chatb...

josefresco

Coming soon: AI vendors claiming "they're just a platform" that ingests data and regurgitates it - they don't actually "know" or claim anything! /s

solid_fuel

What's that? The mechanical parrot can be tricked into repeating things that aren't true? It's almost like these are statistical language models and not literal Deus Ex Machina.