Kinney Drugs pulls back AI phone assistant after hundreds of customer complaints

kotaKat

Related: A pharmacy chain in Vermont implemented AI for efficiency (11 days ago, 40 points) https://news.ycombinator.com/item?id=49105190

nubg

> Marraffa says Burt is fully HIPAA compliant and that the AI is not open-source and does not generate or manipulate data. We need to fight the talking point that open source is bad!

cmiles8

Companies will pawn off thier customers to AI at their peril. In many ways this is a repeat of the India call center train wrecks of the 00s. On paper, letting someone in Bangalore vs onshore handle incoming customer service calls looked like a path to amazing savings. In practice the customer experience was horrendous and companies CTRL-Zed these decisions and rapidly brought customer service back onshore again. AI is just that story of shortsighted decisions by weak leadership playing out all over again.

jdw64

I think of this as the kiosk economy. It's like how businesses introduce kiosks even when having a human employee would be more convenient, just for the sake of greater profit. The biggest problem with AI customer service is that a human employee would've let a minor issue slide without escalating it. But a chatbot often inflames the situation, and by the time the customer reaches a human agent, they're already furious. Most people aren't rational or logical. Non verbal feedback, like acknowledging someone's anger and showing empathy, is incredibly important.

eithed

It's interesting that I had to close 3 popups just to read this article, two of them were for tracking cookies

asdev

if you've tried to legitimately productionize LLMs to deliver a specific repeatable outcome, rather than human editable output(text, code), you know the models still aren't there yet and the whole product experience feels hacky

exizt88

As someone who's been doing AI call centres for a while, I'm not surprised. It's really, really hard to build voice AI that works. There are no open source solutions that work out of the box. But the biggest problem is ASR. WER is still atrocious even with SOTA models. When you add drug names and regional accents, it's a recipe for disaster.

newswasboring

The desired speed of AI adoption is what is hindering AI adoption. Big companies are trying to sell this as magic, and it's not. It requires proper use to get anything useful out of this system. But that takes away the magic of it and the investors can't have that as the eventual share price only works if it is magic. Frontier labs will smother their product with their timelines.

pessimizer

It's important to note while people are pumping the AI bubble that a near-moron with a checklist can do this job. They replaced AI with a regular old phone tree. AI couldn't do this job with a checklist. It would forget the entire checklist after two responses, then get fixated on the words that the customer used to complain about how dumb the AI was and loop until they hung up and called back to hopefully clear the context. edit: Unless Kinney had been sold a product that used caller ID to carry over context, then the customer's best chances would be to scream "Agent!" over and over again (to get put on hold for 45 minutes, then transferred to the most screamed-at Philippine minimum-wage gig worker on the planet), or to show up to the drugstore and start screaming at people.

SalaevAl

Pharmacy phone lines are a terrible first use case for voice AI. Older callers, drug names, insurance mess, zero patience. Automating "store hours" is fine. Automating refill exceptions is how you get this outcome.

jonathan-adly

So - I run a company that does AI agents for pharmacy. We are at Series B now and growing well - series C in 9-12 months given growth. We just acquired a competitor of that vendor so very much an insider. First, The technology works, and it scales, but the whole bottleneck is domain expertise and implementation. These are expensive, and hard to scale. We hire pharmacists as project managers, that's how important domain expertise and implementations are. Second, the amount of noise of "Voice AI for <industry>" is incredible. Most of them are completely clueless about the industry & basically "YC-striver" type who can only sell to other yc companies. They fail hard the moment they touch critical functions of the real world. We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".

criddell

This is my favorite AI customer service story: https://arstechnica.com/tech-policy/2024/02/air-canada-must-... > Air Canada essentially argued that “the chatbot is a separate legal entity that is responsible for its own actions”

crystaln

I guess everything is ok these days as long as the President says "We screwed up [our most basic responsibility to customers], and we own that."

dj_axl

FWIW I know of a pharmacy network that's implementing AI. Not only does the AI make egregious errors like speaking in Spanish when the conversation is in English. It also has a limited context window for ground rules of 5000 tokens or less. So correcting all the various errors goes over the instructions limit. As a result if the implementation goes through it will be of poor quality. All I can think is that people with no technical expertise are making these decisions, it is easy for any techie to predict that AI of this level won't be capable.

fn-mote

> wrong dosages, and missed prescription notifications. BS. How is the AI touching that information? Prescription notification is outbound communication. It’s almost a recorded message. A little text to speech if it is supposed to read the drug name. Wrong dosages?? This must be handled by the “enroll a new scrip” functionality, right? So basically B2B doctor interaction. You let AI touch that? Aren’t the vast majority of the costs from customers not doctors? None of this makes sense.

laughingcow

I happened to be in Vermont last week and was unaware of this change. I heard complaints from three separate people unprompted. It was a total dumpster fire put on top of a critical service.

sicktriple

Do we think that as a society there exists some meaningful lever by which we can apply force against this type of carelessness and exploitation? I think we need to take a good hard look at what we want out of this world and ask ourselves. Do we really need to automate this? Like really. What benefit does the world receive having an AI chatbot tell grandma the wrong doage for her meds? This is the naked extraction of value from a service that's already been offshored. Companies like these are gleeful to make their service as shitty as they can because they know their users are doing so against their will. Infact, that's become the goal.

designium

How open source makes it less private?!?

coffeefirst

This doesn’t surprise me. Set up whatever online portal or app you want. I’d prefer to just sign in and click a button. If I’m picking up the phone or sending an email it’s because I have a weird problem and I need to talk to a person.

wolfi1

my experience with AI assistants has not been that great so far, they ask for customer id numbers which are long as ... with several digits and if you group them they don't understand, for example if the number begins with two 1s and you say eleven instead of one one they don't understand you, then they are usually not trained for the local pronunciation or dialect which makes it very hard communicating, my blood pressure rises into unhealthy ranges, but if I use expletives enough I get a human on the line