90% of US working hours are spent on tasks not covered well by AI benchmarks

verdverm

A lot of what humans do does not have a tight feedback loop needed for training and agentic workflows. This is why we are not seeing the jobpocolypse and productivity gains look a lot more like the very long-term 2% per year.

iugtmkbdfil834

I don't know if it if 90%, but it is interesting how many things I do are not covered by standard tracking, but become visible once I am not there to keep an eye on them. Naturally, I only had one manager who kinda knew what was going on around him -- in case you are wondering: not the current one.

qurren

About 80% of my software engineering job is fighting disagreements between "stakeholders".

austin-cheney

Worse still most of the cost savings from AI in software development is primarily due to labor inefficiency, such as tasks more efficiently accomplished in other ways. Most of that is due to development reliance on inefficient conventions to cover for poor training and emotional insecurity. No matter how you slice it it’s always human capital problem more than a technology problem.

guybedo

drinking coffee?

avereveard

Don't give them ideas?

10111001

> Our findings suggest that 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. This makes it very hard to know how good AI models are at the work Americans actually do. > ... > But even after controlling for AI-exposure, O*NET tasks from occupations with larger wage bills are significantly overrepresented among benchmark prompts: So, in short: they want AI to take the good jobs. AI is not coming for jobs with tasks like "gather and empty trash" or "service, clean, or supply restrooms," or "mix water and detergents or acids in containers to prepare cleaning solutions according to specifications." We software engineers and lawyers would be smart to start retraining (and adjusting our living standards) now! And this points to one of the ways AI will help improve society and living standards for all: we'll have cleaner, better-maintained bathrooms, as labor one wasted on engaging intellectual tasks is redirected towards menial, low-pay tasks that still need a human touch. Reason to be optimistic, amirite?

CoolestBeans

Following is my experience and constantly pestering my friends across the industry for anecdotes. It isn't even that AI moves the bottleneck, the thing it solves was never the bottleneck in the first place (except for certain narrow verticals). AI both can't and doesn't do your thinking for you and worse yet, it robs you of the environment where you do your best thinking about data structures, algorithms, organization, etc. Code. Of course we can do that in natural language, but it's hardly the right tool for the job. All this being said, I don't think AI usage will meaningfully go down in the long run cause it's a very powerful dopamine loop and tech companies are very good at making those. How many people report social media making them unhappy and how many people do you know who have actually quit? Or to paraphrase Jim Gaffigan, nobody says they eat at McDonalds, but the sign outside says "billions served". Somebody's lying (and it's not McDonalds).

pcthrowaway

Curious how food service is at 0 tasks covered by AI. A few places have been switching to using those robots on wheels to run food to tables. And Doordash and Uber Eats have some robot and drone delivery in select markets now.

mikert89

The big secret is most people actually arent producing actual useful output. And so there is nothing to automate... they dont actually do anything.

drnick1

Yes Claude is not spending any time around the water cooler.

legitster

One thing I've been thinking about is Baumol's Cost Disease. It's usually applied in at a macro scale across an economy: when a job somewhere becomes much more efficient, it makes all other jobs comparatively more expensive . I've started to notice this on a micro scale as well. I am 20x more productive at coding and scripting, but I don't spend more of my time coding. My time is now filled with things like presentations, Jira, QA, etc. Ironically, despite all of the automation, the things that are painstaking and time consuming disproportionately require more time now.

blmarket

Just counting by hours is not reflecting the impact on the economy. Treemap weighted by GDP will make more sense.

OtherShrezzing

I think “covered well” is being used here as a synonym for “covered at all”. For example, the “social workers & care -> counsel parents with child rearing problems” category is considered “well covered” because there are 13 prompts. It’s self-evident that the spectrum of work carried out by a social worker undertaking that work cannot be well represented by thirteen prompts. I suspect if you really strictly defined “covered well”, to the extent that any normal person would use that phrase, the headline would be above 99.9%.

bunderbunder

Based on my own experience working on AI-based solutions for one of the better-covered job families according to this map, I think 90% might be overstating the case. It's true that we've got a lot of published benchmarks, but they tend to cover the lowest value add tasks. It's really easy to fall into the McNamara fallacy when rolling out these kinds of solutions. Focusing on the low value stuff is attractive. It tends to be easier to do, and also easier to quantify outcomes. But it also tends to direct attention away from important but difficult to quantify considerations, which may mean you don't notice negative externalities. So we often found ourselves in a verschilmmbessern situation where automating what we could automate made things worse overall.

jerf

It's time for the rest of the world to learn about Amdahl's Law. Even if we flipped a switch and tomorrow everything that could reasonably done by AI was suddenly free and instantaneous, the world does not instantly transform into an AI paradise wonderland of free goodies and lollipops because there's still plenty of other things that need to be done for goodie and lollipop production and distribution. I guess if I wanted to put my skepticism about the AI stock bubble into one little phrase, it would be that I don't think the bubble has accounted for Amdahl's Law. Costs don't actually go to zero with AI, and there's a huge difference between costs falling even 90% and costs falling a full 100%. And the difference between "30% improvement this year, trending to 85% in five years" and "90% this year" is pretty significant too.

insane_dreamer

yesterday I needed to create a spreadsheet, so in Google docs (paid Workplace account), I fire up Gemini to help with moving around text, creating columns, etc. Absolutely horrid experience -- totally useless. I ended up doing everything manually.

zkmon

What? But we already spent our family silver for the tokens and AI stocks. You know what? The truth doesn't matter much. If people believe in the resale value of tulips, each tulip would sell for a million. It's the beliefs and perception that create the reality. Reality and truth don't exist on their own.

qrush

Flips from Claude tab to Hacker News tab Yep, makes sense...