Get all your news in one place.
100's of premium titles.
One app.
Start reading
Fortune
Fortune
Jeremy Kahn

BIZARRE: Ai models are getting very good at professional tasks new openai research shows - What They Never Told You

A smart phone screen displaying the app icons for a number of AI chatbots, including ChatGPT, Claude, Perplexity, Gemini, and others. (Credit: Jaque Silva—NurPhoto via Getty Images)

Hello and welcome to Eye on AI. In this edition…A new OpenAI benchmark shows how good models are getting at completing professional tasks…California has a new AI law…OpenAI rolls out Instant Purchases in ChatGPT…and AI can pick winning founders better than most VCs.

Google CEO Sundar Pichai was right when he said that while AI companies aspire to create AGI (artificial general intelligence), what we have right now is more like AJI—artificial jagged intelligence. What Pichai meant by this is that today’s AI is brilliant at some things, including some tasks that even human experts find difficult, while also performing poorly at some tasks that a human would find relatively easy.

Thinking of AI in this way partly explains the confusing set of headlines we’ve seen about AI lately—acing international math and coding competitions, while many AI projects fail to achieve a return on investment and people complain about AI-created “workslop” being a drag on productivity. (More on some of these pessimistic studies later. Needless to say, there is often a lot less to these headlines than meets the eye.)

One of the reasons for the seeming disparity in AI’s capabilities is that many AI benchmarks do not reflect real world use cases. Which is why a new gauge published by OpenAI last week is so important. Called GDPval, the benchmark evaluates leading AI models on real-world tasks, curated by experts from across 44 different professions, representing nine different sectors of the economy. The experts had an average of 14 years experience in their fields, which ranged from law and finance to retail and manufacturing, as well as government and healthcare. 

Sign up to read this article
Read news from 100's of titles, curated specifically for you.
Already a member? Sign in here
Related Stories
Top stories on inkl right now
One subscription that gives you access to news from hundreds of sites
Already a member? Sign in here
Our Picks
Fourteen days free
Download the app
One app. One membership.
100+ trusted global sources.