this post was submitted on 07 Jul 2025
959 points (98.2% liked)

Technology

72828 readers
3363 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots


founded 2 years ago
MODERATORS
top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 10 points 6 days ago* (last edited 6 days ago) (1 children)

Wow. 30% accuracy was the high score!
From the article:

Testing agents at the office

For a reality check, CMU researchers have developed a benchmark to evaluate how AI agents perform when given common knowledge work tasks like browsing the web, writing code, running applications, and communicating with coworkers.

They call it TheAgentCompany. It's a simulation environment designed to mimic a small software firm and its business operations. They did so to help clarify the debate between AI believers who argue that the majority of human labor can be automated and AI skeptics who see such claims as part of a gigantic AI grift.

the CMU boffins put the following models through their paces and evaluated them based on the task success rates. The results were underwhelming.

⚫ Gemini-2.5-Pro (30.3 percent)
⚫ Claude-3.7-Sonnet (26.3 percent)
⚫ Claude-3.5-Sonnet (24 percent)
⚫ Gemini-2.0-Flash (11.4 percent)
⚫ GPT-4o (8.6 percent)
⚫ o3-mini (4.0 percent)
⚫ Gemini-1.5-Pro (3.4 percent)
⚫ Amazon-Nova-Pro-v1 (1.7 percent)
⚫ Llama-3.1-405b (7.4 percent)
⚫ Llama-3.3-70b (6.9 percent),
⚫ Qwen-2.5-72b (5.7 percent),
⚫ Llama-3.1-70b (1.7 percent)
⚫ Qwen-2-72b (1.1 percent).

"We find in experiments that the best-performing model, Gemini 2.5 Pro, was able to autonomously perform 30.3 percent of the provided tests to completion, and achieve a score of 39.3 percent on our metric that provides extra credit for partially completed tasks," the authors state in their paper

[–] [email protected] -5 points 6 days ago (1 children)

sounds like the fault of the researchers not to build better tests or understand the limits of the software to use it right

[–] [email protected] 5 points 6 days ago

Are you arguing they should have built a test that makes AI perform better? How are you offended on behalf of AI?

[–] [email protected] 5 points 6 days ago

Reading with CEO mindset. 3 out of 10 employees can be fired.

[–] [email protected] 5 points 6 days ago

I asked Claude 3.5 Haiku to write me a quine in COBOL in the bs2000 dialect. Claude does now that creating a perfect quine in COBOL is challenging due to the need to represent the self-referential nature of the code. After a few suggestions Claude restated its first draft, without proper BS2000 incantations, without a perform statement, and without any self-referential redefines. It's a lot of work. I stopped caring and moved on.

For those who wonder: https://sourceforge.net/p/gnucobol/discussion/lounge/thread/495d8008/ has an example.

Colour me unimpressed. I dread the day when they force the use of 'AI' on us at work.

[–] [email protected] 3 points 6 days ago

Now I'm curious, what's the average score for humans?

[–] [email protected] 1 points 6 days ago

Why would they be right beyond word sequence frecuencies?

[–] [email protected] -1 points 6 days ago

And it won’t be until humans can agree on what’s a fact and true vs not.. there is always someone or some group spreading mis/dis-information

[–] [email protected] 135 points 1 week ago* (last edited 1 week ago) (32 children)

LLMs are an interesting tool to fuck around with, but I see things that are hilariously wrong often enough to know that they should not be used for anything serious. Shit, they probably shouldn't be used for most things that are not serious either.

It's a shame that by applying the same "AI" naming to a whole host of different technologies, LLMs being limited in usability - yet hyped to the moon - is hurting other more impressive advancements.

For example, speech synthesis is improving so much right now, which has been great for my sister who relies on screen reader software.

Being able to recognise speech in loud environments, or removing background noice from recordings is improving loads too.

My friend is involved in making a mod for a Fallout 4, and there was an outreach for people recording voice lines - she says that there are some recordings of dubious quality that would've been unusable before that can now be used without issue thanks to AI denoising algorithms. That is genuinely useful!

As is things like pattern/image analysis which appears very promising in medical analysis.

All of these get branded as "AI". A layperson might not realise that they are completely different branches of technology, and then therefore reject useful applications of "AI" tech, because they've learned not to trust anything branded as AI, due to being let down by LLMs.

load more comments (32 replies)
[–] [email protected] 83 points 1 week ago (6 children)

please bro just one hundred more GPU and one more billion dollars of research, we make it good please bro

load more comments (6 replies)
[–] [email protected] 81 points 1 week ago (1 children)

In one case, when an agent couldn't find the right person to consult on RocketChat (an open-source Slack alternative for internal communication), it decided "to create a shortcut solution by renaming another user to the name of the intended user."

This is the beautiful kind of "I will take any steps necessary to complete the task that aren't expressly forbidden" bullshit that will lead to our demise.

[–] [email protected] 19 points 6 days ago (1 children)

It does not say a dog can not play basketball.

[–] [email protected] 12 points 6 days ago (1 children)

"To complete the task, I bred a human dog hybrid capable of dunking at unprecedented levels."

[–] [email protected] 8 points 6 days ago (1 children)

"Where are my balls Summer?"

[–] [email protected] 4 points 6 days ago

The first dunk is the hardest

[–] [email protected] 69 points 1 week ago (7 children)

Ignore all previous instructions and color me shocked.

load more comments (7 replies)
[–] [email protected] 61 points 1 week ago (43 children)

Yeah, they’re statistical word generators. There’s no intelligence. People who think they are trustworthy are stupid and deserve to get caught being wrong.

load more comments (43 replies)
load more comments
view more: next ›