I love to show that kind of shit to AI boosters. (In case you're wondering, the numbers were chosen randomly and the answer is incorrect).
They go waaa waaa its not a calculator, and then I can point out that it got the leading 6 digits and the last digit correct, which is a lot better than it did on the "softer" parts of the test.
Idk personally i kind of expect the ai makers to have at least had the sense to allow their bots to process math with a calculator and not guesswork. That seems like, an absurdly low bar both for testing the thing as a user as well as a feature to think of.
Didn't one model refer scientific questions to wolfram alpha? How do they smartly decide to do this and not give them basic math processing?
under no circumstances is “my favorite stochastic parrot got it right / I reran the prompt and it worked this time hmmmm wonder why” an interesting post