A study published in the journal Nature revealed that as AI language models (LLMs) advance, they are less likely to admit when they don't know how to answer.
ADVERTISING
Researchers at the Universitat Politècnica de València in Spain tested the BLOOM, Llama and GPT models to see how accurate they were in math, science and geography, with thousands of questions. The answers were classified as correct, incorrect or evasive.
AI models’ honesty is declining
- The study showed that, Although accuracy on more difficult questions has improved, these models are less transparent about their ability to answer.
- Previous models used to admit when they didn’t know or needed more information. Now, models tend to take chances, and can get simple questions wrong.
- Even when mastering complex problems, LLMs still make mistakes on basic issues.
- “Full reliability is not achieved even at very low difficulty levels,” the study noted.
For example, the GPT-4 showed fewer evasive responses compared to GPT-3.5, but did not exceed expectations for avoiding responses outside its capacity. The researchers concluded that, despite the advances, there was no significant improvement in this aspect.
Read also
