Stanford Study Finds 44% of Classic LLM Research Conclusions No Longer Hold

Researchers at Stanford University re-evaluated 50 widely cited LLM research findings across 20 models and discovered that 44% of them are no longer valid, with an average lifespan of just 15.7 months for the invalidated claims. Negative capability claims about what models cannot do expired the fastest: all nine findings on reasoning flaws—such as counting letters in "strawberry" and lost-in-the-middle context issues—have either faded or become obsolete. In contrast, behavioral tendencies such as model preferences, sycophancy, and strategic behavior tend to persist longer.