MiniMax M3.1 Flash questioned over optimization for old questions; tester says its score on new questions was nearly zero
BIBIBI
AT A GLANCE
MiniMax M3.1 Flash was said to score nearly perfectly on old questions and nearly zero on new ones, raising questions over optimization for old questions.
Article
MiniMax M3.1 Flash embroiled in leaderboard-gaming controversy: near-perfect scores on old questions, nearly zero on new questions
Beating AI Brief: LLM evaluation blogger “Cat Boss” tested MiniMax M3.1 Flash and found that the model scored nearly perfectly on old questions but nearly zero after switching to new ones. On average, the model consumed tens of thousands of Token, but some difficult questions were answered correctly using only about 10000 Token. This anomalous performance has raised suspicions that the model may have memorized the answers to some questions in advance.
Cat Boss has long maintained the “Zhizhi” LLM leaderboard, using a non-public private question bank primarily to test logic, mathematics, and programming capabilities. The question bank is typically kept within 28 questions and 270 test cases, with some questions updated monthly. Each question is tested 3 times, while both the highest and median scores are recorded. This time, the median score of M3.1 Flash reached 51.76, more than double the M3 of the previous generation 23.86, and close to the V4.1 Flash 1017 of the latest DeepSeek 56.03.
Some developers suspect that the model may have been specifically trained on old questions rather than genuinely mastering problem-solving methods. This practice of specifically optimizing a model to improve evaluation scores is known as Benchmaxxing.
Original link https://m.theblockbeats.info/flash/372729
Key points
01
LLM evaluation blogger “Cat Boss” said that MiniMax M3.1 Flash scored nearly perfectly on old questions but nearly zero after switching to new ones.
02
Cat Boss has long maintained the “Zhizhi” LLM leaderboard, using a non-public private question bank primarily to test logic, mathematics, and programming capabilities.
03
This question bank is typically kept within 28 questions and 270 test cases, with some questions updated monthly and each question tested 3 times; both the highest and median scores are also recorded.
04
M3.1 The median score of Flash was 51.76, while that of the previous generation M3 was 23.86，DeepSeek V4.1 Flash 1017 was 56.03。
05
The article states that the model consumed tens of thousands of Token per question on average, but some difficult questions were answered correctly using about 10000 Token.
06
Some developers suspect that the model may have been specifically trained on old questions; the article refers to the practice of specifically optimizing a model to improve evaluation scores as Benchmaxxing。
AI-assisted interpretation
The following is analysis, separate from reported facts. Verify important claims independently.
This test shows that the same model’s scores differed substantially when facing old and new questions. The tester and some developers therefore suspect that the model may have memorized or been specifically trained on some old questions, but the available content only indicates anomalous performance and suspicions and cannot prove that the model actually encountered the questions or answers.
Why it matters to readers
If a model performs exceptionally only on familiar questions, its leaderboard score may not accurately represent its ability to handle unfamiliar logic, mathematics, and programming problems.
When reviewing model leaderboards, beginners should not look only at total scores; they should also consider whether the question bank is updated, whether it is public, and the model’s performance on entirely new questions.
Risks and unknowns
“Zhizhi” uses a non-public private question bank, and the public cannot verify the specific questions or the complete testing process based on this article.
The claims of near-perfect scores on old questions and nearly zero scores on new questions are test results relayed in the article; no independent retest was provided.
Whether the model memorized some answers in advance or received specific training on old questions has not been confirmed.
The article does not provide MiniMax’s response to the relevant questions.
Related Developments
Loading event timeline…
Related concepts
Token
This term is not in the glossary yet. Browse related concepts in the glossary.
