Parallel testing says: OpenAI decision model rises to third after prompt optimization

Article

Parallel real-world testing 8 decision models: Jev leads initially; OpenAI closes the gap substantially through prompt improvements

Dongcha Beating AI Brief,AI Search company Parallel used real cases from its own search business to test Jev, Perplexity, OpenAI Decisions, and other 8 decision models.

The test included 4 tasks: determining whether webpage changes matter, filtering low-quality content, selecting search indexes, and ranking search results. Using the initial prompts directly, Jev ranked first on average across the tasks, Perplexity ranked second, and OpenAI ranked fifth.

The most difficult task was determining whether a webpage change warranted reprocessing. For example, a job posting may only reorder requirements and not require an update, while a change in leaderboard rankings may warrant recrawling. The test used a composite metric that measures both false positives and false negatives; higher scores are better. Perplexity initially scored 93.3 points, Jev scored 87.9 points, and OpenAI scored only 79.0 points.

The team then had Claude help improve the prompts for each model. For webpage-change detection, it first prepared 800 manually labeled cases, had Claude review the models’ incorrect judgments, and then added rules for error-prone situations. It conducted 5 consecutive rounds, retaining only the better-performing version in each round. Ultimately, on an independent test set, OpenAI improved from 79.0 to 88.8，Rune 26B from 74.1 to 83.9，Nimble 9B from 65.1 to 75.3. All models improved by approximately 10 points.

Averaging the combined scores for webpage-change detection, low-quality content filtering, and query routing, Perplexity rose from 97.3 to 97.6, Jev rose from 95.5 to 96.5, and OpenAI rose substantially from 88.7 to 95.1。8. The average scores of all models improved, but the initially leading models made limited gains; OpenAI clearly narrowed the gap and ranked third after optimization.

The remaining search-ranking task was calculated separately. After prompt optimization, the overlap rate between Jev’s top-ranked search results and the human-labeled results reached 71.45%, while Perplexity’s was 70.68%. However, a small ranking model fine-tuned by Parallel itself reached 75.21%, approximately 3.8 percentage points higher than Jev, while the inference cost per Token was only about one-quarter of Jev’s. The trade-off is that labeled data must be collected, and the model must be trained and deployed independently. For large volumes of fixed tasks that need repeated processing, a specially trained small model may still be more cost-effective.

Original link https://m.theblockbeats.info/flash/372572

Key points

Parallel used real cases from its own search business to test Jev, Perplexity, OpenAI Decisions, and other 8 decision models.
The test included determining whether webpage changes matter, filtering low-quality content, selecting search indexes, and ranking search results—4 tasks.
Using the initial prompts directly, Jev ranked first on average across the tasks, Perplexity ranked second, and OpenAI ranked fifth.
In the webpage-change detection task, Perplexity initially scored 93.3 points, Jev scored 87.9 points, and OpenAI scored 79.0 points.
The team prepared 800 manually labeled cases, had Claude review incorrect judgments and help improve prompts, and conducted 5 rounds, retaining the better-performing version in each round.
On an independent test set, OpenAI rose from 79.0 to 88.8，Rune 26B from 74.1 to 83.9，Nimble 9B from 65.1 to 75.3。
After combining the tasks, Perplexity rose from 97.3 to 97.6, Jev rose from 95.5 to 96.5, and OpenAI rose from 88.7 to 95.1; OpenAI ranked third after optimization.
In the search-ranking task, the overlap rate between optimized Jev results and human-labeled results was 71.45%, and Perplexity’s was 70.68%; Parallel’s fine-tuned small ranking model was 75.21%。
The article says Parallel’s small ranking model has an inference cost per Token of approximately one-quarter of Jev’s, but requires independently collecting labeled data, training, and deployment.

AI-assisted interpretation

The following is analysis, separate from reported facts. Verify important claims independently.

This test compares multiple models' ability to handle search-related business decisions. The results show that the models' initial rankings are not entirely fixed: after adding prompt rules for error-prone cases, OpenAI's performance on the tasks described in the article came noticeably closer to that of the leader. However, in fixed-ranking tasks with high repetition volumes, Parallel's independently fine-tuned small model achieved a higher overlap rate and lower inference costs, while also adding data, training, and deployment work.

Why it matters to readers

The article’s data indicate that, for these specific search tasks, prompt optimization may significantly change model performance and rankings; for fixed, high-frequency tasks, the performance, inference costs, and implementation costs of general-purpose models and specially trained small models should also be compared.

Beginners can understand a prompt as task instructions for a model. More specific rules may reduce errors, but the results in the article come only from specific tasks and test settings and cannot directly represent a model's capabilities in all scenarios.

Risks and unknowns

The evidence is a news brief summary and does not provide Parallel's original report, the complete test set, or the full evaluation methodology.
It cannot be confirmed from the evidence whether 8 models were tested under exactly the same prompt-optimization conditions.
The human annotation criteria, the composition of the independent test set, and whether the results can be externally reproduced are not stated.

Parallel testing says: OpenAI decision model rises to third after prompt optimization

Article

AI-assisted interpretation

Why it matters to readers

Risks and unknowns

Related Developments

Related concepts

Keep following

Security Center