BestSuperintelligent
Explore evaluations
Research

Tests published by others

Every test result we found for the products on this site, run by other people. Each shows who ran it, the source, a short quote and the limit of the result.

All records

One row per result. Open a product for the full wording.

Summary of published test results

Test results published by others, by product
ProductWhat was measuredResultWho ran it
Claude CodeDeepSWE v1.1 resolved, DeepSeek V4.1 Flash inside the Claude Code scaffold69.8%tag:ven:Vendor-reported
OpenAI CodexDeepSWE v1.1 resolved, DeepSeek V4.1 Flash inside the Codex scaffold65.6%tag:ven:Vendor-reported
Claude CodeTerminal-Bench 2.0 resolution rate (Claude Code + Claude Opus 4.5)52.1% ±2.5%tag:src:Academic paper
OpenAI CodexTerminal-Bench 2.0 resolution rate (Codex CLI + GPT-5.2)62.9% ±3.0%tag:src:Academic paper
CursorEffect on task completion time for experienced open-source developers19% longer to complete issues with AI tools allowedtag:src:Independent
LangGraphPipeline speed relative to CrewAI2.2x faster than CrewAItag:src:Independent
PerplexityIncorrect answers when identifying source of news excerpts37 percent of queries answered incorrectlytag:src:Independent
NotebookLMShare of outputs with at least one hallucination13% (ChatGPT and Gemini 40%)tag:src:Academic paper
DeepgramAA-WER (non-streaming speech-to-text), Nova-35.2%tag:src:Independent
DeepgramAverage WER, English lecture recordings8.3tag:src:Academic paper
DeepgramSpeech Arena Elo (text-to-speech), Flux TTS1050 Elo (95% interval -13/13), rank 65 of 98tag:src:Independent
ElevenLabsSpeech Arena Elo (text-to-speech), Eleven v41336 Elo (95% interval -15/15), rank 1 of 98tag:src:Independent
ElevenLabsAA-WER (non-streaming speech-to-text), Scribe v22.2%tag:src:Independent
Murf AISpeech Arena Elo (text-to-speech), Falcon 21157 Elo (95% interval -12/12), rank 27 of 98tag:src:Independent
Otter.aiMedian WER, Zoom-Otter AI live transcription, psychiatric research interviews19.2% (IQR 15.1-24.8)tag:src:Academic paper
RunwayText-to-video Elo, Runway Gen-4.5 (720p)909 Elo (95% interval -18/18), rank 24 of 32tag:src:Independent
Claude CodeSWE-bench Verified, Claude Opus 4.8 in the Claude Code harness85.80%tag:src:Independent
GitHub CopilotTime to complete one JavaScript HTTP-server task55.8% faster with Copilottag:src:Academic paper
Snowflake Cortex AnalystBIRD-SQL text-to-SQL accuracy with a generated semantic model78% average, against 57% for Claude 3.5 Sonnet without the semantic modeltag:ven:Vendor-reported
OpenAI CodexSWE-bench Verified, GPT-5.5 Codex in the Codex harness76.4%tag:src:Independent
ElevenLabsReal-time voice median latency, 24-hour1963.0 ms median, 1969 ms at the checktag:src:Independent
CursorSWE-bench Multilingual, Composer 2.579.8%tag:ven:Vendor-reported
CursorSWE-bench Verified, Composer 2.5 in the Cursor CLI harness79.6%tag:src:Independent
GitHub CopilotShare of generated programs found vulnerableapproximately 40% of 1,689 programstag:src:Academic paper
NotebookLMLung cancer staging accuracy on 100 fictional CT cases86% diagnostic accuracy; 95% reference-location accuracytag:src:Academic paper
RunwayVBench total score, Gen-2 (2023-12)80.58 total (quality 82.47, semantic 73.03)tag:ven:Vendor-reported
Snowflake Cortex AnalystSQL accuracy on Snowflake's internal BI questions90%+ for Cortex Analyst; 51% for single-prompt GPT-4otag:ven:Vendor-reported
Otter.aiWord error rate vs human-corrected transcripts, Zoom speech tasks2.50% on the aging-well task, 3.36% on memory recalltag:src:Academic paper
Murf AIPronunciation accuracy, Speech Gen 299.37%tag:ven:Vendor-reported
Murf AINaturalness win rate against five other TTS vendorswins 8 of 10 language comparisons; 60% vs ElevenLabs in Englishtag:ven:Vendor-reported
LangGraphCompany-research pipeline, LLM-judge quality9.42/10 quality, 9.44/10 accuracy, 506s, 8,823 tokenstag:src:Independent
Microsoft Agent FrameworkCompany-research pipeline, LLM-judge quality9.87/10 quality, 93s, 7,006 tokens, std dev 0.10tag:src:Independent
PerplexityCredibility of cited sources on 100 claims86% average source credibility; 0.7% non-credible sourcestag:src:Academic paper

Claude Code

DeepSWE v1.1 resolved, DeepSeek V4.1 Flash inside the Claude Code scaffold: 69.8%

DeepSeek's own model card compares scaffolds at max reasoning effort. This cell is their model, not Claude, running in the Claude Code column. N=8 samples per task.

Source: DeepSeek-V4.1-Flash model card, scaffold table. tag:ven:Vendor-reported. Published 2026-09-10; read 2026-10-28.

Quote: “DeepSWE v1.1 (Resolved) | Claude Code | 69.8”

Limit: Vendor-reported by DeepSeek. The same row's mini-SWE cell is 74.2. It is not an independent test of Claude Code, and it is not a Claude model.

Terminal-Bench 2.0 resolution rate (Claude Code + Claude Opus 4.5): 52.1% ±2.5%

Terminal-Bench 2.0 paper authors ran Claude Code with Claude Opus 4.5 on the 89-task Terminal-Bench 2.0 benchmark (arXiv, Jan 2026).

Source: Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (arXiv:2601.11868), Table 2. tag:src:Academic paper. Published 2026-01-17; read 2026-10-27.

Quote: “Claude Opus 4.5 | Claude Code | 52.1% ±2.5%”

Limit: Result is for one model/agent-version pairing at paper time (v1); other Claude Code models scored 27.5% to 40.1%; later versions are not covered.

SWE-bench Verified, Claude Opus 4.8 in the Claude Code harness: 85.80%

Vals AI archived SWE-bench Verified on 1 September 2026. Rank 16 of 88 uses the Claude Code harness. The named row lined up with that rank is Claude Opus 4.8. Cost $0.67 per test, duration 2m 15s.

Source: SWE-bench Verified leaderboard (Vals AI). tag:src:Independent. Published 2026-09-01; read 2026-10-28.

Quote: “Claude Code | 85.80% | $0.67 | 2m15s”

Limit: This is one older Claude model inside the Claude Code harness, not a current Claude Code score. Vals says the benchmark is saturated and no longer updated. The model name is taken by lining the rank up with the named table on the same page.

Read the Claude Code review →

OpenAI Codex

DeepSWE v1.1 resolved, DeepSeek V4.1 Flash inside the Codex scaffold: 65.6%

DeepSeek's model card scaffold table. This cell is DeepSeek V4.1 Flash in the Codex column, at max reasoning effort, N=8.

Source: DeepSeek-V4.1-Flash model card, scaffold table. tag:ven:Vendor-reported. Published 2026-09-10; read 2026-10-28.

Quote: “DeepSWE v1.1 (Resolved) | Codex | 65.6”

Limit: Vendor-reported by DeepSeek. It does not measure OpenAI's Codex product with an OpenAI model.

Terminal-Bench 2.0 resolution rate (Codex CLI + GPT-5.2): 62.9% ±3.0%

Terminal-Bench 2.0 paper authors ran Codex CLI with GPT-5.2 on the 89-task Terminal-Bench 2.0 benchmark (arXiv, Jan 2026); the paper reports it as the highest average resolution rate (63%).

Source: Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (arXiv:2601.11868), Table 2. tag:src:Academic paper. Published 2026-01-17; read 2026-10-27.

Quote: “Codex CLI paired with GPT-5.2 achieves the highest average resolution rate of 63%.”

Limit: Tests the Codex CLI agent (not the cloud or IDE Codex surfaces) with one model at paper time; other Codex CLI models scored 11.5% to 49.6%.

SWE-bench Verified, GPT-5.5 Codex in the Codex harness: 76.4%

BenchLM's named mirror of the Vals board, 1 September 2026. Rank 37 of 88. The row is labeled GPT-5.5 Codex, configuration Codex, creator OpenAI.

Source: SWE-bench, Vals AI run (BenchLM mirror). tag:src:Independent. Published 2026-09-01; read 2026-10-28.

Quote: “GPT-5.5 Codex | Codex · Codex | OpenAI | 76.4%”

Limit: This is GPT-5.5 Codex, not a newer Codex model. The mirror does not print the $0.65 cost or the 108.54s duration. Archived benchmark.

Read the OpenAI Codex review →

Cursor

Effect on task completion time for experienced open-source developers: 19% longer to complete issues with AI tools allowed

METR randomized controlled trial: 16 experienced open-source developers, 246 issues, AI tools mainly Cursor Pro with Claude 3.5/3.7 Sonnet, early 2025.

Source: METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. tag:src:Independent. Published 2025-07-10; read 2026-10-27.

Quote: “take 19% longer to complete issues”

Limit: Small sample, early-2025 Cursor and models; the METR page says the results are out of date and that updated data was published in February 2026 (not read).

SWE-bench Multilingual, Composer 2.5: 79.8%

BenchLM's October 2026 Claude Code vs Cursor comparison prints Composer 2.5 at 79.8 on SWE-bench Multilingual. RankedAGI's Composer 2.5 card prints the same 79.8% under SWEBench Multilingual.

Source: Claude Code vs Cursor (BenchLM). tag:ven:Vendor-reported. Published 2026-10-26; read 2026-10-28.

Quote: “Composer 2.5 | SWE-bench Multilingual | 79.8”

Limit: BenchLM says the multilingual numbers on this page come from each provider's published results, not from a BenchLM run. It is not the CursorBench 4.0 score of 27.7, which is Cursor's own benchmark.

SWE-bench Verified, Composer 2.5 in the Cursor CLI harness: 79.6%

BenchLM's named mirror of the Vals board, updated 1 September 2026. Rank 25 is Composer 2.5, configuration Cursor CLI, creator Cursor.

Source: SWE-bench, Vals AI run (BenchLM mirror). tag:src:Independent. Published 2026-09-01; read 2026-10-28.

Quote: “Composer 2.5 | Cursor CLI · Cursor CLI | Cursor | 79.6%”

Limit: Archived benchmark. Vals stopped running new models because scores had saturated. This is Cursor's Composer 2.5, not the Cursor editor as a whole.

Read the Cursor review →

LangGraph

Pipeline speed relative to CrewAI: 2.2x faster than CrewAI

AIMultiple ran a five-agent sequential travel-planning workflow on LangGraph, LangChain, AutoGen and CrewAI, 100 runs per framework (means), Claude Haiku 4.5 via OpenRouter, temperature 0.1.

Source: Agentic Orchestration (AIMultiple Research). tag:src:Independent. Published 2026-06-04; read 2026-10-27.

Quote: “LangGraph finished 2.2x faster than CrewAI”

Limit: Single workflow and single model; page dates are inconsistent (updated Jun 4 and Oct 12, 2026); framework versions not recorded by me.

Company-research pipeline, LLM-judge quality: 9.42/10 quality, 9.44/10 accuracy, 506s, 8,823 tokens

Lukasz Grochal ran LangGraph 1.0.x, CrewAI, AutoGen, MS Agent Framework and the OpenAI Agents SDK on the same three-agent pipeline. Local Qwen 3 14B, temperature 0. Three companies, three runs each (45 runs total). Posted 16 February.

Source: I Benchmarked 5 AI Agent Frameworks (Lukasz Grochal, DEV). tag:src:Independent. Published 2026-02-16; read 2026-10-28.

Quote: “LangGraph | 9.42 | 9.11 | 9.44 | 9.89 | 9.22 | 9.78”

Limit: One author's LLM judge, nine LangGraph runs. The author says LangGraph's quality gap versus the others is not significant after correction. Standard deviation 0.32. Not a consortium leaderboard. The 96% error-recovery figure on other sites names no test and is not recorded.

Read the LangGraph review →

Perplexity

Incorrect answers when identifying source of news excerpts: 37 percent of queries answered incorrectly

Tow Center tested eight AI search tools in February 2025 on excerpts from 200 articles (20 publishers x 10), 1,600 queries total, asking for headline, publisher, date and URL.

Source: AI Search Has A Citation Problem (Columbia Journalism Review, Tow Center). tag:src:Independent. Published 2025-03-06; read 2026-10-27.

Quote: “Perplexity answering 37 percent of the queries incorrectly.”

Limit: Narrow task (news-excerpt attribution), tested February 2025; not a general accuracy measure of Perplexity.

Credibility of cited sources on 100 claims: 86% average source credibility; 0.7% non-credible sources

Vykopal, Pikuliak, Ostermann and Šimko compared GPT-4o, GPT-5, Perplexity and Qwen Chat on 100 claims across five topics. Accepted at EACL 2026.

Source: Assessing Web Search Credibility and Response Groundedness in Chat Assistants (arXiv 2510.13749). tag:src:Academic paper. Published 2026-02-19; read 2026-10-28.

Quote: “Perplexity achieved the highest overall credibility (on average 86%) and the lowest non-credibility rate (0.7%)”

Limit: Source-credibility of citations, not whether the answer was correct. Separate from the Tow Center finding that Perplexity misidentified the news source on 37% of queries.

Read the Perplexity review →

NotebookLM

Share of outputs with at least one hallucination: 13% (ChatGPT and Gemini 40%)

Hagar, Agustianto and Diakopoulos tested ChatGPT, Gemini and NotebookLM on a reporting-style task over a 300-document corpus about U.S. TikTok litigation; sentence-level annotation.

Source: Not Wrong, But Untrue: LLM Overconfidence in Document-Based Queries (arXiv:2509.25498). tag:src:Academic paper. Published 2025-09-29; read 2026-10-27.

Quote: “rates approximately three times higher for Gemini and ChatGPT (40%) than for NotebookLM (13%)”

Limit: Small sample (40 annotated responses across tools), one corpus, preprint not peer-reviewed, model versions not recorded here.

Lung cancer staging accuracy on 100 fictional CT cases: 86% diagnostic accuracy; 95% reference-location accuracy

Tozuka et al. gave NotebookLM the Japanese lung-cancer staging guideline and asked it to stage 100 fictional cases. GPT-4o scored 39% with the same knowledge and 25% without it.

Source: Application of NotebookLM for Lung Cancer Staging (arXiv 2410.10869). tag:src:Academic paper. Published 2024-10-08; read 2026-10-28.

Quote: “NotebookLM achieved 86% diagnostic accuracy in the lung cancer staging experiment”

Limit: Fictional cases, one guideline, one task. The paper also prints 95% accuracy in searching reference locations. Not a general NotebookLM accuracy score.

Read the NotebookLM review →

Deepgram

AA-WER (non-streaming speech-to-text), Nova-3: 5.2%

Artificial Analysis non-streaming speech-to-text leaderboard lists Deepgram Nova-3 at 5.2% AA-WER (lower is better); ElevenLabs Scribe v2 is listed at 2.2% on the same page.

Source: Speech to Text AI Model & Provider Leaderboard (Artificial Analysis). tag:src:Independent. Published ; read 2026-10-27.

Quote: “calculated as an audio-duration-weighted average of WER across ~8 hours from three datasets: AA-AgentTalk (50%), VoxPopuli-Cleaned-AA (25%), and Earnings22-Cleaned-AA (25%)”

Limit: Live leaderboard with no visible update date; the figure may change and the quote shows the method, not the 5.2% row (read via page summary).

Average WER, English lecture recordings: 8.3

Kuhn et al. (Stuttgart Media University) compared eleven ASR services on 120 three-minute samples, run 22.06.2023; Deepgram averaged 8.3 WER (English), 12.8 (LibriSpeech), 11.4 (ESL), 19.3 (German).

Source: Measuring the Accuracy of Automatic Speech Recognition Solutions (arXiv 2408.16287, ACM TACCESS). tag:src:Academic paper. Published 2024-08-29; read 2026-10-27.

Quote: “The cloud transcriptions were run on 22.06.2023.”

Limit: Mid-2023 Deepgram model version, batch mode without custom vocabulary; mostly university lectures; figures are from the paper's Table 4 as read via page summary.

Speech Arena Elo (text-to-speech), Flux TTS: 1050 Elo (95% interval -13/13), rank 65 of 98

Blind-vote arena for text-to-speech. This is Deepgram's speech-generation model, not its transcription.

Source: Artificial Analysis API. tag:src:Independent. Published 2026-10-27; read 2026-10-27.

Quote: “API fields: name "Flux TTS", elo 1050, rank 65, ci95 "-13/13".”

Limit: Listener votes on voice quality only; says nothing about Deepgram's speech-to-text accuracy, latency or price.

Read the Deepgram review →

ElevenLabs

Speech Arena Elo (text-to-speech), Eleven v4: 1336 Elo (95% interval -15/15), rank 1 of 98

Artificial Analysis ranks models by Elo from blind listener votes. Eleven v4 Turbo scores 1333 (rank 2).

Source: Artificial Analysis API. tag:src:Independent. Published 2026-10-27; read 2026-10-27.

Quote: “API fields: name "Eleven v4", elo 1336, rank 1, ci95 "-15/15".”

Limit: Votes come from Artificial Analysis listeners, not a controlled test of your use case; rank and Elo move as votes arrive.

AA-WER (non-streaming speech-to-text), Scribe v2: 2.2%

Artificial Analysis non-streaming speech-to-text leaderboard lists ElevenLabs Scribe v2 at 2.2% AA-WER (lower is better), versus 5.2% for Deepgram Nova-3.

Source: Speech to Text AI Model & Provider Leaderboard (Artificial Analysis). tag:src:Independent. Published ; read 2026-10-27.

Quote: “Measures transcription accuracy across 3 datasets to evaluate models in real-world speech.”

Limit: Concerns ElevenLabs' speech-to-text model, not text-to-speech; live board, no visible date; page does not necessarily show Scribe v2 as the current leader.

Real-time voice median latency, 24-hour: 1963.0 ms median, 1969 ms at the check

VoiceBenchmark ranks live voice agents by response time. ElevenLabs was 5th of the providers listed, status fair.

Source: Voice AI Leaderboard. tag:src:Independent. Published 2026-10-28; read 2026-10-28.

Quote: “ElevenLabs | 1969ms | 1963.0ms”

Limit: A live board. The page printed 1963.0 ms as the 24-hour median, not 1,975.0 ms. It does not print a P95 or a standard deviation. Latency only, not voice quality.

Read the ElevenLabs review →

Murf AI

Speech Arena Elo (text-to-speech), Falcon 2: 1157 Elo (95% interval -12/12), rank 27 of 98

Same blind-vote arena as above. Murf Speech Gen 2 scores 986 (rank 82).

Source: Artificial Analysis API. tag:src:Independent. Published 2026-10-27; read 2026-10-27.

Quote: “API fields: name "Falcon 2", elo 1157, rank 27, ci95 "-12/12".”

Limit: Listener votes, not a controlled test; the arena rates one model's voices, not Murf's editing studio or workflow.

Pronunciation accuracy, Speech Gen 2: 99.37%

Murf's Gen 2 API page. An independent blind test across five languages, 300,000 news sentences. Two native speakers scored each word. The separate benchmark page's heading says 99.38% and does not name Falcon or Gen 2 in that line.

Source: Gen 2 model page (Murf API docs). tag:ven:Vendor-reported. Published ; read 2026-10-28.

Quote: “achieving 99.37% pronunciation accuracy”

Limit: Vendor-reported. The listeners had no stated tie to Murf on this page, but Murf designed the test and published the number. It is not the Falcon 2 Speech Arena Elo.

Naturalness win rate against five other TTS vendors: wins 8 of 10 language comparisons; 60% vs ElevenLabs in English

Murf's own benchmark page. Blind pairs, native listeners hired through a third party, more than 11,000 pairs. Ties excluded. English (US, UK, AU, IN) win rates: AWS 68%, ElevenLabs 60%, Google 78%, Azure 71%, OpenAI 50%. German vs ElevenLabs 66%. French vs ElevenLabs 80%.

Source: TTS model benchmarking (Murf). tag:ven:Vendor-reported. Published ; read 2026-10-28.

Quote: “8 out of 10 times Murf voices win on naturalness vs. competition”

Limit: Vendor-reported. Murf chose the voices, scripts and competitors. German vs AWS is 14% on the same table, so the headline is not a win in every language.

Read the Murf AI review →

Otter.ai

Median WER, Zoom-Otter AI live transcription, psychiatric research interviews: 19.2% (IQR 15.1-24.8)

Seyedi et al. compared Amazon Transcribe (8.9%), Whisper (14.8%) and Zoom-Otter AI (19.2%) on research interviews from 65 interviewees.

Source: Comparison of Transcription Performance for Psychiatric Interviews (JMIR Mental Health 10:e48517). tag:src:Academic paper. Published 2023-10-31; read 2026-10-27.

Quote: “and lastly, Zoom-Otter AI at 19.2% (IQR 15.1-24.8)”

Limit: Tested Otter AI as embedded in Zoom (2023), not the Otter.ai app, on clinical interview audio; research-team transcripts as reference.

Word error rate vs human-corrected transcripts, Zoom speech tasks: 2.50% on the aging-well task, 3.36% on memory recall

Pfeifer, Chilton, Grilli and Mehl (Behavior Research Methods, 21 May 2024) transcribed younger (n=100) and older (n=92) adults with Otter.ai, then humans corrected the transcripts.

Source: How Ready Is Speech-to-Text for Psychological Language Research? (Behav Res Methods). tag:src:Academic paper. Published 2024-05-21; read 2026-10-28.

Quote: “an average word-error-rate (WER) of 2.50% (SD = 2.84%) in the aging well task (ST1) and 3.36% (SD = 3.3%)”

Limit: Humans corrected the Otter transcript rather than transcribing from scratch, which the authors say can lower the error rate. Quiet home recordings, one speaker. Not the 2023 Zoom-Otter live interview result of 19.2%.

Read the Otter.ai review →

Runway

Text-to-video Elo, Runway Gen-4.5 (720p): 909 Elo (95% interval -18/18), rank 24 of 32

Image-to-video: 1265 Elo, rank 23 of 75. Replaces an earlier press-report figure from November 2025.

Source: Artificial Analysis API. tag:src:Independent. Published 2026-10-27; read 2026-10-27.

Quote: “API fields: name "Runway Gen-4.5 (720p)", elo 909, rank 24.”

Limit: Blind-vote Elo on Artificial Analysis prompts, one resolution; the leaderboard has moved since the November 2025 launch coverage.

VBench total score, Gen-2 (2023-12): 80.58 total (quality 82.47, semantic 73.03)

BenchmarkList's VBench table, sampled 3 October 2026, rank 43. The row links the score to Runway's Gen-2 research page and marks it self-reported.

Source: VBench video generation leaderboard (BenchmarkList). tag:ven:Vendor-reported. Published 2026-10-03; read 2026-10-28.

Quote: “Gen-2 (2023-12) | 80.58 | 82.47% | 73.03%”

Limit: This is Runway Gen-2 from December 2023, not Gen-4.5. The aggregator labels the row self-reported. It is separate from the Artificial Analysis Elo for Gen-4.5.

Read the Runway review →

GitHub Copilot

Time to complete one JavaScript HTTP-server task: 55.8% faster with Copilot

Peng, Kalliamvakou, Cihon and Demirer ran a controlled experiment. Developers implemented an HTTP server in JavaScript. The group with GitHub Copilot finished 55.8% faster than the group without it.

Source: The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (arXiv 2302.06590). tag:src:Academic paper. Published 2023-02-13; read 2026-10-28.

Quote: “The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group.”

Limit: One task, recruited developers, 2023 Copilot. The paper does not print a 28.8% to 35.2% snippet acceptance rate, and it is not a HumanEval score.

Share of generated programs found vulnerable: approximately 40% of 1,689 programs

Pearce, Ahmad, Tan, Dolan-Gavitt and Karri prompted GitHub Copilot on 89 scenarios tied to high-risk weaknesses (MITRE Top 25). Accepted at IEEE Symposium on Security and Privacy 2022.

Source: Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions (arXiv 2108.09293). tag:src:Academic paper. Published 2021-12-16; read 2026-10-28.

Quote: “Of these, we found approximately 40% to be vulnerable.”

Limit: 2021 Copilot. The abstract's figure is approximate, across 89 scenarios and 1,689 programs. It is not a current Copilot security score.

Read the GitHub Copilot review →

Snowflake Cortex Analyst

BIRD-SQL text-to-SQL accuracy with a generated semantic model: 78% average, against 57% for Claude 3.5 Sonnet without the semantic model

Snowflake's engineering post. Four BIRD dev subsets, Claude 3.5 Sonnet in both columns: debit_card_specializing 83% vs 52%, california_schools 80% vs 63%, thrombosis_prediction 70% vs 45%, toxicology 79% vs 69%.

Source: Agentic Semantic Model Improvement (Snowflake Engineering). tag:ven:Vendor-reported. Published ; read 2026-10-28.

Quote: “Average | 57% | 78% | 21%”

Limit: Vendor-reported on a filtered BIRD subset, adapted from MySQL to Snowflake SQL. Not an independent run, and not the separate claim of 90%+ on Snowflake's own 150-question BI set.

SQL accuracy on Snowflake's internal BI questions: 90%+ for Cortex Analyst; 51% for single-prompt GPT-4o

Snowflake's engineering post. The internal set is about 150 questions on sales, marketing and finance. The same post says Cortex Analyst was nearly twice as accurate as single-prompt GPT-4o.

Source: Evaluating Text-to-SQL Accuracy for Real-World BI (Snowflake Engineering). tag:ven:Vendor-reported. Published ; read 2026-10-28.

Quote: “achieve an impressive 90%+ SQL accuracy on real-world use cases”

Limit: Vendor-reported. The 90%+ is Snowflake's own figure, and 51% is what they print for GPT-4o on their internal set. It is not the 78% BIRD subset, and it is not an independent run.

Read the Snowflake Cortex Analyst review →

Microsoft Agent Framework

Company-research pipeline, LLM-judge quality: 9.87/10 quality, 93s, 7,006 tokens, std dev 0.10

Same Grochal comparison. The table row is MS Agent Framework 1.0.0b, the highest quality and the fastest of the five.

Source: I Benchmarked 5 AI Agent Frameworks (Lukasz Grochal, DEV). tag:src:Independent. Published 2026-02-16; read 2026-10-28.

Quote: “MS Agent | 9.87 | 10.00 | 10.00 | 10.00 | 9.33 | 10.00”

Limit: Same single-author test as the LangGraph row: local Qwen 3 14B, nine runs, LLM judge. The post calls the build a 1.0.0 beta. Not an independent lab leaderboard.

Read the Microsoft Agent Framework review →

Frequently asked questions

Are these your test results?

No. Every figure was run by someone else and recorded as printed on the source page. BestSuperintelligent has not reproduced any of them, so a different version or sample can give a different number.

Why do some products have no result?

We found no independent test we could read and cite for them as of 27 October 2026. That is not evidence that a product performs well or badly, only that no usable test was found.

How is this different from the benchmark records?

Benchmark records are numbers vendors publish about their own models. These are tests run by outside parties such as academics, independent labs and journalists, so we keep the two lists apart.