Leaderboard (expand a row for run history)
| # | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | openai-codex/gpt-5.5 | openai-codex/gpt-5.5 | 54/54 | 3588.2 | 10518.0 | 6747 | 364,314 | 14 | 2026-06-29 | |
| 2 | zai/glm-5.2 | zai/glm-5.2 | 53/54 | 5029.4 | 10625.9 | 8122 | 438,597 | 13 | 2026-06-29 | |
| 3 | framework/ornith | framework/ornith | 52/54 | 956.5 | 2225.8 | — | — | 1 | 2026-06-29 | |
| 4 | foundry/Qwen3.6-35B-A3B-FP8 | foundry/Qwen3.6-35B-A3B-FP8 | 51/54 | 2257.4 | 5113.6 | — | — | 15 | 2026-06-29 | |
| 5 | foundry/diffusiongemma-26B-A4B-it-NVFP4 | foundry/diffusiongemma-26B-A4B-it-NVFP4 | 50/54 | 704.7 | 1801.6 | — | — | 2 | 2026-06-29 |
Model × case heatmap (green = pass, red = fail; hover a cell for latency/tokens/error)
| Model | ast_grep_includes_language_hint | ast_grep_rewrite_preview | ast_grep_selects_ast_grep_for_structural_search | ast_grep_with_path_and_limit | bash_includes_timeout_argument | bash_preserves_quotes_and_pipe_in_command | bash_selects_bash_for_shell_command | bash_uses_canonical_command_not_cmd_alias | browser_connect_mode_with_url | browser_includes_headed_mode | browser_passes_command_argument | browser_selects_browser_for_page_open | code_search_high_max_tokens_edge_case | code_search_includes_max_tokens | code_search_passes_query_argument | code_search_selects_code_search_for_api_docs | edit_allows_multiple_matches | edit_passes_path_and_edits_array | edit_selects_edit_for_text_replacement | edit_supports_multiple_replacements_in_one_call | find_includes_kind_and_limit | find_includes_mixed_kind_and_offset | find_passes_required_query | find_selects_find_for_path_search | grep_includes_path_limit_and_context | grep_passes_required_pattern | grep_selects_grep_for_content_search | grep_zero_context_edge_case | read_multiple_files_in_one_call | read_single_file_selects_read | read_single_file_uses_paths_array | read_slice_includes_offset_and_limit | skill_includes_reference_path | skill_passes_name_argument | skill_selects_skill_for_loading_guidance | subagents_includes_concurrency | subagents_includes_optional_task_fields | subagents_passes_tasks_with_intent | subagents_selects_subagents | view_image_passes_required_prompt | view_image_selects_view_image_for_image_inspection | view_image_with_absolute_path | web_fetch_includes_max_tokens | web_fetch_passes_url_argument | web_fetch_selects_web_fetch_for_url_content | web_search_domain_filter | web_search_duplicate_query_normalization | web_search_num_results | web_search_queries_array | web_search_recency_filter | web_search_single_query | write_passes_path_and_content | write_preserves_multiline_content | write_selects_write_for_new_file | Accuracy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| openai-codex/gpt-5.5 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 100% |
| zai/glm-5.2 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 98% |
| framework/ornith | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 96% |
| foundry/Qwen3.6-35B-A3B-FP8 | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 94% |
| foundry/diffusiongemma-26B-A4B-it-NVFP4 | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 93% |
Tokens per case (green = fewer, red = more; right column = each run's total)
| Model | ast_grep_includes_language_hint | ast_grep_rewrite_preview | ast_grep_selects_ast_grep_for_structural_search | ast_grep_with_path_and_limit | bash_includes_timeout_argument | bash_preserves_quotes_and_pipe_in_command | bash_selects_bash_for_shell_command | bash_uses_canonical_command_not_cmd_alias | browser_connect_mode_with_url | browser_includes_headed_mode | browser_passes_command_argument | browser_selects_browser_for_page_open | code_search_high_max_tokens_edge_case | code_search_includes_max_tokens | code_search_passes_query_argument | code_search_selects_code_search_for_api_docs | edit_allows_multiple_matches | edit_passes_path_and_edits_array | edit_selects_edit_for_text_replacement | edit_supports_multiple_replacements_in_one_call | find_includes_kind_and_limit | find_includes_mixed_kind_and_offset | find_passes_required_query | find_selects_find_for_path_search | grep_includes_path_limit_and_context | grep_passes_required_pattern | grep_selects_grep_for_content_search | grep_zero_context_edge_case | read_multiple_files_in_one_call | read_single_file_selects_read | read_single_file_uses_paths_array | read_slice_includes_offset_and_limit | skill_includes_reference_path | skill_passes_name_argument | skill_selects_skill_for_loading_guidance | subagents_includes_concurrency | subagents_includes_optional_task_fields | subagents_passes_tasks_with_intent | subagents_selects_subagents | view_image_passes_required_prompt | view_image_selects_view_image_for_image_inspection | view_image_with_absolute_path | web_fetch_includes_max_tokens | web_fetch_passes_url_argument | web_fetch_selects_web_fetch_for_url_content | web_search_domain_filter | web_search_duplicate_query_normalization | web_search_num_results | web_search_queries_array | web_search_recency_filter | web_search_single_query | write_passes_path_and_content | write_preserves_multiline_content | write_selects_write_for_new_file | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| openai-codex/gpt-5.5 | 6,158 | 6,159 | 6,155 | 6,158 | 6,139 | 6,170 | 6,144 | 6,149 | 6,192 | 6,156 | 6,152 | 6,151 | 6,185 | 6,152 | 6,146 | 6,147 | 6,199 | 6,179 | 12,399 | 6,237 | 6,155 | 6,152 | 6,148 | 6,148 | 6,155 | 6,152 | 6,142 | 6,153 | 12,385 | 12,359 | 12,361 | 12,361 | 6,141 | 6,140 | 6,140 | 6,286 | 6,222 | 6,273 | 6,211 | 6,192 | 6,174 | 6,195 | 6,170 | 6,149 | 6,167 | 6,189 | 6,252 | 6,185 | 6,196 | 6,207 | 6,191 | 6,145 | 6,149 | 6,142 | 364,314 |
| zai/glm-5.2 | 7,149 | 7,198 | 7,218 | 7,166 | 7,137 | 7,176 | 7,149 | 7,157 | 7,200 | 7,147 | 7,130 | 7,194 | 7,160 | 7,152 | 7,138 | 7,150 | 14,372 | 7,190 | 14,344 | 14,429 | 7,147 | 7,168 | 7,128 | 7,134 | 7,172 | 7,137 | 7,122 | 7,167 | 14,478 | 14,309 | 14,365 | 14,560 | 7,139 | 7,140 | 7,137 | 7,439 | 7,546 | 7,379 | 7,515 | 7,191 | 7,165 | 7,215 | 7,170 | 7,156 | 7,154 | 7,178 | 7,172 | 7,148 | 7,166 | 7,168 | 7,153 | 7,145 | 7,128 | 7,150 | 438,597 |
| framework/ornith | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · |
| foundry/Qwen3.6-35B-A3B-FP8 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · |
| foundry/diffusiongemma-26B-A4B-it-NVFP4 | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · | · |
Raw runs & metadata (45)
| Model | Timestamp | Backend | Cases dir | Accuracy |
|---|---|---|---|---|
| zai/glm-5.2 | 2026-06-29 14:57:31 | zai/glm-5.2 | evals/magi-code-evals/cases | 98% |
| framework/ornith | 2026-06-29 14:52:21 | framework/ornith | evals/magi-code-evals/cases | 96% |
| foundry/diffusiongemma-26B-A4B-it-NVFP4 | 2026-06-29 14:51:19 | foundry/diffusiongemma-26B-A4B-it-NVFP4 | evals/magi-code-evals/cases | 93% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-29 14:50:27 | foundry/Qwen3.6-35B-A3B-FP8 | evals/magi-code-evals/cases | 94% |
| openai-codex/gpt-5.5 | 2026-06-29 14:48:00 | openai-codex/gpt-5.5 | evals/magi-code-evals/cases | 100% |
| zai/glm-5.2 | 2026-06-27 21:25:18 | zai/glm-5.2 | evals/magi-code-evals/cases | 94% |
| foundry/diffusiongemma-26B-A4B-it-NVFP4 | 2026-06-27 21:19:25 | foundry/diffusiongemma-26B-A4B-it-NVFP4 | evals/magi-code-evals/cases | 91% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-27 21:18:34 | foundry/Qwen3.6-35B-A3B-FP8 | evals/magi-code-evals/cases | 93% |
| openai-codex/gpt-5.5 | 2026-06-27 21:16:06 | openai-codex/gpt-5.5 | evals/magi-code-evals/cases | 98% |
| openai-codex/gpt-5.5 | 2026-06-27 21:08:46 | openai-codex/gpt-5.5 | evals/magi-code-evals/cases | 100% |
| openai-codex/gpt-5.5 | 2026-06-27 21:03:43 | openai-codex/gpt-5.5 | evals/magi-code-evals/cases | 100% |
| zai/glm-5.2 | 2026-06-27 02:37:20 | zai/glm-5.2 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 98% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-27 01:51:03 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 94% |
| zai/glm-5.2 | 2026-06-27 01:13:02 | zai/glm-5.2 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 98% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-27 01:08:12 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 96% |
| openai-codex/gpt-5.5 | 2026-06-27 01:05:43 | openai-codex/gpt-5.5 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 100% |
| zai/glm-5.2 | 2026-06-27 00:40:27 | zai/glm-5.2 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 96% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-27 00:35:29 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 96% |
| openai-codex/gpt-5.5 | 2026-06-27 00:32:52 | openai-codex/gpt-5.5 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 98% |
| zai/glm-5.2 | 2026-06-26 23:48:36 | zai/glm-5.2 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 91% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-26 23:42:25 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 94% |
| openai-codex/gpt-5.5 | 2026-06-26 23:39:55 | openai-codex/gpt-5.5 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 98% |
| zai/glm-5.2 | 2026-06-26 23:00:49 | zai/glm-5.2 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 89% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-26 22:55:05 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 89% |
| zai/glm-5.2 | 2026-06-26 22:49:23 | zai/glm-5.2 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 87% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-26 22:44:35 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 76% |
| zai/glm-5.2 | 2026-06-26 22:37:34 | zai/glm-5.2 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 83% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-26 22:32:04 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 89% |
| openai-codex/gpt-5.5 | 2026-06-26 22:29:54 | openai-codex/gpt-5.5 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 96% |
| zai/glm-5.2 | 2026-06-26 22:13:05 | zai/glm-5.2 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 91% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-26 22:06:45 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 83% |
| openai-codex/gpt-5.5 | 2026-06-26 22:04:39 | openai-codex/gpt-5.5 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 91% |
| zai/glm-5.2 | 2026-06-26 21:22:45 | zai/glm-5.2 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 89% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-26 21:16:21 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 83% |
| openai-codex/gpt-5.5 | 2026-06-26 21:14:11 | openai-codex/gpt-5.5 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 91% |
| zai/glm-5.2 | 2026-06-26 19:21:44 | zai/glm-5.2 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 83% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-26 19:16:00 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 78% |
| openai-codex/gpt-5.5 | 2026-06-26 19:13:56 | openai-codex/gpt-5.5 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 98% |
| zai/glm-5.2 | 2026-06-26 18:33:47 | zai/glm-5.2 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 81% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-26 18:29:04 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 80% |
| openai-codex/gpt-5.5 | 2026-06-26 18:27:05 | openai-codex/gpt-5.5 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 94% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-26 18:23:23 | foundry/Qwen3.6-35B-A3B-FP8 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 0% |
| openai-codex/gpt-5.5 | 2026-06-26 18:23:16 | openai-codex/gpt-5.5 | /Users/magimetal/Dev/magi-code/evals/magi-code-evals/cases | 94% |
| foundry/Qwen3.6-35B-A3B-FP8 | 2026-06-23 22:02:26 | foundry/Qwen3.6-35B-A3B-FP8 | evals/magi-code-evals/cases | 84% |
| openai-codex/gpt-5.5 | 2026-06-23 22:00:23 | openai-codex/gpt-5.5 | evals/magi-code-evals/cases | 98% |