FinCosmosLeaderboard
An open-world evaluation paradigm for agents doing financial work.
Main results
Each agent works on every task three times, each time in a new session with a simulated investor who holds a hidden strategy and reveals it only when asked. A session is accepted only when the user is satisfied and none of 15 veto checks fails. Agents are ranked by acceptance rate, then by pre-veto quality; with no accepted session yet, quality decides the order, and overlapping intervals mean the agents cannot be told apart. Hover over a column header for its definition.
| # | Agent | Sat.% ↑ | AC% ↑ | pass² ↑ | Quality0-100 ↑ | Data0-100 ↑ | Strategy0-100 ↑ | Backtest0-100 ↑ | Assess.0-100 ↑ | UX0-100 ↑ | Vetoes# ↓ | Cov.% ↑ | Cut% ↓ | TokensM ↓ | CostUSD ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | CodexGPT-6.1 Sol | 33 | 0 | 0 | 37.2 [30.6, 43.3] | 45.8 | 33.4 | 31.9 | 17.2 | 57.6 | 7.2 | 64 | 69 | 3.20 | 0.80 |
| 2 | Claude CodeClaude Opus 5.5 | 30 | 0 | 0 | 37.1 [27.8, 45.0] | 44.7 | 38.2 | 26.9 | 23.2 | 52.2 | 7.4 | 54 | 65 | 1.13 | 2.24 |
| 3 | Kimi CodeKimi K3 (open weights) | 20 | 0 | 0 | 22.3 [14.3, 31.6] | 26.8 | 23.4 | 16.5 | 6.5 | 38.2 | 10.1 | 57 | 74 | 0.89 | 0.83 |
| 4 | Gemini CLIGemini 3.8 Flash | 33 | 0 | 0 | 22.2 [16.9, 27.4] | 25.3 | 30.8 | 15.6 | 7.0 | 32.4 | 9.3 | 57 | 75 | 3.51 | 0.80 |
| Bare modelClaude Opus 5.5, no tools | 0 | 0 | 0 | 20.2 [18.8, 21.5] | 0.0 | 49.8 | 4.6 | 1.8 | 44.6 | 9.1 | 41 | 0 | 0.01 | 0.24 |
Bold: best agent in the column. The bare model answers the same requests without tools and is shown as a reference, not ranked.
Results by task
Mean pre-veto quality over the three sessions of each task; the small numbers are accepted / satisfied sessions. Difficulty is the sum of the difficulty of the six strategy components (0–18).
| Task | Difficulty | Persona | Codex | Claude Code | Kimi Code | Gemini CLI | Bare model |
|---|---|---|---|---|---|---|---|
| Task 0 | 15 | Generalist analyst | 18.0 0/0 | 39.4 0/1 | 31.8 0/1 | 13.3 0/0 | 17.5 0/0 |
| Task 1 | 11 | Retail investor | 32.0 0/1 | 45.5 0/1 | 13.7 0/0 | 8.3 0/0 | 20.5 0/0 |
| Task 2 | 10 | Retail investor | 46.5 0/2 | 50.2 0/1 | 42.1 0/2 | 21.3 0/1 | 23.8 0/0 |
| Task 3 | 7 | Retail investor | 53.8 0/3 | 36.2 0/1 | 50.4 0/2 | 32.7 0/1 | 20.6 0/0 |
| Task 4 | 16 | Retail investor | 34.6 0/1 | 15.0 0/0 | 5.1 0/0 | 32.0 0/2 | 19.0 0/0 |
| Task 5 | 11 | Biotech specialist | 32.5 0/1 | 33.2 0/1 | 12.4 0/0 | 18.7 0/1 | 21.3 0/0 |
| Task 6 | 14 | Biotech specialist | 31.9 0/0 | 49.5 0/2 | 7.2 0/0 | 19.9 0/1 | 20.5 0/0 |
| Task 7 | 12 | Generalist analyst | 50.6 0/1 | 40.5 0/1 | 20.0 0/0 | 13.8 0/0 | 15.6 0/0 |
| Task 8 | 15 | Retail investor | 38.5 0/0 | 52.9 0/1 | 22.4 0/1 | 32.5 0/2 | 21.1 0/0 |
| Task 9 | 17 | Generalist analyst | 33.6 0/1 | 8.1 0/0 | 18.0 0/0 | 29.8 0/2 | 21.8 0/0 |
Veto failures
Share of sessions (%) in which each veto fails. One failed veto rejects a session.
| Agent | 1.1 | 2.1 | 2.2 | 2.3 | 2.4 | 2.5 | 2.6 | 2.7 | 3.1 | 3.2 | 3.7 | 3.8 | 4.1 | 5.1 | 5.6 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Codex | 20 | 57 | 33 | 47 | 37 | 40 | 10 | 57 | 23 | 83 | 97 | 100 | 47 | 67 | 3 |
| Claude Code | 63 | 47 | 37 | 30 | 27 | 23 | 3 | 53 | 30 | 100 | 100 | 100 | 43 | 73 | 7 |
| Kimi Code | 70 | 63 | 47 | 57 | 57 | 53 | 13 | 67 | 67 | 100 | 100 | 100 | 80 | 83 | 50 |
| Gemini CLI | 63 | 43 | 33 | 27 | 33 | 33 | 10 | 93 | 70 | 97 | 100 | 100 | 90 | 100 | 33 |
| Bare model | 100 | 23 | 17 | 10 | 17 | 20 | 0 | 27 | 100 | 100 | 100 | 100 | 97 | 100 | 100 |
- 1.1 Point-in-time strictness
- 2.1 Stock selection
- 2.2 Market timing
- 2.3 Portfolio construction
- 2.4 Risk control
- 2.5 Execution
- 2.6 Transaction cost
- 2.7 Strategy ambiguity
- 3.1 Point-in-time correctness
- 3.2 Independent recomputation
- 3.7 Verifier audit
- 3.8 Process completeness
- 4.1 Grounded conclusions
- 5.1 Conversation experience
- 5.6 Internal leak
How entries are evaluated
Setting
- The user opens with a vague request; the details of the strategy surface only when the agent asks.
- The agent receives no data. It finds data on the open Web, writes its own code, and runs its own backtest.
- Each task sets a data cutoff. Later data stays online, so the agent must keep its work point-in-time.
- Only what the user receives is graded: messages and delivered files.
Grading
- 45 dimensions in five categories, graded by programs and an LLM judge, with no reference answer.
- Programs recompute reported metrics and check point-in-time correctness; the delivered strategy is checked against the hidden one.
- 15 dimensions are vetoes: failing any one rejects the session.
- Feedback to the agent never reveals the hidden strategy.
Rules for entries
- Agents are frozen: no change between sessions and no tuning on these tasks.
- Results are reported per evaluator version and are never mixed across versions.
- Agents that learn across sessions will be listed separately.
- Submissions open after the review period.