We gave 8 frontier models 40 tiny, exact tasks — hash a string, cross a half-hour DST boundary, amortize a 289-month loan. Unaided they averaged 68%. With TinyFn's MCP tools the same models scored 99%, and 6 of 8 were perfect.
Same model, same prompt, same grader — the only difference is tool access.
Wrong answers aren't approximate — they're confidently, plausibly wrong.
it saidc0b566d7ae8a9df97a3f8a9c04652c8eea4ea751c9c32b4143f9840c9bf69ebd truth700a72e1c8e7eb1aed4cfc76f4d45d0a7a8553cb528bf4b715b4ba03af2da375
it said3f5a1c9e8b7d2064af13c5e9d78b2a41 truth102723b946d6d281adcf4ba9af3e7529
4oaS → 4oKs)it said…IGZpbmFsOiDigKw1My44MzsgcmVjZWlwdCD… truth…IGZpbmFsOiDigqw1My44MzsgcmVjZWlwdCD…
it said89137 truth89213
it said43 truth44
it said2025-10-04 12:30 truth2025-10-04 13:30
How many of the 8 models solved each task — unaided vs. with TinyFn tools.
| Task | Category | Unaided | With TinyFn |
|---|---|---|---|
| md5 | hash | 0/8 | 8/8 |
| sha256 | hash | 0/8 | 8/8 |
| hmac-sha256 | hash | 0/8 | 8/8 |
| crc32 | hash | 1/8 | 8/8 |
| nth-prime | number | 1/8 | 8/8 |
| rot13 | encode | 1/8 | 7/8 |
| base64-encode | encode | 3/8 | 8/8 |
| collatz-length | number | 5/8 | 8/8 |
| count-th | string | 5/8 | 8/8 |
| sample-stddev | stats | 5/8 | 8/8 |
| haversine | geo | 5/8 | 8/8 |
| base64-decode | encode | 5/8 | 7/8 |
| regex-double-letters | regex | 5/8 | 7/8 |
| unix-weekday | datetime | 6/8 | 8/8 |
| iso-to-unix | datetime | 6/8 | 8/8 |
| tz-half-hour-dst | time | 6/8 | 8/8 |
| miles-to-km | convert | 6/8 | 8/8 |
| liters-to-uk-gallons | convert | 6/8 | 8/8 |
| dec-to-hex | convert | 6/8 | 8/8 |
| big-multiply | math | 6/8 | 8/8 |
| gcd | math | 6/8 | 8/8 |
| invert-then-hsl | color | 6/8 | 8/8 |
| pearson-correlation | stats | 6/8 | 8/8 |
| geometric-mean | stats | 6/8 | 8/8 |
| loan-payment | finance | 6/8 | 8/8 |
| compound-daily | finance | 6/8 | 8/8 |
| days-between-century | datetime | 7/8 | 8/8 |
| iso-week-edge | datetime | 7/8 | 8/8 |
| add-1000-business-days | time | 7/8 | 8/8 |
| hex-encode | encode | 7/8 | 8/8 |
| factorial-47 | math | 7/8 | 8/8 |
| contrast-ratio | color | 7/8 | 8/8 |
| bmr-tdee | health | 7/8 | 8/8 |
| add-10000-days | datetime | 8/8 | 8/8 |
| weekday-1834 | datetime | 8/8 | 8/8 |
| tz-kathmandu-chatham | time | 8/8 | 8/8 |
| url-encode | encode | 8/8 | 8/8 |
| luhn-19-digit | validate | 8/8 | 8/8 |
| percentile-37-5 | stats | 8/8 | 8/8 |
| broadcast-address | ip | 8/8 | 8/8 |
Design. Every model runs every task twice with identical prompts:
unaided (no tools) and with TinyFn (the task's TinyFn MCP category
server, e.g. api.tinyfn.io/mcp/hash/, attached as a toolset via
pydantic-ai, models served through OpenRouter).
Grading. Ground truth is computed with the Python standard library,
independently of the TinyFn API. Checkers normalize case, whitespace, and number
formats, with tolerances on rounded values, and answers must arrive on an explicit
ANSWER: line — so verbose models aren't penalized and lucky prose
doesn't count. Tool use is verified from message history.
Full transparency. The harness, all 40 task definitions, raw per-evaluation results, and instructions to reproduce are on GitHub: github.com/tinyfn-io/tinybench.
The benchmark keeps score on both sides of the API. Building it
exposed three real bugs in TinyFn itself — time/convert
silently treating IANA timezone names as UTC, math/multiply
losing precision above 253, and four length/speed endpoints
using rounded conversion factors. All three are fixed and deployed; the
tasks that caught them are still in the suite.
The three remaining tool-condition misses are reported as-is: claude-sonnet-5's ROT13 run hit a provider content filter (high-entropy output, not a tool failure), and claude-haiku-4.5 mis-transcribed one character of a correct Base64 result and miscounted a correct regex result. Nothing is excluded from the totals.
One MCP URL. 500+ deterministic tools. No more plausible-looking hashes.