Jev has me thinking about router latency

Jev is fast AF. What about the routers?

Jev is now available through OpenRouter and Vercel AI Gateway, not just directly from TypeSafe.

Usually, with a classic LLM, you’re waiting a while for an answer. At least long enough that the overhead of going through a router basically doesn’t fucking matter. Jev though, Jev is fast enough that you can put it in more hot paths, where the rest of your app is waiting on that decision.

So if a request only takes ~100 ms, how much does going through a router add? Let’s fuck around and find out.

The results

This isn’t a scientific test, I’m really just looking for an excuse to take GPT-6 Sol for a spin. I ran 3,000 requests from my Mac.

Noul, Choice, and Score each got batches of 1, 4, and 16 questions, plus a mixed batch of 16. Each workload ran 100 times per route across two sessions, one request at a time, with provider order balanced within each comparison. Timing ended after the full response arrived.

Across those ten workloads, OpenRouter’s median extra wait was 31–48 ms, comparing each request with its matching direct call. Every request through OpenRouter and TypeSafe completed.

Here’s one example: the batch containing six Noul questions, five Choice questions, and five Score questions. These medians describe successful requests only:

Route	Median total latency	Completed requests
TypeSafe directly	207 ms	100 / 100
OpenRouter	245 ms	100 / 100
Vercel AI Gateway	297 ms	35 / 100

Vercel only completed 389 of its 1,000 requests. Another 593 hit a rate limit, and 18 returned server errors. I guess Vercel is rate limiting since they’re giving it away at the moment. Which is neat, I guess. Even when I slowed the test I got 503s. I get the rate limit, but 🤷 TypeSafe and OpenRouter completed every request.

This isn’t an uptime test, so I’ll let you draw your own conclusions.

The chart shows each workload’s median matched delay and the percentage of requests that completed in this run. Vercel’s latency points include only the requests that succeeded.

When would I care about those milliseconds?

You could use Jev to change a UI on the fly as someone interacts with it. Maybe the user describes what they’re trying to do, and Jev picks which controls or suggestions to show next. You’re asking one or two small questions, and the person is waiting for the interface to respond.

Or put Jev behind an if in a request router: can this go to a cheap model, or does it need the bigger one? Every request waits for that decision before the actual work starts.

Those are the cases I’d pay attention to: small batches, called frequently, where keeping latency down helps the app feel responsive. An extra few dozen milliseconds might matter there. I’d measure that interaction before deciding whether to go direct.

If you use OR, keep using it

If you’re already using OpenRouter, these results wouldn’t make me switch. It added a few dozen milliseconds here, and every request completed. For the background evals in the previous FAFO post, I’d happily take that extra wait to keep model access, billing, and logs in one place.

If Jev becomes something my app calls repeatedly while the user waits, I’d measure the whole workflow and see whether going direct helps enough to matter. Until then, I’d keep using the router.

The full benchmark report has the results for every workload and the measurement limitations. The Go benchmark is in the FAFO repo if you want to run it yourself.