A Weekend With Jev: Cheaper Decisions, Real Tradeoffs

I Wanted Cheaper Decisions

I spent this weekend experimenting with Jev in our recruiting platform. The question was straightforward: where were we using a language model to make a small decision that software could consume directly?

We had some obvious places to try. Before an email enters a more involved workflow, a model decides whether it is relevant and what kind of message it is. That happens in both recruiting operations and sales workflows. Those first decisions do not need a generated explanation every time.

I wanted to find out whether Jev could do that work cheaply and quickly enough to justify giving up some accuracy. Our evals did not establish that it was the smartest option. I was still willing to make the tradeoff.

As of September 19, the implementation is in pull-request review. These are results from this weekend's experiments, not a report of long-term production savings.

What Made Jev Interesting

TypeSafe AI introduced Jev on September 15 as its first System One model. You give it state and typed questions. It returns choices, scores, or boolean probabilities rather than generating prose.

That interface fits a lot of ordinary application code. A classifier needs to return a category. A router needs to choose from known destinations. The application already knows what to do with those answers.

We were already using structured output with language models, so the useful distinction was not simply getting JSON back. Jev is built specifically for these decisions. I wanted to see what that changed about latency, price, and the mistakes we had to handle.

A valid category can still be the wrong category. Returning the right shape does not settle whether the decision is useful.

Two Places We Put It to Work

The first was our recruiting email classifier. Its job is to recognize messages that belong in a recruiting workflow and categorize them before the more capable agent gets involved. We changed that first classification step to Jev.

The second was our sales email classifier. It has different categories and different rules about which messages should proceed. The same separation still works: Jev supplies the classification, and code decides what happens next.

The downstream agents still have their jobs. Reading context, working through a request, using tools, and writing something useful can require a language model. We kept that work separate from the initial classification.

At a high level, the flow is:

Incoming message
    -> Jev classifies it
    -> Existing application rules decide whether to continue
    -> A language-model agent handles the more involved work

That gave us a small part of each workflow we could change and test without redesigning the whole application.

The Evals Did Not Give Us a Clean Win

We tested against existing email datasets and kept the labels and quality targets in place.

On the recruiting set, Jev identified 59 of 63 relevant messages, or about 93.7% recall. On the sales set, it identified 167 of 183, or about 91.3%. Both missed our existing 98% recall target.

I did not want to redefine the target and call that a pass. The result was a quality tradeoff, and I accepted it for this experiment.

False positives matter too. A classifier that sends too much irrelevant material to the next model can eat into the savings. False negatives are a different problem: a message that gets filtered out never reaches that next stage.

That is why I would not describe this as a drop-in upgrade in intelligence. The question for me was whether the speed and economics made it worth trying in these bounded steps. For now, my answer is yes. Production experience still has to tell us whether that decision holds up.

The Price Was Worth Investigating

TypeSafe's published launch pricing was $0.042 per million input tokens, with no output-token charge. That made repeated classification experiments inexpensive enough to explore.

In a separate 100-item structured-evaluation replay this weekend, we used about 296,000 input tokens. At that catalog rate, the estimated model cost was roughly 1.2 cents. Median observed call latency was 257 milliseconds; the sample's 95th percentile was about 2.4 seconds, with five calls in flight at a time.

Those are measurements from one workload, not an end-to-end speed comparison with our previous email pipeline. The price calculation is a catalog estimate, not a production invoice. I want to measure the whole workflow too, including downstream calls and failures.

Still, it changes which experiments feel worthwhile. When a small judgment costs very little and usually comes back quickly, I am more willing to try it in places where an LLM call previously felt excessive.

Keep the Integration Ordinary

During the PR, we moved the integration onto AI SDK 7's native evaluation API through our existing Vercel AI Gateway setup. Jev is available there as typesafe-ai/jev.

That let us use the model without maintaining another standalone client and credential path. The application still owns the workflow, the retries, and the decision about what to do when a provider is unavailable.

For these email classifiers, the final fallback is Luna. Jev gets the first attempt and its retries. If it cannot return a valid result, Luna takes over. We kept the existing classification prompts, output contracts, and privacy handling around that boundary.

That fallback handles unavailable or invalid responses. It does not catch a confident, well-formed classification that happens to be wrong. Those errors still count in the evals.

What I Am Taking From the Weekend

I started by looking for places to try a new model. I ended up spending a lot of time deciding which parts of a workflow really needed language generation.

Jev made sense for short, typed decisions with clearly defined consequences. The more capable language model remained useful for the work that followed. Code connected the two.

I am comfortable choosing a cheaper, faster decision step that performs worse on some of our evals, provided I understand the misses and keep the evidence visible. I do not need every model in the system to be the smartest model available.

I do need to know what job it is doing, what happens when it is wrong, and whether the tradeoff is still worthwhile once people are using it.