TypeSafe AI’s new models work with machines, not humans

Today’s large language models are verbose, even if they’re being asked to recommend a simple decision, driving up usage costs through the sheer volume of tokens they consume or generate. Enterprises looking to incorporate AI into automated workflows will want something less verbose — both because machines are often just looking for a categorical answer, and because automated workflows are likely to result in far greater volumes of decisions, and thus token consumption, than human-mediated workflows.

TypeSafe AI, a startup founded by former OpenAI researcher and RLHF co-inventor Diogo Almeida, thinks it can help with a new LLM, Jev, which generates responses that can be consumed directly by software applications or other AI models as part of an automated workflow: which tool to invoke, which action to take next, whether a request should be approved, or when a task should be handed off to another model.

Jev takes the current state of a task or workflow as input and returns a defined decision, along with the probability of that decision, in a concise response rather than generating a long sequence of tokens to express an answer in natural language, Almeida wrote in a blog post.

In addition to being cheaper, he said, Jev can respond faster because it does not have to generate text tokens sequentially, with latency ranging from 70 milliseconds to 500 milliseconds compared with several seconds for the LLMs TypeSafe tested.

Jev could cut more than token costs

That could give IT leaders more headroom in their AI budgets, allowing them to scale agentic workloads, said David Linthicum, an independent technology consultant.

“Most enterprises are using general-purpose LLMs as a universal intelligence layer, even when the application only needs a fast, bounded decision. That is like using a full enterprise service bus to answer a yes/no routing question,” Linthicum said.

“Combine that with the dynamics of an agentic workflow, where a single task can trigger many model calls, the cost and latency of using a general-purpose LLM for every decision can quickly add up. A Jev-like model that can handle some of the bounded decisions of the workflow through probabilistic function calls gives enterprises a way to reserve the general-purpose LLMs for the parts that actually require them,” he said.

That division of labor between the models could also reduce the engineering work enterprise teams need to undertake to make general-purpose LLMs usable in automated workflows, he said.

Today, engineers may have to build layers of prompts, schemas, validation, retries, guardrails, and exception handling around general-purpose LLMs to make their free-form outputs reliable enough for software to consume, Linthicum said. “Models such as Jev, which return outputs that can be consumed directly by software or another model, could reduce that overhead for many routine decisions in a workflow,” he added.

The  benefits could extend to developers as well, particularly in how they structure agentic workflows. Instead of treating every step as a prompt-and-response interaction, developers could break an agentic application into a series of smaller decisions with defined outputs, allowing application code to determine what happens next based on the model’s response, he said.

That approach could make the control flow more explicit and easier to test, said Advait Patel, senior site reliability engineer (SRE) at Broadcom, because developers would be putting more workflow logic back into code rather than embedding it in prompts.

However, there are tradeoffs to consider, said Stephanie Walter, practice lead of AI stack at HyperFrame Research.

“Because Jev is designed to return a defined decision rather than an open-ended response, developers have to specify questions, possible outputs, thresholds, and escalation paths in advance for that part of the workflow, which could be a significant task,” she said.

Balance of probabilities

The way that Jev returns each decision also adds another concern.

Jev attaches a probability to each decision, but developers still need to determine what that probability means for a particular workflow and whether it can be relied on in production, Walter said.

For CIOs, that could raise reliability concerns when evaluating Jev for production workloads.

They would need to validate the model’s accuracy and calibration on their own data, including how it performs as that data changes over time, Walter noted.

Specifically, for enterprises in regulated industries, using Jev could draw questions around auditability, according to Paul Chada, cofounder of agentic AI startup Doozer AI.

“A probability can show how confident a model was in a decision, but it does not explain why the model made that decision. That distinction could matter when enterprises need to explain or defend automated decisions to regulators or auditors,” Chada said.

Additionally, CIOs would also need to weigh the risks of adopting a specialized model from an early-stage vendor, including security, data residency, service-level commitments, and potential vendor dependency, Patel said.

Those concerns extend to deployment, with Jev currently offered as a hosted service in a single region, Patel added.

Jev is likely to complement, not replace, ‘traditional’ LLMs

These concerns are also likely to shape how quickly enterprises adopt Jev, analysts pointed out.

“Enterprises are likely to start with internal automation and other workloads where the cost and performance benefits can be measured without impacting critical workloads,” Patel said.

However, that doesn’t necessarily mean that enterprises will replace general-purpose LLMs with Jev-like models altogether.

“Ideally, Jev could sit alongside general-purpose models in a workflow, with the latter handling open-ended reasoning, summarization and interaction and the former handling frequent structured decisions such as routing, scoring, verification, policy checks and workflow orchestration,” Linthicum said.

Jev is currently maintaining a waitlist for companies interested in trying it out. The model, according to Almeida, is priced at $0.042 per million input tokens, while output tokens are, TypeSafe said, “too cheap to meter.”

This article first appeared on InfoWorld.