Nvidia says NeMo microservices now generally available

Nvidia today announced the general availability of neural module (NeMo) microservices, a modular platform for building and customizing gen AI models and AI agents.

NeMo microservices integrate with partner platforms to provide features including prompt tuning, supervised fine-tuning, and knowledge retrieval tools.

Nvidia views its NeMo microservices as the building blocks that help enterprises build data flywheels, or feedback loops where data is collected from interactions or processes and then used to continuously refine AI models, leading to better interactions and processes.

Joey Conway, the company’s senior director for generative AI software for enterprise, says data flywheels enable enterprise IT to onboard AI agents as digital teammates that tap into user interactions and AI-generated data from inferences to continuously improve model performance.

The system processes data from interactions, uses it to customize the model powering an AI agent, evaluates the model to ensure it’s improved in skills, then deploys that updated model with guardrails to keep it focused and on topic, and improves information retrieval to maximize accuracy.

“Our view is that every [AI] agent will need a data flywheel,” Conway said in a press conference on Tuesday. “The data flywheel is the way we can go from enterprise data — things like inference data, business intelligence, and user feedback — to power and improve the agent so it gains new capabilities and skills, and learns from its experiences.”

The NeMo microservices made generally available Wednesday include:

• NeMo Data Curator for gathering enterprise data, classifying it, and deduplicating it.
• NeMo Customizer for fine-tuning. This microservice uses post-training techniques like supervised fine-tuning and low-rank adoption.
• NeMo Evaluator for evaluating AI models and workflows based on custom and industry benchmarks.
• NeMo Guardrails for improving compliance protection with safety and security measures that align with organizational policies and guidelines.
• NeMo Retriever for building multimodal extraction, reranking, and embedding pipelines that improve accuracy of answers.

Agents at the ready

Conway noted that these microservices are reaching general availability as enterprises are starting to build large-scale, multi-agent systems with hundreds of specialized agents assisting human employees across nearly all work functions.

“This enterprise-wide impact positions AI agents as a trillion-dollar opportunity — with applications spanning automated fraud detection, shopping assistants, predictive machine maintenance, and document review — and underscores the critical role data flywheels play in transforming data into actionable insights,” Conway wrote in a blog post earlier today.

He also detailed how Nvidia partners are leveraging NeMo microservices in their AI agent platforms. AT&T, for example, is using agentic AI to support its call centers. Working with Arize and Quantiphi, AT&T has built an AI agent that leverages NeMo microservices to process its knowledge base of nearly 10,000 documents, which are refreshed weekly. Conway noted that NeMo microservices helped improve the agent’s accuracy by 40%, which also lowered the compute overhead. Then there’s BlackRock, which is leveraging NeMo microservices for agentic AI capabilities in its Aladdin tech platform, unifying the company’s investment management process through a common data language. Cisco’s Outshift team, working with partner Galileo, is also using NeMo microservices for a coding assistant, resulting in 40% fewer tool selection errors and up to 10 times faster response times.

Conway further added that NeMo microservices support open models including Llama, the Microsoft Phi family of small language models, Google Gemma, Mistral, and Llama Nemotron Ultra. Meta offers new connectors for Meta Llamastack, and AI software providers including Cloudera, Datadog, Dataiku, DataRobot, DataStax, SuperAnnotate, and Weights & Biases — have integrated NeMo microservices into their platforms.