Skip to content
Tiatra, LLCTiatra, LLC
Tiatra, LLC
Information Technology Solutions for Washington, DC Government Agencies
  • Home
  • About Us
  • Services
    • IT Engineering and Support
    • Software Development
    • Information Assurance and Testing
    • Project and Program Management
  • Clients & Partners
  • Careers
  • News
  • Contact
 
  • Home
  • About Us
  • Services
    • IT Engineering and Support
    • Software Development
    • Information Assurance and Testing
    • Project and Program Management
  • Clients & Partners
  • Careers
  • News
  • Contact

Tether solves the memory challenge for AI builders who don’t want to rely on the cloud

Thanks to the rise of a new generation of highly capable small AI models, on-device or local inference has become increasingly viable for a greater range of applications.   

However, limited memory capacity often prevents local AI from reaching its full potential. A model that fits comfortably on a laptop or phone can hit a wall the moment it has to hold a long conversation or search a large database.   

To advance on-device AI capabilities, Tether’s QVAC SDK tackles the memory challenge from two directions.

By integrating TurboQuant and TurboVec, the software development kit shrinks two of the biggest memory costs in a local AI pipeline: the model’s working memory during inference, and the index it searches when answering questions from your own documents.  

Compressing what AI remembers  

Modern language models generate text through a process called self-attention, in which each token (the basic units of text) gets encoded into three vectors: a query, a key, and a value. The query is what the current token is “looking for”; the key is what each earlier token offers up to be matched against that query; and the value is the actual content pulled forward once a match is found.   

AI models don’t reprocess the whole conversation for every new word. They keep a running store of key and value projections for every token already seen: the KV cache.  

Originally developed by Google researchers, TurboQuant is an open-source compression algorithm for this KV cache that transforms 16- or 32-bit vectors into smaller ones, without losing too much accuracy. The result is up to a five times reduction in the amount of memory used to store context during inference.  

Reducing the size of a model’s KV cache enables AI running on consumer hardware to process longer context prompts that would otherwise exceed local memory capacity.   

Consider a four billion parameter model like Qwen3.5-4B running at 262,000 tokens of context.   

While the model weights alone are small enough to fit on a smartphone’s VRAM, with that context, a standard 16-bit KV cache would balloon to around 8 GB.

Accounting for the model itself and the necessary overhead, that’s more than what the average consumer GPU can handle.

In contrast, by applying TurboQuant to the same model, a 262,000 token KV cache is reduced to a much more manageable 1.6 GB.  

This orientation toward everyday hardware is exactly what Tether is targeting with QVAC. For instance, TurboQuant was integrated into QVAC SDK v0.12.0 with Windows and Linux support for machines equipped with NVIDIA, AMD, and Intel GPUs.   

Compressing what AI searches  

While applying TurboQuant to a model’s KV cache streamlines AI inference, another tool in the QVAC SDK, TurboVec, applies the same quantization algorithm to a different bottleneck: vector indexing.  

To respond to user queries, modern AI tools aren’t limited to knowledge contained within their training data; they can also search for relevant information on the internet or specific databases using a technique known as retrieval-augmented generation (RAG).   

To enable this search, the system turns text into vectors, then orders them so vectors with similar meaning sit close together. Each search then compares your question’s vector against the stored ones and returns passages of text with the closest values.   

After applying TurboVec, ten million typical embedding vectors shrink from roughly 61GB to about 8GB. In terms of the necessary RAM, that’s the difference between an expensive custom setup and a standard desktop computer.  

In its original form, TurboVec was an open-source Rust project developed by Ryan Codrai, but QVAC doesn’t depend on that Rust code. Instead, the Tether team implemented the same quantizer natively in their own stack, equipping SDK users with another powerful compression tool to boost speed and reduce memory requirements.   

One algorithm, two efficiency boosts  

For QVAC, TurboQuant and TurboVec apply the same compression math to different ends of the AI pipeline. One shrinks the memory a model uses while it thinks. The other shrinks the memory it searches to find what to think about. For anyone trying to run AI on a desktop or smartphone, the respective VRAM and RAM savings mean greater capabilities for memory-constrained devices.  

By bringing RAG on to everyday edge devices, apps can plug into sensitive databases without users having to worry about potential leaks. AI can recall its own usage data and agent memory, or private documents such as legal or medical records without uploading anything to the cloud.   

Reducing cloud dependency can also save money. When every gigabyte TurboQuant and TurboVec claw back is a gigabyte that doesn’t have to be rented from a cloud provider, the long-term cost savings are significant.  

Interested in learning more? Discover how the QVAC SDK could help reduce your app’s memory footprint and slash your cloud bill.  


Read More from This Article: Tether solves the memory challenge for AI builders who don’t want to rely on the cloud
Source: News

Category: NewsNovember 18, 2026
Tags: art

Post navigation

PreviousPrevious post:Rewrite the rules of enterprise recoveryNextNext post:8 reasons why you should attend IGEL Now & Next 2027 in Fort Lauderdale

Related posts

Microsoft keeps failing. Why can’t its customers leave?
November 19, 2026
FAA ground stop offers IT leaders stark modernization lessons
November 19, 2026
The dark side of AI in hiring: When job applicants become security threats
November 19, 2026
AI leadership: 3 success lessons from CBA’s chief AI officer
November 19, 2026
Trusted AI by design, not by default
November 18, 2026
Keep skyrocketing AI costs in check
November 18, 2026
Recent Posts
  • Microsoft keeps failing. Why can’t its customers leave?
  • FAA ground stop offers IT leaders stark modernization lessons
  • The dark side of AI in hiring: When job applicants become security threats
  • AI leadership: 3 success lessons from CBA’s chief AI officer
  • Trusted AI by design, not by default
Recent Comments
    Archives
    • November 2026
    • October 2026
    • September 2026
    • August 2026
    • July 2026
    • June 2026
    • May 2026
    • April 2026
    • March 2026
    • February 2026
    • January 2026
    • December 2025
    • November 2025
    • October 2025
    • September 2025
    • August 2025
    • July 2025
    • June 2025
    • May 2025
    • April 2025
    • March 2025
    • February 2025
    • January 2025
    • December 2024
    • November 2024
    • October 2024
    • September 2024
    • August 2024
    • July 2024
    • June 2024
    • May 2024
    • April 2024
    • March 2024
    • February 2024
    • January 2024
    • December 2023
    • November 2023
    • October 2023
    • September 2023
    • August 2023
    • July 2023
    • June 2023
    • May 2023
    • April 2023
    • March 2023
    • February 2023
    • January 2023
    • December 2022
    • November 2022
    • October 2022
    • September 2022
    • August 2022
    • July 2022
    • June 2022
    • May 2022
    • April 2022
    • March 2022
    • February 2022
    • January 2022
    • December 2021
    • November 2021
    • October 2021
    • September 2021
    • August 2021
    • July 2021
    • June 2021
    • May 2021
    • April 2021
    • March 2021
    • February 2021
    • January 2021
    • December 2020
    • November 2020
    • October 2020
    • September 2020
    • August 2020
    • July 2020
    • June 2020
    • May 2020
    • April 2020
    • January 2020
    • December 2019
    • November 2019
    • October 2019
    • September 2019
    • August 2019
    • July 2019
    • June 2019
    • May 2019
    • April 2019
    • March 2019
    • February 2019
    • January 2019
    • December 2018
    • November 2018
    • October 2018
    • September 2018
    • August 2018
    • July 2018
    • June 2018
    • May 2018
    • April 2018
    • March 2018
    • February 2018
    • January 2018
    • December 2017
    • November 2017
    • October 2017
    • September 2017
    • August 2017
    • July 2017
    • June 2017
    • May 2017
    • April 2017
    • March 2017
    • February 2017
    • January 2017
    Categories
    • News
    Meta
    • Log in
    • Entries feed
    • Comments feed
    • WordPress.org
    Tiatra LLC.

    Tiatra, LLC, based in the Washington, DC metropolitan area, proudly serves federal government agencies, organizations that work with the government and other commercial businesses and organizations. Tiatra specializes in a broad range of information technology (IT) development and management services incorporating solid engineering, attention to client needs, and meeting or exceeding any security parameters required. Our small yet innovative company is structured with a full complement of the necessary technical experts, working with hands-on management, to provide a high level of service and competitive pricing for your systems and engineering requirements.

    Find us on:

    FacebookTwitterLinkedin

    Submitclear

    Tiatra, LLC
    Copyright 2016. All rights reserved.