Edge NPU & AI Models
NPU and AI Model Optimization Platform
A platform that quantizes open AI models to run on edge NPUs and optimizes the kernels and runtime as well.
Development centers on the Tenstorrent Tensix NPU in our robot development kits, and we also publish quantized models for GPUs, CPUs and Apple silicon.

Field challenges
Cloud AI is hard to use where the network drops or where data cannot leave the site. Edge NPUs differ from GPUs in memory capacity and number formats, so an open model moved over as is either does not fit in device memory or loses accuracy.
How it works
We quantize the weights to the NPU's block floating-point formats with GPTQ, a published quantization method, and choose the precision per layer to fit device memory. Our own NPU kernels and runtime run every model operation on the NPU.
Target models
- LLMs
- VLMs
- MoE models
- Hybrid linear attention models
- Speech recognition
- Speech synthesis
System layout
The host only streams weights, tokenizes and schedules; every model operation runs on the NPU.

- Host PC
- NPU accelerator
- Robot compute bay
- Weight heat map
01
Block floating-point quantization
A tool that quantizes open models from Hugging Face to BFP4 and BFP8, the Tensix NPU's block floating-point formats. It uses GPTQ, a published quantization algorithm, and saves the result as an ordinary checkpoint, so GPUs and CPUs can check the same weights the NPU runs.
- BFP conversion that matches Tenstorrent's host packer bit for bit
- A BFP4 correction that lowers a block's exponent one step when that reduces error
- Per-layer mixed BFP4 and BFP8 precision
- An output format the NPU loads without quantizing again
02
NPU kernels
We develop kernels that run the operations of LLM inference on the Tensix NPU. They run on top of Tenstorrent's open software stack without modifying it.
- Decode matmuls with fused output stages
- An output head that returns token IDs directly
- MoE with routing and the chosen experts computed on the NPU
- Linear attention such as Gated DeltaNet and Mamba-2
03
On-device runtime
A runtime that runs the quantized models on the Tensix NPU. The embedding lookup, the output head's token selection and the multi-token prediction draft model all run on the NPU.
- Every model operation on the NPU
- Speculative decoding with multi-token prediction
- Designed for a small host computer
- Commands to download, run and chat with a model
04
Open models
We quantize open models such as Qwen, Gemma, Llama, Mistral, GLM and gpt-oss and publish them on Hugging Face. Besides the Tensix NPU models, we release GGUF builds for llama.cpp and MLX builds for Apple silicon.
- LLMs, VLMs, MoE and hybrid models
- Speech recognition, speech synthesis and speech enhancement models
- Model cards with accuracy against the original model
- Extending to image generation models and VLA robot policies
From model to device
Quantize
We quantize the open model's weights to the NPU's formats.
Verify
Test runs on GPUs and CPUs compare the quantized weights with the original model.
Load
We pack the weights into the NPU's tile format without quantizing again and load them on the device.
Run
Our own kernels and runtime run inference on the device.
Capabilities
- Quantization to NPU block floating-point formats (BFP4, BFP8)
- Per-layer mixed precision
- Decode matmul and output head kernels
- MoE kernels that read only the chosen experts
- Linear attention kernels such as Gated DeltaNet and Mamba-2
- Multi-token prediction draft models on the NPU
- Model cards with accuracy against the original model
- GGUF quantizations for GPUs and CPUs, MLX for Apple silicon
Deployment goals
- AI models that work on the device without a network
- Inference that keeps site data on site
- Edge AI devices built around a small host computer
Frequently asked questions
- Which models do you optimize?
- Open LLMs and VLMs such as Qwen, Gemma, Llama, Mistral, GLM and gpt-oss, plus speech recognition and speech synthesis models. We are extending the work to image generation models and VLA robot policies.
- Doesn't quantization cost accuracy?
- We use GPTQ, a published method that reduces the error by spreading it over the weights not yet quantized. Each model card lists the format of every tensor class and the accuracy against the original model.
- Can I run the open tools myself?
- The quantization tool, NPU kernels and runtime are on GitHub under the Apache license. They run on Tenstorrent's open software stack, and without hardware they run on Tenstorrent's simulator.
- Can you apply this to our own models or devices?
- Tell us the target models and devices and the accuracy and response speed you need, and we will review the scope.