OctoAI

OctoAI, originally named OctoML, was founded in 2019 as a spinout of the Paul G. Allen School of Computer Science at the University of Washington in Seattle. The founding team comprised the creators of Apache TVM, an open-source deep learning compiler stack used to deploy neural networks across heterogeneous hardware including NVIDIA GPUs, AWS Inferentia, Intel CPUs, ARM processors and edge devices. The company's initial thesis was that machine learning models perform very differently depending on target hardware and that hand-tuning for each target is labor-intensive; OctoML aimed to commercialize TVM-based automated compilation and optimization so enterprises could deploy faster and cheaper models without in-house compiler engineering teams.

The first product, Octomizer, accepted ONNX, PyTorch or TensorFlow models, let users select a target (cloud GPU, edge device, specific CPU), and returned an optimized deployable artifact. It combined Apache TVM with ONNX Runtime, NVIDIA TensorRT and Intel OpenVINO, with claimed improvements of up to 10x over unoptimized deployment. In June 2023 the company launched OctoAI, described as a self-optimizing compute service for AI, and reoriented around hosting foundation models — Llama 2 (7B/13B/70B and chat variants), Code Llama Instruct, Mistral 7B Instruct, and Stable Diffusion XL with custom LoRA asset support — behind an OpenAI-compatible API, along with a bring-your-own-model option for fine-tuned Llama 2 variants, a Python SDK and REST API. A later product, OctoStack, was described as a toolkit for deploying, running and scaling generative AI models on a hardware-agnostic basis.

NVIDIA acquired OctoAI in late September 2024. On 26 September 2024 customers were emailed notice that commercial availability of the services would wind down and that access to all OctoAI services and accounts would terminate effective 31 October 2024, giving developers roughly five weeks to migrate. No successor product was offered; a migration guide pointed users to competing inference providers.

Founding story

OctoML was founded in 2019 out of the Paul G. Allen School of Computer Science at the University of Washington in Seattle by the creators of Apache TVM. Named founders include Luis Ceze (CEO), a UW computer science professor and ACM Fellow who retained his faculty position throughout the company's existence; Tianqi Chen (CTO), a UW PhD and creator of both Apache TVM and XGBoost; Jason Knight (CPO), covering product strategy and go-to-market; Jared Roesch (Chief Architect), a core TVM contributor working on compiler architecture; and Thierry Moreau (VP of Technology Partnerships), responsible for hardware integration and ecosystem work.

Business model

OctoAI sold AI infrastructure software and services to developers and enterprises: initially a model compilation and optimization service (Octomizer), later an API-based inference platform hosting open-source and customer-supplied foundation models, and the OctoStack toolkit for deploying and scaling generative AI models across hardware.

Sources describe a commercial developer platform accessed via API and SDK with a free sign-up tier; specific pricing terms are not stated in the sources.

Traction

Within 10 months of the OctoAI product launch, the platform reported more than 25,000 developers and over 100 high-growth business customers.

Latest developments

NVIDIA acquired OctoAI in late September 2024 in a deal one source describes as reportedly worth hundreds of millions of dollars. Customers were notified on 26 September 2024 that commercial services would wind down, with all service access and accounts terminated on 31 October 2024 and no successor product; a migration guide directed users to Fireworks AI, Together AI, Anyscale and Groq. The octo.ai domain now serves NVIDIA's corporate website.