Centaur

Centaur.ai (formerly Centaur Labs) is a Boston-based data annotation company that produces labeled datasets for AI development in healthcare and other high-stakes domains. Its platform organizes annotation as a competition: subject matter experts and AI models label the same items independently, are continually measured and rewarded on measured performance, and their outputs are then aggregated with weights based on demonstrated accuracy rather than credentials or years of experience. The company states this aggregation yields results more accurate than either the best individual human annotator or the best individual AI model.

The workflow described by the company spans dataset design (using either a customer's raw data or data Centaur supplies), competitive annotation drawing on a stated network of more than 100,000 subject matter experts and/or AI models, aggregation of the strongest signals, and delivery of production-ready data via API, JSON, CSV, or DICOM for model training and evaluation. Annotation covers text, image, audio, video, and waveform data, with published examples including skin lesion classification, retinal OCT grading, ICD-10 code mapping, lung nodule segmentation, colonoscopy video annotation, and chest X-ray datasets. Experts compete on the company's mobile app, DiagnosUs, and work is delivered on a HIPAA-compliant platform.

Alongside labeling, Centaur publishes model evaluation and benchmark work, including health evaluations of frontier large language models, a calorie-estimation benchmark comparing humans with eight frontier AI models, and the MedAESQA medical question-answering benchmark.

Founding story

Centaur grew out of research at the MIT Center for Collective Intelligence. While founder Erik Duhaime was conducting PhD research on collective intelligence, he was inspired by his wife's medical training and began running experiments on combining the opinions of multiple medical professionals and AI algorithms to assess skin lesions for cancer. He found that weighting opinions by performance-based metrics rather than experience or credentials, and identifying how different human and machine opinions complement each other, produced more accurate results than the best individual doctor or the best AI algorithm alone. That research became the basis for the company, originally called Centaur Labs.

Business model

B2B services and platform: Centaur.ai sells annotated datasets and model evaluation work to organizations building AI, delivering production-ready data through APIs and standard file and imaging formats. Work is sourced from a distributed network of subject matter experts who compete on the company's DiagnosUs app and are rewarded according to measured annotation performance.

Traction

The company reports a network of over 100,000 subject matter experts on its site, while 2024 press coverage of its Series B cited over 50,000 experts. Published customer results include Paige improving an F1 score from 0.6 to 0.83, Eight Sleep moving from 70% to 93% accuracy with gamified labeling, SciBite reducing a project timeline by two months and surfacing roughly 5,000 candidate vocabulary synonyms, and a customer describing label volumes 10x to 20x what it had achieved internally. Other work includes a VUNO brain MRI FDA clearance case study and contributions to Microsoft's PadChest-GR dataset.

Latest developments

In November 2024 the company announced a $750,000 grant supporting research with Brigham and Women's Hospital. Recent work includes contributing to Microsoft's PadChest-GR chest X-ray dataset (January 2025), a partnership with Autoblocks (April 2025), the MedAESQA medical question-answering benchmark (August 2025), a HealthAwards.com Gold award for mobile digital health resources (December 2025), a calorie-estimation benchmark in which humans outperformed eight frontier AI models (July 2026), and a Grok 4.5 health evaluation (August 2026). The site also promotes a Centaur Arena in which experts and AI models compete on labeling tasks.