Diffbot

Market position

Diffbot positions itself as operating the world's largest automated Knowledge Graph and the largest structured database of the public web, and as one of few independent full-web crawlers outside Google and Bing. In web search APIs it benchmarks itself against Tavily, Exa and Firecrawl, competing on latency and self-hostability while its own published accuracy figure trails Exa.

Key differentiators claimed in the sources are rule-free extraction driven by computer vision rather than site-specific scrapers, resolution of extracted facts into a linked entity graph rather than flat dumps, an independent full-web crawl on self-owned hardware, and a web search index that customers can own and self-host without ads or telemetry.

Technology

Diffbot runs an independent crawl of the public web — separate from Google and Bing — on custom-assembled hardware in its own California data center. Its stack combines computer-vision-based page classification and rendering for rule-free extraction with natural language processing for entity and relation extraction, and applies entity linking, coreference resolution, knowledge fusion, knowledge inference and information retrieval to build the Knowledge Graph. The graph underpins GraphRAG-style applications and a Knowledge-Graph-grounded language model announced in January 2025. Company-published benchmarks report a 240 ms P90 latency for the Web Search API versus 998 ms (Tavily), 1,231 ms (Exa) and 1,335 ms (Firecrawl), with a stated accuracy of 0.65 against 0.8 for Exa; Extract returns results in roughly 300 ms.

Go-to-market

Developer-led distribution through public API documentation, an API reference site, a free plan, tutorials and a company blog, plus integrations aimed at AI agent frameworks (diffbot-skills, an MCP server for DQL and Enhance). Product pages include interactive demos that issue live API calls. The company publicizes launches through press releases and maintains presence on LinkedIn, Bluesky and Mastodon.

Companies worldwide that need large-scale structured data from the public web, including search and e-commerce products, business intelligence and predictive analytics vendors, supply chain risk tools, and developers building AI agents and RAG pipelines. Named users include DuckDuckGo (product data structuring for shopping search), ProQuo AI (organization data for predictive business development) and Contingent (news data for supply chain insight).

Geography

Headquartered in Menlo Park, United States, with its own data center in California; the company states it crawls the entire public web and serves customers internationally.

History

Public press releases trace the company from a 2011 production API applying visual learning to web content, a Page Classifier developer tool in 2012, and a computer-vision Product API in 2013, to the Discussions API in 2015 and a BBC Click television feature the same year. Funding milestones include $2 million from technology veterans announced in May 2012 and a $10M Series A in February 2016, when the team was described as 12 AI engineers. The Knowledge Graph launched in August 2018, a Knowledge-as-a-Service NLP system in September 2020, a free plan in April 2024, and a Knowledge-Graph-grounded language model in January 2025, followed by the self-hostable Web Search API. Documentation states the company has maintained its APIs for over 15 years.

Risks & controversies

Diffbot's own published benchmark shows its search accuracy (0.65) below Exa (0.8), indicating a latency-versus-accuracy tradeoff. The company frames current web search as ad-monetized and scraped for API access, a market dynamic it competes against. No litigation, regulatory action or other controversies appear in the provided sources.