Watchful

Watchful.io built a data-centric "machine teaching" platform intended to help data science teams produce high-quality labeled training data for machine learning models in hours rather than weeks. The product let data scientists and subject-matter experts codify domain expertise into repeatable, automated labeling workflows, targeting projects where labeling requires specialist knowledge, where explainability across the ML pipeline is required, or where models must be retrained frequently because of data and model drift (for example in adversarial domains such as fraud detection).

A central concept of the platform was probabilistic labeling. Rather than collapsing labeler disagreement into a single deterministic 0/1 label via majority vote, Watchful encoded the ambiguity as a probability, on the basis that probabilistic labels carry richer information about a candidate record's relationship to a class. Documentation described several downstream uses of these labels: training directly on the probabilities, thresholding to most-likely labels, filtering out low-confidence labels at the cost of coverage, sampling from the distribution, or combining labels with other dataset features. At the time of the published documentation the product supported CSV imports and worked best with textual rather than numerical data, with stated plans to extend to other modalities such as time series, images and video.

In April 2024 Thoughtworks acquired Watchful's team of seven employees and its technology assets. Thoughtworks said it would infuse the technology into several of its data and AI services, covering knowledge graph, text classification and taxonomy creation, AI-assisted data labeling, accelerated discovery of unstructured data, and prospectively Data Mesh-based contextual business data.

Business model

Watchful sold access to its labeling software on a subscription basis; Thoughtworks stated that customers who had purchased subscriptions to use Watchful software could continue until the end of their current term, after which contracts would not be renewed.

Latest developments

On 17 April 2024 Thoughtworks acquired Watchful's team and technology assets in what it described as its sixth acquisition in three years. Co-founders Shayan Mohanty and John Singleton joined Thoughtworks along with the rest of the seven-person team, reporting to Chief AI Officer Mike Mason. Financial terms were not disclosed. The transaction did not include Watchful's business operations; existing software subscriptions were to run to the end of their terms without renewal, and the Watchful brand and legacy business were to wind down over time.