Toloka

Toloka, based in Amsterdam, is a crowdsourcing and generative AI services provider.

The company helps development of artificial intelligence from training to evaluation and provides generative artificial intelligence and large language model-related services.

History

Toloka was founded in 2014 by Olga Megorskaya, a member of the board of directors of Yandex, as a crowdsourcing and microtasking platform. It was founded primarily for data markup to improve machine learning and search algorithms

As generative AI evolved, the platform adapted to provide expert data labeling to generational AI app producers.

In 2024, the company's Russian operations were sold to Russian investors.

Services

Generative AI

In the generative AI domain, Toloka provides services such as model fine tuning, reinforcement learning from human feedback, evaluation, adhoc datasets, which require large volumes of highly skilled experts annotation.

Machine learning

On Toloka, trainers are tasked with identifying the presence or absence of objects in content, as specified by algorithms. They also assess chatbot responses within given dialogues for relevance and engagement. Additionally, translation verification tasks involve evaluating the accuracy of translations from multiple annotators. For the fine-tuning of large language models (LLMs), experts are required to generate and provide context-based prompts that can be single-turn or multi-turn, serving various domains and purposes.

Natural language processing

In the natural language processing (NLP) domain, Toloka facilitates optical character recognition and classification, sentiment analysis, named-entity recognition, and search relevance evaluation. It also provides transcription and classification of audio data.

Annotators

Toloka mainly works with domain experts, such as physicists, scientists, lawyers, and software engineers, to develop specialized data for models targeting niche tasks. Toloka also works with freelancers, referred to as "Tolokers," who annotate and create data for diverse applications. They perform tasks such as labeling personally identifiable information for AI projects, translating content, summarizing information, and transcribing audio to text.

Upon completion of each task the performer receives a reward based on the volume of images, videos, and unstructured text.

Research

In May 2019, Toloka's research team began publishing datasets for non-commercial and academic purposes to support the scientific community and attract researchers to Toloka. Such datasets are addressed to researchers in different directions like linguistics, computer vision, testing of result aggregation models, and chatbot training.

Toloka research has been showcased at a range of conferences, including the Conference on Neural Information Processing Systems (NeurIPS), the International Conference on Machine Learning (ICML) and the International Conference on Very Large Data Bases (VLDB).

In February 2024, Toloka conducted a tutorial at the AAAI Conference on Artificial Intelligence, focusing on aligning Large Language Models to Low-Resource Languages.

The company participated in BigCode, a joint scientific initiative led by HuggingFace and ServiceNow, where it served as the primary data partner.

Controversies

Enabling arrests of protesters via facial recognition software (March 2024)

In March 2024, Toloka's Russian division was criticized for helping develop the facial recognition software used by Russia to track and arrest protesters after the death of Alexei Navalny. The company's Russian operations were sold in July 2024.

References

External links

* {{Official, https://toloka.ai/

Companies based in Amsterdam
Crowdsourcing
Human-based computation
Social information processing
Web services