Wu Dao ( zh, t=悟道, p=wùdào, l=road to awareness)
is a
multimodal artificial intelligence
Artificial intelligence (AI) is the capability of computer, computational systems to perform tasks typically associated with human intelligence, such as learning, reasoning, problem-solving, perception, and decision-making. It is a field of re ...
developed by the
Beijing Academy of Artificial Intelligence
Beijing Academy of Artificial Intelligence (BAAI) (), also known as Zhiyuan Institute, is a Chinese non-profit artificial intelligence (AI) research laboratory. BAAI conducts AI research and is dedicated to promoting collaboration among academia ...
(BAAI).
Wu Dao 1.0 was first announced on January 11, 2021;
an improved version, Wu Dao 2.0, was announced on May 31.
It has been compared to
GPT-3
Generative Pre-trained Transformer 3 (GPT-3) is a large language model released by OpenAI in 2020.
Like its predecessor, GPT-2, it is a decoder-only transformer model of deep neural network, which supersedes recurrence and convolution-based ...
,
and is built on a similar architecture; in comparison, GPT-3 has 175 billion
parameters
A parameter (), generally, is any characteristic that can help in defining or classifying a particular system (meaning an event, project, object, situation, etc.). That is, a parameter is an element of a system that is useful, or critical, when ...
— variables and inputs within the machine learning model — while Wu Dao has 1.75 trillion parameters.
Wu Dao was trained on 4.9 terabytes of images and texts (which included 1.2 terabytes of Chinese text and 1.2 terabytes of English text),
while GPT-3 was trained on 45 terabytes of text data.
Yet, a growing body of work highlights the importance of increasing both data and parameters.
The chairman of BAAI said that Wu Dao was an attempt to "create the biggest, most powerful AI model possible".
Wu Dao 2.0, was called "the biggest language A.I. system yet".
It was interpreted by commenters as an attempt to "compete with the United States".
Notably, the type of architecture used for Wu Dao 2.0 is a
mixture-of-experts (MoE) model,
unlike GPT-3, which is a "dense" model:
while MoE models require much less computational power to train than dense models with the same numbers of parameters,
trillion-parameter MoE models have shown comparable performance to models that are hundreds of times smaller.
Wu Dao's creators demonstrated its ability to perform
natural language processing
Natural language processing (NLP) is a subfield of computer science and especially artificial intelligence. It is primarily concerned with providing computers with the ability to process data encoded in natural language and is thus closely related ...
and image recognition, in addition to generation of text and images.
The model can not only write essays, poems and couplets in traditional Chinese, it can both generate alt text based on a static image and generate nearly photorealistic images based on natural language descriptions. Wu Dao also showed off its ability to power virtual idols (with a little help from
Microsoft
Microsoft Corporation is an American multinational corporation and technology company, technology conglomerate headquartered in Redmond, Washington. Founded in 1975, the company became influential in the History of personal computers#The ear ...
-spinoff
Xiaoice
Xiaoice (, IPA ) is the AI system developed by Microsoft (Asia) Software Technology Center (STCA) in 2014 based on emotional computing framework. In July 2018, Microsoft Xiaoice released the 6th generation.
Xiaoice Company, formerly known as AI ...
) and predict the 3D structures of proteins like
AlphaFold
AlphaFold is an artificial intelligence (AI) program developed by DeepMind, a subsidiary of Alphabet, which performs predictions of protein structure. It is designed using deep learning techniques.
AlphaFold 1 (2018) placed first in the overall ...
.
History
Wu Dao's development began in October 2020, several months after the May 2020 release of
GPT-3
Generative Pre-trained Transformer 3 (GPT-3) is a large language model released by OpenAI in 2020.
Like its predecessor, GPT-2, it is a decoder-only transformer model of deep neural network, which supersedes recurrence and convolution-based ...
.
The first iteration of the model, Wu Dao 1.0, "initiated large-scale research projects"
via four related models.
* Wu Dao – Wen Yuan, a 2.6-billion-parameter pretrained language model, was designed for tasks like open-domain answering, sentiment analysis, and grammar correction.
* Wu Dao – Wen Lan, a 1-billion-parameter multimodal graphic model, was trained on 50 million image pairs to perform image captioning.
* Wu Dao – Wen Hui, an 11.3-billion-parameter generative language model, was designed for "essential problems in general artificial intelligence from a cognitive perspective"; ''Synced'' says that it can "generate poetry, make videos, draw pictures, retrieve text, perform complex reasoning, etc".
* Wu Dao – Wen Su, based on Google's BERT language model and trained on the 100-gigabyte UNIPARC database (as well as thousands of gene sequences), was designed for biomolecular structure prediction and protein folding tasks.
WuDao Corpora
WuDao Corpora (also written as WuDaoCorpora), as of version 2.0, was a large dataset constructed for training Wu Dao 2.0. It contains 3 terabytes of text scraped from web data, 90 terabytes of graphical data (incorporating 630 million text/image pairs), and 181 gigabytes of Chinese dialogue (incorporating 1.4 billion dialogue rounds).
Wu Dao 2.0 was trained using FastMoE, a variant of the mixture of experts architecture published by
Google
Google LLC (, ) is an American multinational corporation and technology company focusing on online advertising, search engine technology, cloud computing, computer software, quantum computing, e-commerce, consumer electronics, and artificial ...
.
''TheNextWeb'' said in June 2021 that "details as to exactly how Wu Dao was trained, what was in its various datasets, and what practical applications it can be used for remain scarce".
OpenAI
OpenAI, Inc. is an American artificial intelligence (AI) organization founded in December 2015 and headquartered in San Francisco, California. It aims to develop "safe and beneficial" artificial general intelligence (AGI), which it defines ...
's policy director called Wu Dao an example of "model diffusion", a neologism describing a situation in which multiple entities develop models similar to OpenAI's.
References
{{reflist, refs=
[{{Cite web
, url = https://resource.wudaoai.cn/home
, title = resource
, website = resource.wudaoai.cn
]
[{{cite web
, url = https://mp.weixin.qq.com/s/BUQWZ5EdR19i40GuFofpBg
, title = 面向认知，智源研究院联合多家单位发布超大规模新型预训练模型"悟道·文汇"
, author = 智源研究院
, date = January 11, 2021
]
[{{Cite web
, url = https://reason.com/2021/06/11/amazing-new-chinese-a-i-powered-language-model-wu-dao-2-0-unveiled/
, title = Amazing New Chinese A.I.-Powered Language Model Wu Dao 2.0 Unveiled
, date = June 11, 2021
, first = Ronald
, last = Bailey
, website = Reason
]
[{{Cite web
, url = https://www.politico.eu/article/meet-wu-dao-2-0-the-chinese-ai-model-making-the-west-sweat/
, title = Meet Wu Dao 2.0, the Chinese AI model making the West sweat
, date = June 9, 2021
, website = Politico
, first = Melissa
, last = Heikkilä
]
[{{Cite web
, url = https://thenextweb.com/news/china-wu-dao-ai-10x-bigger-than-gpt-3
, title = China's 'Wu Dao' AI is 10X bigger than GPT-3, and it can sing
, first = Tristan
, last = Greene
, date = June 3, 2021
, website = TNW | Neural
]
[{{Cite web
, url = https://www.engadget.com/chinas-gigantic-multi-modal-ai-is-no-one-trick-pony-211414388.html
, title = China's gigantic multi-modal AI is no one-trick pony
, website = Engadget
, first = A.
, last = Tarantola
, date = June 2, 2021
]
[{{Cite web
, url = https://venturebeat.com/2021/06/04/ai-weekly-chinas-massive-multimodal-model-highlights-ai-research-gap/
, title = AI Weekly: China's massive multimodal model highlights AI research gap
, date = June 4, 2021
, first = Kyle
, last = Wiggers
, website = VentureBeat
]
[{{Cite web
, url = https://www.scmp.com/tech/tech-war/article/3135764/us-china-tech-war-beijing-funded-ai-researchers-surpass-google-and
, title = Beijing-funded AI language model tops Google and OpenAI in raw numbers
, date = June 2, 2021
, website = South China Morning Post
, first = Coco
, last = Feng
]
[{{Cite web
, url = http://www.china.org.cn/business/2021-06/03/content_77546375.htm
, title = Record-breaking AI model WuDao 2.0 unveiled in Beijing
, website = China.org.cn
, date = June 3, 2021
]
[{{Cite web
, url = https://syncedreview.com/2021/03/23/chinas-gpt-3-baai-introduces-superscale-intelligence-model-wu-dao-1-0/
, title = China's GPT-3? BAAI Introduces Superscale Intelligence Model 'Wu Dao 1.0' | Synced
, date = March 23, 2021
, website = syncedreview.com
]
[{{Cite web
, url = https://mixed.de/sprach-ki-aus-china-wu-dao-2-0-ist-zehnmal-groesser-als-gpt-3/
, title = Sprach-KI aus China: Wu Dao 2.0 ist zehnmal größer als GPT-3
, first = Maximilian
, last = Schreiner
, date = 13 June 2021
]
[{{Cite news
, url = https://fortune.com/2021/06/08/can-a-i-help-investors-find-the-next-hot-technology/
, title = Can A.I. help investors find the next hot technology? A number of companies are betting on it.
, website = Fortune
]
[{{Cite web
, url = https://www.tomshardware.com/news/china-touts-wudao-2-ai-advancements
, title = China Says WuDao 2.0 AI Is an Even Better Conversationalist than OpenAI, Google
, first = Nathaniel
, last = Mott
, date = June 3, 2021
, website = Tom's Hardware
]
[{{Cite web
, url = https://www.lantidiplomatico.it/dettnews-wudao_20_lintelligenza_artificiale_cinese_pi_potente_di_openai_e_google/39099_41699/
, title = WuDao 2.0, l'intelligenza artificiale cinese più potente di OpenAI e Google
, website = lantidiplomatico.it
, date = 6 June 2021
]
[{{Cite news
, url = https://elcomercio.pe/tecnologia/actualidad/wu-dao-20-el-modelo-de-ia-multimodal-de-aprendizaje-profundo-mas-grande-del-mundo-tecnologia-noticia/
, title = Wu Dao 2.0, el modelo de IA multimodal de aprendizaje profundo más grande del mundo
, date = June 3, 2021
, newspaper = El Comercio
]
[{{Cite web
, url = https://3dnews.ru/1041172/kitayskaya-neyroset-wudao-20-s-175-trln-parametrov-prevoshodit-analogi-ot-google-i-openai
, title = Китайская нейросеть WuDao 2.0 с 1,75 трлн параметров превосходит аналоги от Google и OpenAI
, website = 3DNews - Daily Digital Digest
]
[{{Cite web
, url = https://geeko.lesoir.be/2021/06/05/la-chine-developpe-une-ia-pour-concurrencer-les-etats-unis/
, title = La Chine développe une IA pour concurrencer les États-Unis
, first = Jennifer
, last = Mertens
, date = June 5, 2021
]
[{{Cite arXiv
, last = B. Brown
, first = Tom
, eprint = 2005.14165
, title = Language Models are Few-Shot Learners
, class = cs.CL
, date = 2020
]
[{{Cite arXiv
, last = Hoffmann
, first = Jordan
, eprint = 2203.15556
, title = Training Compute-Optimal Large Language Models
, class = cs.CL
, date = 2022
]
[{{Cite arXiv
, last = Artetxe
, first = Mikel
, eprint = 2112.10684
, title = Efficient Large Scale Language Modeling with Mixtures of Experts
, class = cs.CL
, date = 2021
]
Deep learning software applications
Language modeling