AI-generated Misinformation

Generative artificial intelligence (Generative AI, GenAI, or GAI) is a subfield of artificial intelligence that uses generative models to produce text, images, videos, or other forms of data. These models learn the underlying patterns and structures of their training data and use them to produce new data based on the input, which often comes in the form of natural language prompts.

Generative AI tools have become more common since an "AI boom" in the 2020s. This boom was made possible by improvements in transformer-based deep neural networks, particularly large language models (LLMs). Major tools include chatbots such as ChatGPT, Copilot, Gemini, Grok, and DeepSeek; text-to-image models such as Stable Diffusion, Midjourney, and DALL-E; and text-to-video models such as Sora and Veo. Technology companies developing generative AI include OpenAI, Anthropic, Meta AI, Microsoft, Google, DeepSeek, and Baidu.

Generative AI has raised many ethical questions. It can be used for cybercrime, or to deceive or manipulate people through fake news or deepfakes. Even if used ethically, it may lead to mass replacement of human jobs. The tools themselves have been criticized as violating intellectual property laws, since they are trained on and emulate copyrighted works of art.

Generative AI is used across many industries. Examples include software development, healthcare, finance, entertainment, customer service, sales and marketing, art, writing, fashion, and product design.

History

Early history

The first example of an algorithmically generated media is likely the Markov chain. Markov chains have long been used to model natural languages since their development by Russian mathematician Andrey Markov in the early 20th century. Markov published his first paper on the topic in 1906, and analyzed the pattern of vowels and consonants in the novel ''Eugeny Onegin'' using Markov chains. Once a Markov chain is learned on a text corpus, it can then be used as a probabilistic text generator.

Computers were needed to go beyond Markov chains. By the early 1970s, Harold Cohen was creating and exhibiting generative AI works created by AARON, the computer program Cohen created to generate paintings.

The terms generative AI planning or generative planning were used in the 1980s and 1990s to refer to AI planning systems, especially computer-aided process planning, used to generate sequences of actions to reach a specified goal. Generative AI planning systems used symbolic AI methods such as state space search and constraint satisfaction and were a "relatively mature" technology by the early 1990s. They were used to generate crisis action plans for military use, process plans for manufacturing and decision plans such as in prototype autonomous spacecraft.

Generative neural networks (2014–2019)

Since inception, the field of machine learning has used both discriminative models and generative models to model and predict data. Beginning in the late 2000s, the emergence of deep learning drove progress, and research in image classification, speech recognition, natural language processing and other tasks. Neural networks in this era were typically trained as discriminative models due to the difficulty of generative modeling.

In 2014, advancements such as the variational autoencoder and generative adversarial network produced the first practical deep neural networks capable of learning generative models, as opposed to discriminative ones, for complex data such as images. These deep generative models were the first to output not only class labels for images but also entire images.

In 2017, the Transformer network enabled advancements in generative models compared to older Long-Short Term Memory models, leading to the first generative pre-trained transformer (GPT), known as GPT-1, in 2018. This was followed in 2019 by GPT-2, which demonstrated the ability to generalize unsupervised to many different tasks as a Foundation model.

The new generative models introduced during this period allowed for large neural networks to be trained using unsupervised learning or semi-supervised learning, rather than the supervised learning typical of discriminative models. Unsupervised learning removed the need for humans to manually label data, allowing for larger networks to be trained.

Generative AI boom (2020–)

In March 2020, the release of 15.ai, a free web application created by an anonymous MIT researcher that could generate convincing character voices using minimal training data, marked one of the earliest popular use cases of generative AI. The platform is credited as the first mainstream service to popularize AI voice cloning (audio deepfakes) in memes and content creation, influencing subsequent developments in voice AI technology.

In 2021, the emergence of DALL-E, a transformer-based pixel generative model, marked an advance in AI-generated imagery. This was followed by the releases of Midjourney and Stable Diffusion in 2022, which further democratized access to high-quality artificial intelligence art creation from natural language prompts. These systems demonstrated unprecedented capabilities in generating photorealistic images, artwork, and designs based on text descriptions, leading to widespread adoption among artists, designers, and the general public.

In late 2022, the public release of ChatGPT revolutionized the accessibility and application of generative AI for general-purpose text-based tasks. The system's ability to engage in natural conversations, generate creative content, assist with coding, and perform various analytical tasks captured global attention and sparked widespread discussion about AI's potential impact on work, education, and creativity.

In March 2023, GPT-4's release represented another jump in generative AI capabilities. A team from Microsoft Research controversially argued that it "could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system." However, this assessment was contested by other scholars who maintained that generative AI remained "still far from reaching the benchmark of 'general human intelligence'" as of 2023. Later in 2023, Meta released ImageBind, an AI model combining multiple modalities including text, images, video, thermal data, 3D data, audio, and motion, paving the way for more immersive generative AI applications.

In December 2023, Google unveiled Gemini, a multimodal AI model available in four versions: Ultra, Pro, Flash, and Nano. The company integrated Gemini Pro into its Bard chatbot and announced plans for "Bard Advanced" powered by the larger Gemini Ultra model. In February 2024, Google unified Bard and Duet AI under the Gemini brand, launching a mobile app on Android and integrating the service into the Google app on iOS.

In March 2024, Anthropic released the Claude 3 family of large language models, including Claude 3 Haiku, Sonnet, and Opus. The models demonstrated significant improvements in capabilities across various benchmarks, with Claude 3 Opus notably outperforming leading models from OpenAI and Google. In June 2024, Anthropic released Claude 3.5 Sonnet, which demonstrated improved performance compared to the larger Claude 3 Opus, particularly in areas such as coding, multistep workflows, and image analysis.

Asia–Pacific countries are significantly more optimistic than Western societies about generative AI and show higher adoption rates. Despite expressing concerns about privacy and the pace of change, in a 2024 survey, 68% of Asia-Pacific respondents believed that AI was having a positive impact on the world, compared to 57% globally. According to a survey by SAS and Coleman Parkes Research, China in particular has emerged as a global leader in generative AI adoption, with 83% of Chinese respondents using the technology, exceeding both the global average of 54% and the U.S. rate of 65%. This leadership is further evidenced by China's intellectual property developments in the field, with a UN report revealing that Chinese entities filed over 38,000 generative AI patents from 2014 to 2023, substantially surpassing the United States in patent applications. A 2024 survey on the Chinese social app Soul reported that 18% of respondents born after 2000 used generative AI "almost every day", and that over 60% of respondents like or love AI-generated content, while less than 3% dislike or hate it.

Applications

A generative AI system is constructed by applying unsupervised machine learning (invoking for instance neural network architectures such as generative adversarial networks (GANs), variation autoencoders (VAEs), transformers, or self-supervised machine learning trained on a dataset. The capabilities of a generative AI system depend on the output (modality) of the data set used. Generative AI can be either ''unimodal'' or ''multimodal''; unimodal systems take only one type of input, whereas multimodal systems can take more than one type of input. For example, one version of OpenAI's GPT-4 accepts both text and image inputs.

Generative AI has made its appearance in a wide variety of industries, radically changing the dynamics of content creation, analysis, and delivery. In healthcare, generative AI is instrumental in accelerating drug discovery by creating molecular structures with target characteristics and generating radiology images for training diagnostic models. This extraordinary ability not only enables faster and cheaper development but also enhances medical decision-making. In finance, generative AI is invaluable as it generates datasets to train models and automates report generation with natural language summarization capabilities. It automates content creation, produces synthetic financial data, and tailors customer communications. It also powers chatbots and virtual agents. Collectively, these technologies enhance efficiency, reduce operational costs, and support data-driven decision-making in financial institutions. The media industry makes use of generative AI for numerous creative activities such as music composition, scriptwriting, video editing, and digital art. The educational sector is impacted as well, since the tools make learning personalized through creating quizzes, study aids, and essay composition. Both the teachers and the learners benefit from AI-based platforms that suit various learning patterns.

Text and software code

Generative AI systems trained on words or word tokens include GPT-3, GPT-4, GPT-4o, LaMDA, LLaMA, BLOOM (language model), BLOOM, Gemini and others (see List of large language models). They are capable of natural language processing, machine translation, and natural language generation and can be used as foundation models for other tasks. Data sets include BookCorpus, Wikipedia, and others (see List of text corpora).

In addition to natural language text, large language models can be trained on programming language text, allowing them to generate source code for new computer programs. Examples include OpenAI Codex, Tabnine, GitHub Copilot, Microsoft Copilot, and VS Code Fork (software development), fork Cursor (code editor), Cursor.

Some AI assistants help candidates cheat during online coding interviews by providing code, improvements, and explanations. Their clandestine interfaces minimize the need for eye movements that would expose cheating to the interviewer.

Images

Producing high-quality visual art is a prominent application of generative AI. Generative AI systems trained on sets of images with Caption (text), text captions include Imagen (text-to-image model), Imagen, DALL-E, Midjourney, Adobe Firefly, FLUX.1, Stable Diffusion and others (see Artificial intelligence art, Generative art, and Synthetic media). They are commonly used for text-to-image generation and neural style transfer. Datasets include LAION, LAION-5B and others (see List of datasets in computer vision and image processing).

Audio

Generative AI can also be trained extensively on audio clips to produce natural-sounding speech synthesis and text-to-speech capabilities. An early pioneer in this field was 15.ai, launched in March 2020, which demonstrated the ability to clone character voices using as little as 15 seconds of training data. The website gained widespread attention for its ability to generate emotionally expressive speech for various fictional characters, though it was later taken offline in 2022 due to copyright concerns. Commercial alternatives subsequently emerged, including ElevenLabs' context-aware synthesis tools and Meta Platforms, Meta Platform's Voicebox.

Generative AI systems such as MusicLM and MusicGen can also be trained on the audio waveforms of recorded music along with text annotations, in order to generate new musical samples based on text descriptions such as ''a calming violin melody backed by a distorted guitar riff''.

Audio deepfakes of music lyrics have been generated, like the song Savages, which used AI to mimic rapper Jay-Z's vocals. Music artist's instrumentals and lyrics are copyrighted but their voices are not protected from regenerative AI yet, raising a debate about whether artists should get royalties from audio deepfakes.

Many AI music generators have been created that can be generated using a text phrase, Music genre, genre options, and Loop (music), looped Library (computing), libraries of Bar (music), bars and riffs.

Video

Generative AI trained on annotated video can Text-to-video model, generate temporally-coherent, detailed and photorealistic video clips. Examples include Sora by OpenAI, Runway (company), Runway, and Make-A-Video by Meta Platforms.

Robotics

Generative AI can also be trained on the motions of a robotic system to generate new trajectories for motion planning or Robot navigation, navigation. For example, UniPi from Google Research uses prompts like ''"pick up blue bowl"'' or ''"wipe plate with yellow sponge"'' to control movements of a robot arm. Multimodal "vision-language-action" models such as Google's RT-2 can perform rudimentary reasoning in response to user prompts and visual input, such as picking up a toy dinosaur when given the prompt ''pick up the extinct animal'' at a table filled with toy animals and other objects.

3D modeling

Artificially intelligent computer-aided design (CAD) can use text-to-3D, image-to-3D, and video-to-3D to automate 3D modeling. AI-based Digital library#CAD library, CAD libraries could also be developed using Linked data, linked open data of schematics and diagrams. AI CAD Virtual assistant, assistants are used as tools to help streamline workflow.

Software and hardware

Generative AI models are used to power chatbot products such as ChatGPT, programming tools such as GitHub Copilot, text-to-image products such as Midjourney, and text-to-video products such as Runway (company), Runway Gen-2. Generative AI features have been integrated into a variety of existing commercially available products such as Microsoft Office (Microsoft Copilot), Google Photos, and the Adobe Inc., Adobe Suite (Adobe Firefly). Many generative AI models are also available as open-source software, including Stable Diffusion and the LLaMA language model.

Smaller generative AI models with up to a few billion parameters can run on smartphones, embedded devices, and personal computers. For example, LLaMA-7B (a version with 7 billion parameters) can run on a Raspberry Pi 4 and one version of Stable Diffusion can run on an iPhone 11.

Larger models with tens of billions of parameters can run on laptop or desktop computers. To achieve an acceptable speed, models of this size may require AI accelerator, accelerators such as the GPU chips produced by NVIDIA and AMD or the Neural Engine included in Apple silicon products. For example, the 65 billion parameter version of LLaMA can be configured to run on a desktop PC.

The advantages of running generative AI locally include protection of Information privacy, privacy and intellectual property, and avoidance of rate limiting and censorship. The Reddit, subreddit r/LocalLLaMA in particular focuses on using Consumer electronics, consumer-grade gaming graphics cards through such techniques as Large language model#Compression, compression. That forum is one of only two sources Andrej Karpathy trusts for Language model#Evaluation and benchmarks, language model benchmarks. Yann LeCun has advocated open-source models for their value to Vertical market software, vertical applications and for improving AI safety.

Language models with hundreds of billions of parameters, such as GPT-4 or PaLM, typically run on datacenter computers equipped with arrays of GPUs (such as NVIDIA's Hopper (microarchitecture), H100) or AI accelerator chips (such as Google's Tensor Processing Unit, TPU). These very large models are typically accessed as Cloud computing, cloud services over the Internet.

In 2022, the United States New Export Controls on Advanced Computing and Semiconductors to China imposed restrictions on exports to China of GPU and AI accelerator chips used for generative AI. Chips such as the NVIDIA A800 and the Biren Technology BR104 were developed to meet the requirements of the sanctions.

There is free software on the market capable of recognizing text generated by generative artificial intelligence (such as GPTZero), as well as images, audio or video coming from it. Potential mitigation strategies for Artificial intelligence content detection, detecting generative AI content include digital watermarking, Data provenance, content authentication, information retrieval, and Supervised learning, machine learning classifier models. Despite claims of accuracy, both free and paid AI text detectors have frequently produced false positives, mistakenly accusing students of submitting AI-generated work.

Generative models and training techniques

Generative adversarial networks

Generative adversarial networks (GANs) are an influential generative modeling technique. GANs consist of two neural networks—the generator and the discriminator—trained simultaneously in a competitive setting. The generator creates synthetic data by transforming random noise into samples that resemble the training dataset. The discriminator is trained to distinguish the authentic data from synthetic data produced by the generator. The two models engage in a minimax game: the generator aims to create increasingly realistic data to "fool" the discriminator, while the discriminator improves its ability to distinguish real from fake data. This continuous training setup enables the generator to produce high-quality and realistic outputs.

Variational autoencoders

Variational autoencoders (VAEs) are deep learning models that probabilistically encode data. They are typically used for tasks such as noise reduction from images, data compression, identifying unusual patterns, and Facial recognition system, facial recognition. Unlike Autoencoder, standard autoencoders, which compress input data into a fixed latent representation, VAEs model the latent space as a probability distribution, allowing for smooth sampling and interpolation between data points. The encoder ("recognition model") maps input data to a latent space, producing means and variances that define a probability distribution. The decoder ("generative model") samples from this latent distribution and attempts to reconstruct the original input. VAEs optimize a loss function that includes both the reconstruction error and a Kullback–Leibler divergence term, which ensures the latent space follows a known prior distribution. VAEs are particularly suitable for tasks that require structured but smooth latent spaces, although they may create blurrier images than GANs. They are used for applications like image generation, data interpolation and anomaly detection.

= Transformers

=
Transformers became the foundation for many powerful generative models, most notably the generative pre-trained transformer (GPT) series developed by OpenAI. They marked a major shift in natural language processing by replacing traditional Recurrent neural network, recurrent and Convolutional neural network, convolutional models. This architecture allows models to process entire sequences simultaneously and capture long-range dependencies more efficiently. The Attention (machine learning), self-attention mechanism enables the model to capture the significance of every word in a sequence when predicting the subsequent word, thus improving its contextual understanding. Unlike recurrent neural networks, transformers process all the tokens in parallel, which improves the training efficiency and scalability. Transformers are typically pre-trained on enormous corpora in a self-supervised manner, prior to being Fine-tuning (deep learning), fine-tuned.

Law and regulation

In the United States, a group of companies including OpenAI, Alphabet, and Meta signed a voluntary agreement with the Biden administration in July 2023 to watermark AI-generated content. In October 2023, Executive Order 14110 applied the Defense Production Act to require all US companies to report information to the federal government when training certain high-impact AI models.

In the European Union, the proposed Artificial Intelligence Act includes requirements to disclose copyrighted material used to train generative AI systems, and to label any AI-generated output as such.

In China, the Interim Measures for the Management of Generative AI Services introduced by the Cyberspace Administration of China regulates any public-facing generative AI. It includes requirements to watermark generated images or videos, regulations on training data and label quality, restrictions on personal data collection, and a guideline that generative AI must "adhere to socialist core values".

Copyright

Training with copyrighted content

Generative AI systems such as ChatGPT and Midjourney are trained on large, publicly available datasets that include copyrighted works. AI developers have argued that such training is protected under fair use, while copyright holders have argued that it infringes their rights.

Proponents of fair use training have argued that it is a transformative use and does not involve making copies of copyrighted works available to the public. Critics have argued that image generators such as Midjourney can create nearly-identical copies of some copyrighted images, and that generative AI programs compete with the content they are trained on.

As of 2024, several lawsuits related to the use of copyrighted material in training are ongoing.
Getty Images has sued Stability AI over the use of its images to train Stable Diffusion. Both the Authors Guild and The New York Times have sued Microsoft and OpenAI over the use of their works to train ChatGPT.

Copyright of AI-generated content

A separate question is whether AI-generated works can qualify for copyright protection. The United States Copyright Office has ruled that works created by artificial intelligence without any human input cannot be copyrighted, because they lack human authorship. Some legal professionals have suggested that ''Monkey selfie copyright dispute, Naruto v. Slater'' (2018), in which the U.S. 9th Circuit Court of Appeals held that non-humans cannot be copyright holders of Animal-made art, artistic works, could be a potential precedent in copyright litigation over works created by generative AI. However, the office has also begun taking public input to determine if these rules need to be refined for generative AI.

In January 2025, the United States Copyright Office (USCO) released extensive guidance regarding the use of AI tools in the creative process, and established that "...generative AI systems also offer tools that similarly allow users to exert control. [These] can enable the user to control the selection and placement of individual creative elements. Whether such modifications rise to the minimum standard of originality required under Feist Publications, Inc., v. Rural Telephone Service Co., Feist will depend on a case-by-case determination. In those cases where they do, the output should be copyrightable" Subsequently, the USCO registered the first visual artwork to be composed of entirely AI-generated materials, titled "A Single Piece of American Cheese".

Concerns

The development of generative AI has raised concerns from governments, businesses, and individuals, resulting in protests, legal actions, calls to Pause Giant AI Experiments: An Open Letter, pause AI experiments, and actions by multiple governments. In a July 2023 briefing of the United Nations Security Council, Secretary-General of the United Nations, Secretary-General António Guterres stated "Generative AI has enormous potential for good and evil at scale", that AI may "turbocharge global development" and contribute between $10 and $15 trillion to the global economy by 2030, but that its malicious use "could cause horrific levels of death and destruction, widespread trauma, and deep psychological damage on an unimaginable scale". In addition, generative AI has a significant carbon footprint.

Job losses

From the early days of the development of AI, there have been arguments put forward by ELIZA creator Joseph Weizenbaum and others about whether tasks that can be done by computers actually should be done by them, given the difference between computers and humans, and between quantitative calculations and qualitative, value-based judgements. In April 2023, it was reported that image generation AI has resulted in 70% of the jobs for video game illustrators in China being lost. In July 2023, developments in generative AI contributed to the 2023 Hollywood labor disputes. Fran Drescher, president of the Screen Actors Guild, declared that "artificial intelligence poses an existential threat to creative professions" during the 2023 SAG-AFTRA strike. Voice generation AI has been seen as a potential challenge to the voice acting sector.

The intersection of AI and employment concerns among underrepresented groups globally remains a critical facet. While AI promises efficiency enhancements and skill acquisition, concerns about job displacement and biased recruiting processes persist among these groups, as outlined in surveys by Fast Company. To leverage AI for a more equitable society, proactive steps encompass mitigating biases, advocating transparency, respecting privacy and consent, and embracing diverse teams and ethical considerations. Strategies involve redirecting policy emphasis on regulation, inclusive design, and education's potential for personalized teaching to maximize benefits while minimizing harms.

Racial and gender bias

Generative AI models can reflect and amplify any cultural bias present in the underlying data. For example, a language model might assume that doctors and judges are male, and that secretaries or nurses are female, if those biases are common in the training data. Similarly, an image model prompted with the text "a photo of a CEO" might disproportionately generate images of white male CEOs, if trained on a racially biased data set. A number of methods for mitigating bias have been attempted, such as altering input prompts and reweighting training data.

Deepfakes

Deepfakes (a portmanteau of "deep learning" and "fake") are AI-generated media that take a person in an existing image or video and replace them with someone else's likeness using artificial neural networks. Deepfakes have garnered widespread attention and concerns for their uses in Deepfake pornography, deepfake celebrity pornographic videos, revenge porn, fake news, hoaxes, health disinformation, Accounting scandals, financial fraud, and covert foreign election interference. This has elicited responses from both industry and government to detect and limit their use.

In July 2023, the fact-checking company Logically (company), Logically found that the popular generative AI models Midjourney, DALL-E 2 and Stable Diffusion would produce plausible disinformation images when prompted to do so, such as images of electoral fraud in the United States and Muslim women supporting India's Hindu nationalist Bharatiya Janata Party.

In April 2024, a paper proposed to use blockchain (distributed ledger technology) to promote "transparency, verifiability, and decentralization in AI development and usage".

Audio deepfakes

Instances of users abusing software to generate controversial statements in the vocal style of celebrities, public officials, and other famous individuals have raised ethical concerns over voice generation AI. In response, companies such as ElevenLabs have stated that they would work on mitigating potential abuse through safeguards and Identity document, identity verification.

Concerns and fandoms have spawned from AI-generated music. The same software used to clone voices has been used on famous musicians' voices to create songs that mimic their voices, gaining both tremendous popularity and criticism. Similar techniques have also been used to create improved quality or full-length versions of songs that have been leaked or have yet to be released.

Generative AI has also been used to create new digital artist personalities, with some of these receiving enough attention to receive record deals at major labels. The developers of these virtual artists have also faced their fair share of criticism for their personified programs, including backlash for "dehumanizing" an artform, and also creating artists which create unrealistic or immoral appeals to their audiences.

Illegal imagery

Many websites that allow AI pornography, explicit AI generated images or videos have been created, and this has been used to create illegal content, such as Rape pornography, rape, child sexual abuse material, necrophilia, and zoophilia.

Cybercrime

Generative AI's ability to create realistic fake content has been exploited in numerous types of cybercrime, including phishing scams. Deepfake video and audio have been used to create disinformation and fraud. In 2020, former Google click fraud czar Shuman Ghosemajumder argued that once deepfake videos become perfectly realistic, they would stop appearing remarkable to viewers, potentially leading to uncritical acceptance of false information. Additionally, large language models and other forms of text-generation AI have been used to create fake reviews of e-commerce websites to boost ratings. Cybercriminals have created large language models focused on fraud, including WormGPT and FraudGPT.

A 2023 study showed that generative AI can be vulnerable to jailbreaks, reverse psychology and prompt injection attacks, enabling attackers to obtain help with harmful requests, such as for crafting Social engineering (security), social engineering and Phishing, phishing attacks. Additionally, other researchers have demonstrated that open-source models can be Fine-tuning (deep learning), fine-tuned to remove their safety restrictions at low cost.

Reliance on industry giants

Training Frontier model, frontier AI models requires an enormous amount of computing power. Usually only Big Tech companies have the financial resources to make such investments. Smaller start-ups such as Cohere and OpenAI end up buying access to data centers from Google and Microsoft respectively.

Energy and environment

AI has a significant carbon footprint due to growing energy consumption from both training and usage. Scientists and journalists have expressed concerns about the environmental impact that the development and deployment of generative models are having: high CO₂ emissions, large amounts of freshwater used for data centers, and high amounts of electricity usage. There is also concern that these impacts may increase as these models are incorporated into widely used search engines such as Google Search and Bing, as chatbots and other applications become more popular, and as models need to be retrained.

The carbon footprint of generative AI globally is estimated to be growing steadily, with potential annual emissions ranging from 18.21 to 245.94 million tons of CO2 by 2035, with the highest estimates for 2035 nearing the impact of the United States beef industry on emissions (currently estimated to emit 257.5 million tons annually as of 2024).

Proposed mitigation strategies include factoring potential environmental costs prior to model development or data collection, increasing efficiency of data centers to reduce electricity/energy usage, building more efficient Machine learning, machine learning models, minimizing the number of times that models need to be retrained, developing a government-directed framework for auditing the environmental impact of these models, regulating for transparency of these models, regulating their energy and water usage, encouraging researchers to publish data on their models' carbon footprint, and increasing the number of subject matter experts who understand both machine learning and climate science.

Content quality

''The New York Times'' defines AI slop, slop as analogous to Spamming, spam: "shoddy or unwanted A.I. content in social media, art, books and ... in search results." Journalists have expressed concerns about the scale of low-quality generated content with respect to social media content moderation, the monetary incentives from social media companies to spread such content, false political messaging, spamming of scientific research paper submissions, increased time and effort to find higher quality or desired content on the Internet, the indexing of generated content by search engines, and on journalism itself.

A paper published by researchers at Amazon Web Services AI Labs found that over 57% of sentences from a sample of over 6 billion sentences from Common Crawl, a snapshot of web pages, were Machine translation, machine translated. Many of these automated translations were seen as lower quality, especially for sentences that were translated across at least three languages. Many lower-resource languages (ex. Wolof language, Wolof, Xhosa language, Xhosa) were translated across more languages than higher-resource languages (ex. English, French).

In September 2024, Open Mind Common Sense, Robyn Speer, the author of wordfreq, an open source database that calculated word frequencies based on text from the Internet, announced that she had stopped updating the data for several reasons: high costs for obtaining data from Reddit and Twitter, excessive focus on generative AI compared to other methods in the natural language processing community, and that "generative AI has polluted the data".

The adoption of generative AI tools led to an explosion of AI-generated content across multiple domains. A study from University College London estimated that in 2023, more than 60,000 scholarly articles—over 1% of all publications—were likely written with LLM assistance. According to Stanford University's Institute for Human-Centered AI, approximately 17.5% of newly published computer science papers and 16.9% of peer review text now incorporate content generated by LLMs. Many academic disciplines have concerns about the factual reliably of academic content generated by AI.

Visual content follows a similar trend. Since the launch of DALL-E 2 in 2022, it is estimated that an average of 34 million images have been created daily. As of August 2023, more than 15 billion images had been generated using text-to-image algorithms, with 80% of these created by models based on Stable Diffusion.

If AI-generated content is included in new data crawls from the Internet for additional training of AI models, defects in the resulting models may occur. Training an AI model exclusively on the output of another AI model produces a lower-quality model. Repeating this process, where each new model is trained on the previous model's output, leads to progressive degradation and eventually results in a "model collapse" after multiple iterations. Tests have been conducted with pattern recognition of handwritten letters and with pictures of human faces. As a consequence, the value of data collected from genuine human interactions with systems may become increasingly valuable in the presence of LLM-generated content in data crawled from the Internet.

On the other side, synthetic data is often used as an alternative to data produced by real-world events. Such data can be deployed to validate mathematical models and to train machine learning models while preserving user privacy, including for structured data. The approach is not limited to text generation; image generation has been employed to train computer vision models.

Misuse in journalism

In January 2023, ''Futurism.com'' broke the story that CNET had been using an undisclosed internal AI tool to write at least 77 of its stories; after the news broke, CNET posted corrections to 41 of the stories.

In April 2023, the German tabloid ''Die Aktuelle'' published a fake AI-generated interview with former racing driver Michael Schumacher, who had not made any public appearances since 2013 after sustaining a brain injury in a skiing accident. The story included two possible disclosures: the cover included the line "deceptively real", and the interview included an acknowledgment at the end that it was AI-generated. The editor-in-chief was fired shortly thereafter amid the controversy.

Other outlets that have published articles whose content or byline have been confirmed or suspected to be created by generative AI models – often with false content, errors, or non-disclosure of generative AI use – include:

* NewsBreak
* outlets owned by Arena Group
** ''Sports Illustrated''
** TheStreet
** ''Men's Journal''
* B&H Photo
* outlets owned by Gannett
** ''The Columbus Dispatch''
** Reviewed (website), Reviewed
** USA Today
** ''Journal Star (Peoria), Journal Star''
** ''El Paso Times''
** ''Fort Collins Coloradoan''
** ''The Record (Stockton, California), The Record''
** ''The Augusta Chronicle''
** ''The Providence Journal''
** ''Argus Leader''
** ''Southwest Times Record''
** ''The Des Moines Register''
** ''North Jersey Media Group''
** ''Pocono Record''
* MSN
* News Corp
* outlets owned by G/O Media
** ''Gizmodo''
** ''Jalopnik''
** ''A.V. Club''
** ''Quartz (publication), Quartz''
** ''Deadspin''
** ''G/O Media, The Takeout''
* ''The Irish Times''
* outlets owned by Red Ventures
** Bankrate
* BuzzFeed
* Newsweek
* Hoodline
* outlets owned by Outside Inc.
** ''Yoga Journal''
** ''Backpacker (magazine), Backpacker''
** ''Clean Eating''
* Hollywood Life
* Us Weekly
* The Los Angeles Times
* Cody Enterprise
* Cosmos (Australian magazine), Cosmos
* outlets owned by McClatchy
** Miami Herald
** Sacramento Bee
** Tacoma News Tribune
** The Rock Hill Herald
** The Modesto Bee
** Fort Worth Star-Telegram
** Merced Sun-Star
** Ledger-Enquirer
** The Kansas City Star
** Raleigh News & Observer
* outlets owned by Ziff Davis
** PC Magazine
** Mashable
** AskMen
* outlets owned by Hearst Communications, Hearst
** Good Housekeeping
* outlets owned by IAC Inc.
** ''People (magazine), People''
** ''Parents (magazine), Parents''
** ''Food & Wine''
** ''InStyle''
** ''Real Simple''
** ''Travel + Leisure''
** ''Better Homes and Gardens (magazine), Better Homes & Gardens''
** ''Southern Living''
* outlets owned by Street Media
** LA Weekly
** The Village Voice
* Riverfront Times
* Apple Intelligence
In May 2024, Futurism noted that a content management system video by AdVon Commerce, who had used generative AI to produce articles for many of the aforementioned outlets, appeared to show that they "had produced tens of thousands of articles for more than 150 publishers."''''

News broadcasters in Kuwait, Greece, South Korea, India, China and Taiwan have presented news with anchors based on Generative AI models, prompting concerns about job losses for human anchors and audience trust in news that has historically been influenced by parasocial relationships with broadcasters, content creators or social media influencers. Algorithmically generated anchors have also been used by allies of ISIS for their broadcasts.

In 2023, Google reportedly pitched a tool to news outlets that claimed to "produce news stories" based on input data provided, such as "details of current events". Some news company executives who viewed the pitch described it as "[taking] for granted the effort that went into producing accurate and artful news stories."

In February 2024, Google launched a program to pay small publishers to write three articles per day using a beta generative AI model. The program does not require the knowledge or consent of the websites that the publishers are using as sources, nor does it require the published articles to be labeled as being created or assisted by these models.

Many defunct news sites (''The Hairpin'', ''The Frisky'', ''Apple Daily, Ashland Daily Tidings'', ''List of newspapers in Iowa, Clayton County Register'', ''Southwest Journal'') and blogs (''The Unofficial Apple Weblog'', ''iLounge'') have undergone cybersquatting, with articles created by generative AI.

United States Senators Richard Blumenthal and Amy Klobuchar have expressed concern that generative AI could have a harmful impact on local news. In July 2023, OpenAI partnered with the American Journalism Project to fund local news outlets for experimenting with generative AI, with Axios noting the possibility of generative AI companies creating a dependency for these news outlets.

Meta AI, a chatbot based on Llama 3 which summarizes news stories, was noted by ''The Washington Post'' to copy sentences from those stories without direct attribution and to potentially further decrease the traffic of online news outlets.

In response to potential pitfalls around the use and misuse of generative AI in journalism and worries about declining audience trust, outlets around the world, including publications such as ''Wired (magazine), Wired'', Associated Press, The Quint, Rappler or ''The Guardian'' have published guidelines around how they plan to use and not use AI and generative AI in their work.

In June 2024, Reuters Institute published their ''Digital News Report for 2024''. In a survey of people in America and Europe, Reuters Institute reports that 52% and 47% respectively are uncomfortable with news produced by "mostly AI with some human oversight", and 23% and 15% respectively report being comfortable. 42% of Americans and 33% of Europeans reported that they were comfortable with news produced by "mainly human with some help from AI". The results of global surveys reported that people were more uncomfortable with news topics including politics (46%), crime (43%), and local news (37%) produced by AI than other news topics.

See also

*
*
*
*
*
*
*
*
*
*
*
*
*
*

References

Further reading

*

{{Authority control

Artificial neural networks
Deep learning
Machine learning
Generative artificial intelligence,
2020s in computing
2023 in computing
2024 in computing
2025 in computing