Synthetic media

Also called Deepfakes, AI-generated propaganda, Machine-made content

Machine-made or machine-rewritten content presented as human witness, journalism, or research.

History and effect

The production-line successor to older forgery techniques, enabled by generative text, image, and video models; deepfakes and AI-anchor video are its most visible forms.

It presents machine output with the evidentiary weight of eyewitness or expert testimony, trading on the audience's assumption that a human is vouching for what they see.

Lineage: Deepfakes; generative text/image/video as a production line

How it works

Synthetic media joins two things: content produced or substantially altered by a machine, and a presentation that tells the audience a person or a camera stands behind it. Face generators (first generative adversarial networks, or GANs, then diffusion models), voice cloning, text-to-video avatars and large language models each replace a human job that used to limit volume or realism: photographer, voice actor, presenter, writer. What documented operations share is less the tool than the wrapper. A generated face becomes the profile photo of an account with a name, a hometown and a job. A cloned voice becomes a phone call or a "leaked" recording. An avatar becomes the anchor of a channel with a news-style name. A batch of model-written articles becomes the output of an "institute" or a local news site.

The audience sees the usual marks of testimony (a face, a voice, a byline, a masthead, a timestamp during a breaking event), and the message depends on those marks going unchecked long enough to be shared. The archive shows the common routes. The AI-generated image of the Hollywood sign on fire image borrowed the look of an eyewitness photo during a real wildfire. The AI-generated image of Donald Trump being arrested series was labelled as invented by its creator, and the label was lost as copies spread. The 'Gennady Rakitin,' an invented pro-war poet account gave a generated face a biography and a poetry career. In Matryoshka's midterm clips: borrowed faces and a borrowed logo, real celebrity footage was recut with cloned audio and a news logo, and in The Hanover Institute: reports written for chatbots to cite, unsigned text that a detector scored as AI-written was published as policy research.

Two structural points matter for analysis. First, synthetic content is usually one component inside a larger operation (fake accounts, laundered sources, paid or coordinated amplification), and it is that operation, not the file, that supplies reach. Platform and researcher reports repeatedly describe polished synthetic assets that drew little authentic engagement (OpenAI 2024; Graphika 2023). Second, the existence of convincing fakes changes how real evidence is received. Robert Chesney and Danielle Citron called this the "liar's dividend": once the public knows fakes are possible, a person caught on a genuine recording can claim it was fabricated (Chesney and Citron 2019).

Why it works

The first reason is the truth default. Timothy Levine's Truth-Default Theory holds that people presume what they see and hear is honest unless something specific triggers suspicion, a default that is efficient and usually correct (Levine 2014). Synthetic media is built to avoid those triggers by looking like evidence people have long taken at face value: a photograph, a familiar voice, a person on camera.

Images add weight even when they prove nothing. In experiments by Eryn Newman and colleagues, pairing a claim with a related photograph that offered no evidence for it made people more likely to judge the claim true (Newman et al. 2012). Sophie Nightingale and Hany Farid found that participants distinguished StyleGAN2 faces from real photographs at close to chance, and rated the synthetic faces as slightly more trustworthy (Nightingale and Farid 2022). Nils Köbis and colleagues found that people could not reliably detect deepfake videos yet were confident they could (Köbis, Doležalová and Soraperra 2021). Generation also makes repetition cheap, and repeated claims feel more true (Hasher, Goldstein and Toppino 1977). FRAME's Learn library covers the repetition effect in Illusory truth effect.

The size of the effect is contested, and it is smaller than early warnings suggested. Chloe Wittenberg and colleagues found that political video was more believable than the same content as text, but that its advantage in changing attitudes was small (Wittenberg et al. 2021). Cristian Vaccari and Andrew Chadwick, testing a deepfake of Barack Obama on a UK sample, found that fewer viewers were outright deceived than were left uncertain, and that the uncertainty was associated with lower trust in news (Vaccari and Chadwick 2020). On current evidence, synthetic media often does its damage less by selling one fake than by raising doubt about all evidence.

Examples across eras and sides

• Hand-made forerunner: Soviet photo retouching (1920s to 1950s). Under Stalin, officials who fell from favor were removed from photographs by retouchers, and the altered prints were published as the historical record (Sources: King 1997). This was manual work, included only as the forerunner the modern technique automates.
• Generated faces on fake accounts (2019 to 2022, United States and elsewhere). In December 2019 Facebook removed a network it linked to the U.S.-based Epoch Media Group whose accounts used AI-generated profile photos, one of the first large documented uses of GAN faces (Sources: Facebook 2019; Graphika and DFRLab 2019). In 2022 Graphika and the Stanford Internet Observatory documented a pro-Western covert network on several platforms that also used GAN-generated faces, and Meta attributed a network in its takedown to individuals associated with the U.S. military (Sources: Graphika and SIO 2022; Meta 2022).
• Wartime deepfake of a head of state (Ukraine, March 2022). A video in which a synthetic Volodymyr Zelensky appeared to tell Ukrainian soldiers to lay down their arms was posted after hackers compromised the website of the broadcaster Ukraine 24. Zelensky rebutted it on video, and Meta removed copies (Sources: Allyn 2022). It was widely described as crude and did not achieve its apparent aim.
• AI news anchors (China-aligned, 2022 to 2023). Graphika documented the pro-Chinese Spamouflage network promoting videos of a fictitious outlet, "Wolf News," whose presenters were avatars produced with a commercial text-to-video service. The videos drew very little authentic engagement (Sources: Graphika 2023).
• Election-eve audio (Slovakia, September 2023). Days before the parliamentary election, an audio clip circulated in which voices resembling Progressive Slovakia leader Michal Šimečka and journalist Monika Tódová appeared to discuss rigging the vote. Both said it was fake, and fact-checkers found signs of synthesis. It spread during the legal pre-election silence period, when rebuttal in the media was restricted (Sources: Meaker 2023).
• U.S. domestic campaigns, both parties' orbits (2023 to 2024). In June 2023 the DeSantis presidential campaign's "DeSantis War Room" account posted a video attacking Donald Trump that included apparently AI-generated images of Trump embracing Anthony Fauci (Sources: Nehamas 2023). In January 2024 a robocall using a cloned voice of President Joe Biden told New Hampshire Democrats not to vote in the primary; it was commissioned by Steve Kramer, a consultant then working for the Democratic challenger Dean Phillips, whose campaign said it had no involvement. The Federal Communications Commission fined Kramer $6 million (Sources: FCC 2024).
• Model-written text at scale (several states and a commercial firm, 2023 to 2024). OpenAI reported disrupting operations that used its models to generate comments and articles: two Russian (Bad Grammar and Doppelganger), one Chinese (Spamouflage), one Iranian (the International Union of Virtual Media) and one run by STOIC, a political marketing firm in Tel Aviv. Meta separately removed the STOIC network. OpenAI reported that none of the five had achieved significant authentic reach (Sources: OpenAI 2024; Meta 2024). They differ in structure and duration: the Russian and Chinese networks are long-running state-aligned efforts, while STOIC was a commercial firm that The New York Times reported was paid by Israel's Ministry of Diaspora Affairs for the campaign (Sources: Frenkel 2024).

How to spot it

• Trace the item to its first appearance with a reverse image or video search. An earliest post that says "I made this" or "AI-generated," as with the Trump arrest images, settles much of the question.
• Check the event against independent sources: a fire or an arrest shown only in one viral image, and in no agency or local report, is suspect.
• For profile photos, look for the regularities researchers have used to identify GAN networks: eyes in the same position in every face, mismatched earrings or glasses, distorted hair edges and warped backgrounds (Graphika and DFRLab 2019). Newer generators make fewer of these errors.
• Look for residual machine text: posts containing a model's refusal message, stock assistant phrasing, or the same sentence structure repeated across supposedly unrelated accounts (OpenAI 2024).
• Ask whether the source can be reached: a presenter with no employer, or an "institute" or news site with no named staff, address or legal entity, is a warning sign.
• Treat timing as a signal. Audio or video released in the final days before a vote, or during a legal silence period, arrives when rebuttal is hardest.
• Use detector scores and provenance data (such as C2PA Content Credentials) as evidence, not proof. Detectors give probabilities and err in both directions, and missing credentials prove nothing on their own.

Where it ends: edge cases and legitimate persuasion

The technique ends where disclosure travels with the content. An AI-generated illustration labelled as such, consented voice dubbing, or a clearly marked reconstruction is not synthetic media in FRAME's sense, because nobody is asked to treat it as a record. The Republican National Committee's April 2023 advertisement imagining a second Biden term was built from generated images and carried an on-screen disclosure saying so (Sources: RNC 2023): a disclosed political illustration, persuasive but not deceptive about its making. The AI-generated image of Donald Trump being arrested case sits on the line: disclosed at the source, it became synthetic witness once copies spread without the label. The 'Gennady Rakitin,' an invented pro-war poet persona shows that motive does not change the mechanism: it was built by anti-war exiles as a test, but while the source was hidden it worked exactly like any invented grassroots voice.

Neighbouring techniques overlap. Generated faces are the modern supply for Astroturfing and Manufactured consensus, borrowed logos and famous faces are Transfer, text written for retrieval systems is Corpus poisoning, and routing machine output through a front outlet is Narrative laundering. "Cheapfakes," real footage that is slowed, cut or miscaptioned, do similar work without generation and belong under Card stacking rather than here.

The test a reader can apply: if you learned how this item was made and who commissioned it, would it lose its force as evidence? If it presents itself as a photograph of an event, the voice of a real person, or the work of a human witness or researcher, and a machine produced that element without telling you, it is synthetic media. If the making is disclosed where you encounter it, you are looking at an illustration and can judge its argument on its merits.

Sources

• Allyn, Bobby. 2022. "Deepfake Video of Zelenskyy Could Be 'Tip of the Iceberg' in Info War, Experts Warn." NPR, 16 March 2022.
• Chesney, Robert, and Danielle Citron. 2019. "Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security." California Law Review 107(6): 1753-1820.
• Facebook. 2019. "Removing Coordinated Inauthentic Behavior From Georgia, Vietnam and the US." Facebook Newsroom, 20 December 2019.
• Federal Communications Commission. 2024. Forfeiture order against Steven Kramer for the January 2024 New Hampshire robocalls ($6,000,000). FCC, September 2024.
• Frenkel, Sheera. 2024. "Israel Secretly Targets U.S. Lawmakers With Influence Campaign on Gaza War." The New York Times, 5 June 2024.
• Graphika and Atlantic Council Digital Forensic Research Lab. 2019. #OperationFFS: Fake Face Swarm. Graphika and DFRLab, December 2019.
• Graphika and Stanford Internet Observatory. 2022. Unheard Voice: Evaluating Five Years of Pro-Western Covert Influence Operations. Stanford Internet Observatory, August 2022.
• Graphika. 2023. Deepfake It Till You Make It: Pro-Chinese Actor Promotes AI-Generated Video Footage of Fictitious People. Graphika, February 2023.
• Hasher, Lynn, David Goldstein, and Thomas Toppino. 1977. "Frequency and the Conference of Referential Validity." Journal of Verbal Learning and Verbal Behavior 16(1): 107-112.
• King, David. 1997. The Commissar Vanishes: The Falsification of Photographs and Art in Stalin's Russia. New York: Metropolitan Books.
• Köbis, Nils C., Barbora Doležalová, and Ivan Soraperra. 2021. "Fooled Twice: People Cannot Detect Deepfakes but Think They Can." iScience 24(11): 103364.
• Levine, Timothy R. 2014. "Truth-Default Theory (TDT): A Theory of Human Deception and Deception Detection." Journal of Language and Social Psychology 33(4): 378-392.
• Meaker, Morgan. 2023. "Slovakia's Election Deepfakes Show AI Is a Danger to Democracy." Wired, 3 October 2023.
• Meta. 2022. Quarterly Adversarial Threat Report, Q3 2022. Meta, November 2022.
• Meta. 2024. Quarterly Adversarial Threat Report, Q1 2024. Meta, May 2024.
• Nehamas, Nicholas. 2023. "DeSantis Campaign Uses Apparently Fake Images to Attack Trump on Twitter." The New York Times, 8 June 2023.
• Newman, Eryn J., Maryanne Garry, Daniel M. Bernstein, Justin Kantner, and D. Stephen Lindsay. 2012. "Nonprobative Photographs (or Words) Inflate Truthiness." Psychonomic Bulletin & Review 19(5): 969-974.
• Nightingale, Sophie J., and Hany Farid. 2022. "AI-Synthesized Faces Are Indistinguishable from Real Faces and More Trustworthy." Proceedings of the National Academy of Sciences 119(8): e2120481119.
• OpenAI. 2024. AI and Covert Influence Operations: Latest Trends. OpenAI, May 2024.
• Republican National Committee. 2023. "Beat Biden." Video advertisement, released 25 April 2023.
• Vaccari, Cristian, and Andrew Chadwick. 2020. "Deepfakes and Disinformation: Exploring the Impact of Synthetic Political Video on Deception, Uncertainty, and Trust in News." Social Media + Society 6(1).
• Wittenberg, Chloe, Ben M. Tappin, Adam J. Berinsky, and David G. Rand. 2021. "The (Minimal) Persuasive Advantage of Political Video over Text." Proceedings of the National Academy of Sciences 118(47): e2114388118.

The science

Research on the psychology this technique relies on, from the Learn library:

• Illusory truth effect — Illusory truth effect

Case studies

• The Hanover Institute: reports written for chatbots to cite — The Hanover Institute: reports written for chatbots to cite
• Matryoshka's midterm clips: borrowed faces and a borrowed logo — Matryoshka's midterm clips: borrowed faces and a borrowed logo
• Israel's Strategic Affairs Ministry paid for news content, and a later ministry's STOIC campaign aimed at U.S. lawmakers — Israel's Strategic Affairs Ministry paid for news content, and a later ministry's STOIC campaign aimed at U.S. lawmakers