What Do Deepfakes Want? Using ‘Digital Forensic Gaze’ and Digital Methods to See (like) Deepfakes

Team Members

Facilitators: Natalia Stanusch, Richard Rogers

Design Facilitators: Noemi Capparelli, Luca Bottani

Participants: Emma Garzoino, Marius Liedtke, Deborah Nyangulu, Omer Rothenstein, Chufan Huang, Michelle Stewart, Shuyu Zhang, Quyang (July) Zhao

Links

posters

Contents

1. Introduction

This project begins in the middle of multiple controversies around deep fakes and generative AI (genAI) images and their rapid ease and reach in production and spread. In January 2024, graphic synthetic images of Taylor Swift flooded X (formerly known as Twitter). As the incident snowballed into a controversy around deepfake pornography, Taylor Swift fans mobilized to take the images down before the platform stepped in. The controversy over Princess of Wales' avowed editing of an official Mother’s Day photo of her surrounded by her children launched an online outcry. Princess of Wales' admission occasioned collective reflections regarding “authentic” and reliable information in an age where digital editing is widely accessible. Tagged online as #KateGate, the controversy points to public standards of and struggles over truth. While deepfakes and genAI are not exactly the same, they raise similar questions about circulation, moderation, and violence. What brings together the use of deepfakes and the pornographic genAI images of Taylor Swift as well as the dissemination and accusations on the inauthetnticity of photos of Princess of Wales is a notion of stochastic violence. Stochastic violence is the use of free and indiscriminate violence on the internet that can be performed via a seemingly decentralized spread of images to execute violence and trolling.

As image manipulation becomes easier and less expensive with the availability of AI tools, “Deep Fakes,” images altered via photo/video editing tools and GenAI are circulating rapidly on social media, engendering debates over the ethical, aesthetic and political stakes of digitally-altered images. Because the technology is still in its early stages, it is often still possible to identify such images with our bare eyes. In this project, we explore how communities of users participate in detailed discussions regarding the eligibility and authenticity of images, how they flag and identify what counts as authentic or inauthentic, doctored or real, and how they imagine and advocate ethical standards that should apply to public and private figures.

In this manner, users and platforms participate in mobilising the “digital forensic gaze”, defined by Lavrence& Cambre (2020) as a way of looking at digital objects that centres around the tension between manipulated and authentic. Though focused on selfie filters within social media feeds, “digital forensic gaze” (Lavrence& Cambre, 2020) has been conceptualized as a way of looking on digital objects (selfies) that centers around the tension between manipulated and authentic; “while the viewer witnesses and confirms the photograph’s veracity, the viewing subject experiences a schism, whereby the digital-forensic gaze is also paradoxically trained to incriminate and look for signs of pathology” (Lavrence& Cambre, 2020, p. 5). Unlike selfies (and the use of filters in them), which have very specific, well-established functions and expectations on both the user’s and viewer’s sides, deepfakes and genAI images are still a phenomenon in formation, allowing one to track what currently is and could be “digital forensic gaze” as applied to these objects and practices.

This case study focuses on the following two main research questions:

RQ1: How can we unpack and operationalize the “digital forensic gaze” (Lavrence& Cambre, 2020) to understand and map the way deepfake and AI images work in and across social media sites?

RQ2: What are the dominant framings, the visual language of evidence, and actors engaged in spreading and securitizing deepfakes and genAI images based on fabricated images of Taylor Swift and Kate Middleton?

and the case study-spesific sub questions:

#TaylorSwiftAI case study:

RQ1: By considering X (Twitter) as a story-telling machine, how can (re)construct an X (Twitter) event that the explicit generative AI images of Taylor Swift constituted?

RQ2: How can we retrace and map out the lifecycle (or a ‘pipeline’) of generative AI images and deepfakes as seen in the #TaylorSwiftAI controversy?

RQ3: What are the dominant narratives and activated approaches towards synthetic images that develop under #ProtectTaylorSwift and #TaylorSwiftAI on X?

#KateGate case study:

RQ1: What are the dominant framings, the visual language of evidence, and actors engaged in spreading and securitizing the manipulated and synthetic images of #KateGate controversy?

RQ2: How can we define and understand the role of the platforms’ gaze and the users’ gaze in scrutinizing synthetic imagery such as deepfakes and genAI images?

2. Initial Data Sets

To document these processes, we began with a dataset that was collected between 12.03.2024-21.03.2024 and consisted of a total of 39000 tweets. We filtered this dataset by selecting posts tagged with the hashtags #KateGate, #katespiracy, #WhereIsKate and scraping the ‘top’ results until X.com stopped returning responses to our query. The next step was to “snowball” the collection, which entailed using the retweets as an entry point: the accounts of users who retweeted and quoted the ‘top posts’ were then accessed, and their feeds were scraped. In addition to scraping the retweets, quoted tweets, replies and author’s page, if the author’s page had another tweet on the topic, the page was scraped according to the quotes and retweets, following the snowballing method. This process produced a collection of tweets and retweets containing three hashtags: #KateGate, #katespiracy, #WhereIsKate. We also cleaned the data set to remove scraped tweets and RT’s that fell outside of the period under analysis, leaving a dataset of 3500 tweets.

Another entry point for the data collection was to use Zeeschuimer to scrape a total of 54,000 tweets by operationalizing the hashtags #ProtectTaylorSwift and #AITaylorSwift. The data collection was performed from Feb 12, 2024 to Feb 26, 2024 due to the daily limits of displayed tweets imposed by X. First, the ‘top’ tweets per each hashtag were collected, followed by the retweets of those posts, the feeds of users who retweeted them, and the retweets of relevant posts from their feeds. The dataset was cleaned using 4CAT, filtering out content from one week (Jan 24 to Jan 31) that contained the two core hashtags.

3. Methodology

#KateGate case study:

We began by processing our CSV file with 4CAT applying the co-tag network analysis function. The file was then exported to Gephi to analyse the resulting network and detect relevant communities (filtering by modularity class). Once major conversational clusters were identified, we focused on the biggest cluster around #KateGate. The top 100 most retweeted and engaged with (#comments + #retweets + #likes) posts were qualitatively analysed and filtered for relevance. 42 highly retweeted and high-engagement posts remained. We then qualitatively focused on the hashtag #KateGate, assessing the recurring themes found in the top 3 retweeted and engaged with posts for each day between 10.03. and 17.03.2024.

Due to the interest in the different ways in which “digital forensic gaze” (Lavrence & Cambre, 2020) is mobilised by different online communities, we pursued several different analyses of the reduced corpus. To identify conflicting ground truths, the “official” and “unofficial” mechanisms for weighing the visual evidence of media events were examined closely. X.com’s official Community Notes (https://communitynotes.x.com/guide/en/about/introduction) were analysed to see which posts were moderated by the platform or elicited a response from community members. X’s Community Notes details the moderation actions taken by the platform managers, explains their decisions, and affords a space for community members to report problematic content (disinformation, unfounded opinion, questions of pertinence). Secondly, posts that used various forms of visual media to “investigate” the origins of the royal photo, that we had tagged as “OSINT” (Open Source Intelligence), were identified. These included close analyses of the photo released by the royal family, comparisons to earlier media released by the royal family, and references to, as well as reposts of, mainstream British and American news sources. In each of these two explorations, the aim was to document the ways in which various online communities mobilise platform affordances to establish a shared sense of truth – a community ground truth composed of agreed upon standards (an informal canon) at once technical, ethical, aesthetic, and political – in order to understand and map the way deep fakes and AI images work in and across social media sites.

Finally, a close reading of the overall corpus by day was performed, selecting the top 3 original (doubles were noted but not reanalysed), most retweeted and the top 3 highest engagement posts for the week following from the release of the royal photo, from March 10 to March 17.

#TaylorSwiftAI case study:

A network analysis was conducted using Gephi to identify co-hashtag networks and retweet networks. We used Gephi to identify clusters based on relevance, centrality, and associated hashtags. The use of Gephi to create chronological and epistemological timelines helps in mapping the dissemination of Taylor Swift's deepfakes on Twitter. This tool visualizes the spread and evolution of deepfake content, identifying key clusters and patterns of knowledge production and spread. Detailed analysis of clusters based on relevance, centrality, and associated hashtags reveals how different communities and themes are connected to the dissemination of deepfakes. This helps in understanding the multifaceted nature of the spread.

For the qualitative content analysis, we turned to additional related content (e.g., news articles), as well as the close-reading of the most retweeted actors from the Gephi retweet network per cluster. Based on these mixed data sources, we reassembled a chronological pipeline - or lifecycle - of synthetic (genAI) images of #TaylorSwiftAI. By looking at the available investigations on the deepfakes of Taylor Swift, we outlined, in a chronological way, the pipeline of actors, platforms, and companies involved, actively and passively, in the production, dissemination, and moderation of the deepfakes. For the one-week timespan of the tweets, we created two spreadsheets filtering tweets by top engagement and retweets per day to identify three main narrative labels. Each member of the team individually coded the tweets, resulting in the differentiation among three main narratives, followed by an inter-coder discussion to read reliability and select the most relevant tweets and images from the final spreadsheet.

4. Findings

#KateGate case study:

Three main narratives emerged from the analysis: 1) the “Investigative”, OSINT narrative, 2) royal drama and critique of the monarchy, and 3) satiric and humoristic discourse (jokes, parodies, and memes).

Across all three narratives, images are mobilised in a variety of ways – sharing, editing, comparing, evidentiary analysis, gossip and violence.

Posts that were tagged as “OSINT” deployed online sleuthing techniques. While the initial image was manipulated either by Kate herself or on her behalf, users quickly took it upon themselves to post “enhanced” versions of it. OSINT sleuthing manifested in user efforts to provide “clues” that a “nefarious” plot was taking place in the official communication channels of the royal family. Users participating in constructing this investigative narrative mobilised a variety of sources and techniques: reposting “official” news sources, as well as videos and images dissecting the initial photo via enlarging or zooming in on specific areas, layering other images on it, inserting visual elements, or taking existing images of Kate and comparing them side-by-side to the official photo to demonstrate that the image had been altered.

The second main narrative thread, “Royal drama”, centred upon the royal family and oscillated between gossip and political critique of the institution of the monarchy. We named this thread “Royal drama” due to the preponderance of interest in specific royal family members and the controversies between them. There was a marked sub-narrative regarding critiques of Meghan Markle, but the thread also referenced various ongoing and historical controversies around the royal family (Harry vs. William, the treatment of Princess Diana). We consider it important to note that comments that might be discounted as “gossip” in some contexts offer legitimate political critique of the royal family, either of the monarchy in general, of their public relations work (preferential or unfair treatment by the British or American press), or of the racial politics of the royal family in their treatment of Meghan Markle.

Throughout the controversy, many users took the opportunity to make jokes, memes, and other forms of satirical content, mainly at the expense of the royal family and its members. The posts were also meta-critical, poking fun at the internet frenzy and media attention it gathered. While satire and memes were present throughout the period analysed (the “media event”), we note that most of the satiric content appeared in the initial stages of the event, with smaller peaks following key events like Kate’s apology post issued on March 11th. Overall, satirical posts did not attract the same level of engagement as the OSINT and Royal drama threads.

We also note that there was a chronological dimension to the KateGate media event. The three narratives each peaked at different moments, though they overlapped at times, and were all present to some degree throughout the event. The focus of the controversy – evidentiary work, gossip and satire– seemed to follow a general chronological flow established by moments of secondary controversy. The first significant controversy followed the “kill notice”, the decision by major news agencies to remove the altered photo from their websites. News of the kill notice broke on March 10, 2024, followed by Kate’s admission of having edited the photo herself (March 11th). These events immediately launched factual and counterfactual sleuthing work by users, media companies and platforms. Memes and gossip emerged at the same time, but seemed to peak in the middle of the week. By the end of the week, more conspiracy-oriented threads dominated the discussion – for example, #WhereIsKate? – but with less engagement as the controversy dissipated and mainstream interest diminished.

Throughout the week, the three narrative threads not only overlap, but they feed one another. Users can and do engage in some degree of “digital forensic” work, weighing what kind of image work by public officials (like Kate and the royal family) is legitimate, linking to and reposting mainstream news reports, gossip magazine pieces, and mainstream satire (for example, The Colbert Report or The New Yorker magazine). In short, images, videos, and informational sources of varied provenance and quality serve as evidence.

Platforms also do varied degrees of moderation. Of the 42 relevant posts qualitatively analysed, only 7 had a “community note” attached by X.com. Significantly, none of these notes were shown next to the post and some of the users’ arguments, visible only via the Community Notes page, were also hidden there. Once X decides not to show a debate, one must go to Community Notes, scroll down a page of text, and then enter the specific URL of a post in order to see the controversy and moderation decisions. However, users do visit Community Notes to debate the pertinence of other users’ objections to content and weigh whether notes are “true” or falsifiable or based on fact or personal opinion. Users also link to further sources. And, here too, some satirical comments and memes appear.

#TaylorSwiftAI case study:

The three narratives that emerged are 1) the online “hunting” and reporting of perpetrators while calling for regulation/legislation of genAI content, 2) burying the harming, gaenAI content by sharing “positive,” unrelated tweets by using (and hijacking) the hashtags #taylorswiftai and #protecttaylorswift; c) humanizing Taylor, namely expressing sympathy and stressing the emotional burden she had to endure. The co-hashtag network allowed us to notice the presence and prominence of some thematic clusters, from which emerge the growing concern and call for regulation of genAI images and deepfake content; an extremely cross-cutting issue with extensive ethical-legal implications. Hashtag hijacking cases mainly involved the war on #Gaza and the call for #ceasefire.

In the Co-Hashtag analysis, the results provided a detailed categorization of various clusters, highlighting their relevance, centrality, and associated hashtags. Clusters related to Taylor Swift included discussions about her tours, AI and deepfake safety, and fan activities. Key clusters such as "Protection of T.S. and deep-fakes related themes" and "T.S fanbase (Swifties) and Gaza" show how different themes are interconnected in the discussions on Twitter.

In the retweet analysis over one week and one month datasets, helped in identifying the most influential accounts and how their tweets contributed to the spread of information. For example, certain authors like "tswiftlover1389" and "noejiat" were highly active in spreading content within specific clusters.Clusters identified in the retweet analysis also shed light on the main narratives being retweeted, such as discussions about Taylor Swift's tour, political content, and other heterogeneous themes. The visual language in these posts likely includes memes, images, and videos that are easily shareable and have high engagement rates, contributing to the spread of deepfake content. Some clusters show the different communities and topics interlinked with Taylor Swift’s deepfakes. For instance, clusters included themes like political discussions, protection and safety of Taylor Swift, her fanbase, and various unrelated topics such as AI, tech, and even pornography-related content. Influencers and coordinated networks play a critical role in spreading deepfake content. The document identifies key actors through their high occurrence counts in retweet clusters. These actors, such as "tswiftlover1389" and "khabridadi," were instrumental in disseminating deepfake-related narratives.

The qualitative content analysis collected 258 results containing phases, timestamps, tweet content, and actors, contributing to form a pipeline. We highlighted three main phases of the lifecycle of genAI/deepfake images: production, circulation, and moderation. The three narratives we found constitute three ways of community protection, which are intertwined with actions taken by the users to reduce the visibility of genAI content. We note the absence of the material of the actual genAI images, in the dataset, suggesting a rather successful burying and taking down of content by both Taylor Swift fans and the platform.

5. Discussion

Although Kate Middleton is a public figure, who must accept a certain degree of public scrutiny and critique, we note that the sheer volume and rapidity of the online and mediated discussion of #KateGate constitutes a kind of stochastic violence (cf. Angove 2024 and Bender et al. 2021). Stochastic violence is characterised by the decentralised spread of images, intensive and swarmlike harassment, and violence online, that particularly targets women. This swarmlike harassment has roots in trolling and deeper internet subcultures. The rapid dissemination of random and swift attention, criticism and violence affects the full participation of women online, promoting self-critique or self-cancellation, i.e, abstention from cultural and political life on and offline (Ging 2023). In our study, memes and gossip constituted the more visible potential forms of stochastic violence, but the sleuthing efforts in the OSINT threads also contributed. As Lavrence and Cambre (2020, p.5) note, “while the viewer witnesses and confirms the photograph’s veracity, the viewing subject experiences a schism, whereby the digital-forensic gaze is also paradoxically trained to incriminate and look for signs of pathology”.

Understanding the patterns and dynamics of how the public shares, reacts to, and interprets deepfake content through analyzing and (re)framing Twitter events framed by Taylor Swift’s deepfake is critical to developing strategies to counter disinformation and educating the public on critical media consumption. It helps identify key influencers, peak activity times, and types of narratives that gain traction, thereby informing policy and platform design to mitigate the spread of disinformation. By studying the lifecycle of deepfakes, researchers can identify key stages where interventions are most effective. This includes creation, dissemination, peak of influence, and eventual debunking or fading away. Understanding these stages helps design timely interventions to reduce the impact of deepfakes. For example, platforms can take steps to detect and flag deepfakes early in their lifecycle, minimizing their reach and impact. By revealing how deepfakes affect public perception and narrative formation, three types of narritives are used to assess the psychological and social impact of deepfakes on Taylor, user, and collective beliefs. Insights drawn from this analysis can help raise awareness of deepfakes and their potential to distort reality. It can also inform the development of tools to effectively detect and debunk false narratives. This research can guide stakeholders such as celebrities, media professionals, and policymakers in addressing the vulnerabilities and responsibilities that come with this influence. It also highlights the need for strong measures to protect individuals from reputational damage caused by deepfakes. The knowledge from this research can be used to develop more sophisticated detection algorithms and ethical guidelines for creators and consumers of digital content. It highlights the need for a collaborative approach among technologists, ethicists, and policymakers to address the deepfake phenomenon. Insights gained from studying the life cycle of deepfakes can help develop policies and regulations aimed at controlling their spread and mitigating their impact. This research can provide evidence-based recommendations for legislation to protect individuals and society from the harms caused by deepfakes. It can also support the establishment of a framework for international cooperation between social media platforms and combating digital false information.

6. Conclusions

While conspiracies are abundant in the online sphere, they usually stay on the fringes of the general discourse. Every once in a while, an event can push them to the front stage, when an event or important public figure adds the right amount of fuel to the fire, lighting up the conversation about them. Such was the case of #KateGate, when one poorly modified/fabricated photo led to the convergence of several communities around a single topic. #KateGate points to the ways in which online controversies are marked by the blurring of the lines between evidentiary work, gossip, satire, political critique, and stochastic violence in cases where female public figures are involved. In addition to its outsized speed, volume and intensity, stochastic violence is often sexualised and racialised. Thus, quite apart from the themes of the content shared, modes of critical engagement can contribute to online violence. In this particular controversy, all three threads mobilised the digital forensic gaze to establish their ground truths, be these questions of inauthentic public representation, “Fake Kate” vs. Meghan, jokes, or pointed critiques of the continued existence of the monarchy in a modern democracy.

This topic is highly relevant due to the increasing prevalence of deepfakes and AI-generated images, which pose significant risks to misinformation, privacy, and reputational harm. By understanding the mechanisms behind their creation, dissemination, and perception, we can develop better strategies for detection, regulation, and public awareness. The focus on Taylor Swift underscores the broader implications for digital security and the societal need to address these challenges. The interaction between platform algorithms and user behaviors significantly affects the lifecycle of deepfakes. Algorithms that promote engaging content can amplify the spread of deepfakes, while user behaviors like retweeting and hashtag usage increase visibility. Effective moderation is crucial for managing the spread of deepfakes. Delays or inefficiencies in moderation practices can allow deepfakes to gain significant reach before being addressed. This research contributes to the broader discourse on the ethical, social, and technical aspects of deepfake technology.

7. References

Angove, J., 2024. Stochastic terrorism: critical reflections on an emerging concept. Critical Studies on Terrorism, 17(1), pp.21-43.

Bender, E.M., Gebru, T., McMillan -Major, A. and Shmitchell, S., 2021, March. On the dangers of stochastic parrots: Can language models be too big?🦜. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 610-623).

Ging, D., 2023. Digital Culture, Online Misogyny, and Gender‐based Violence. The Handbook of Gender, Communication, and Women's Human Rights, pp.213-227.

Lavrence, Christine and Cambre, Carolina. (2020). “‘Do I Look Like My Selfie?’: Filters and the Digital-Forensic Gaze.” Social Media + Society, 6(4). https://doi.org/10.1177/2056305120955182.

Rogers, Richard. (2018). Otherwise Engaged: Social Media from Vanity Metrics to Critical Analytics. International Journal of Communication 12(2018), 450–472.

Steyerl, Hito. (2012). The Wretched of the Screen. Berlin: Sternberg Press.

Steyerl, Hito. (2013). Too Much World: Is the Internet Dead? e-Flux, (49). https://www.e-flux.com/journal/49/60004/too-much-world-is-the-internet-dead/.

Venturini, Tommaso, and Munk, Anders Kristian. (2022). Controversy Mapping: A Field Guide. Polity Press: Cambridge, UK.