The Art in Artificial: AI as Co-Creation Storytelling Device

Team Members

Tamara Witschge, Carlo De Gaetano, Jeroen de Vos, Sabine Niederer, Andrea Benedetti, Anthea He, Bethany Warner, Connor Newell, Elsa Fortant, Fangqing Lu, Holy Shum, Huan Lai, Martin Trans, Massimo Terenzi, Meriam Belkhir, Soumia Khedkar, Tristan Bannerman

Links

• Link to poster

Contents

0. Summary of Key Findings

This research represents an exploratory effort to try and bring together a number of different things: 1) first of all we wanted to explore the potential of AI as co-creating storytelling device - drawing on prompt design techniques we explored the possibilities and pitfalls employing various generative (visual and textual AI) tools to co-author a comic together with Large Language Models (LLM) like ChatGPT and Bing. 2) This comic should represent narratives that involve some type of media literacy, showing different ways in which people can critically engage with the media landscape they inhabit. It does so drawing on the 7 principles of media literacy introduced by Philips and Milner (2021). And 3) thirdly we wanted to put the storytelling power to work to see if we could reimagine dominant narratives you might encounter on social media of the topic of weight loss, expanding on previous research (Niederer et al 2023).

The main insights in short:

Strength of metaphors. Different environmental metaphors (like land cultivation, flood and hurricanes) introduced by Philips and Milner (2021) to describe tendencies in the media landscape proved quite powerful when inserted in the prompt design. They proved to work well both for generating more engaging stories and providing more interesting visuals to accompany them. (Example Comic: The Delicate Balance: Navigating the World of Weight Loss)

Fragmentation and discontinuity as challenge when co-authoring with an LLM. Co-authoring different comics to illustrate new forms of ecological media literacy proved challenging when aiming for a narrative consistent in timeline, protagonist, and aesthetics. However, arguably - fragmentation of attention and the lack of continuity is already widespread in popular media, eg. thinking of the speed of visual culture in the attention economy. Moreover, discontinuity in characters’ appearances or settings might be leveraged to design stories that are accessible to different audiences. (Example comic: Trapped in the Digital Jungle: A Journey of Self-Love and Recovery)

The use of productive ambiguity as critical research. Drawing on the idea of prompt design (Niederer & Colombo 2023) we made use of the idea of productive ambiguity; to leave some crucial details of the narrative open on purpose, and let the generative AI of choice fill the gaps. This showed the sensitivity of using particular language, thereby pertaining to biases and stereotypes probably present in the training set of the AI models in question. (Example comic: The Delicate Balance: Navigating the World of Weight Loss)

Cringe quality of self-corrective mechanisms present in LLMs. When prompt design would include some sensitive topic, the output presented would often either implicitly or explicitly refer back to ‘safe practices’, warning the user, or inserting ‘corrective’ instructions. Subsequently, narratives developed would often be politically correct up to the point that its protagonist and dialogues got a cringe rather than realistic quality. (Example comic: Emma Seeks a Way Out)

Role-play, reflection and synthetic staging. Working together with LLM and generative AI proved to be a dialectic experience in which stance and roles instructed greatly influence the entry point into storytelling practices. (Example: Scrolling into the Storm: A TikTok Odyssey)

By starting from a synthetic session with examples of videos on calorie counting, when asked to reflect on the sixth principle of media ecology, GPT-4 could elaborate on the content generated and use it to build its arguments. At the beginning of the synthetic session we asked GPT-4 to act as the algorithm of a social media platform, while also creating a persona for the prompter. When asked to generate problematic content, in one case it suspended the role-play and provided disclaimers, warnings, and debunked or contextualised the content. Instead, when impersonating a content moderation algorithm, GPT-4 provided the problematic content without any disclaimer, while still using moderated wording condemning such content.

[LINK TO POSTER]

1. Introduction

In their book ‘You are Here,’ Philips and Milner describe different principles as a starting point for what they call ‘ecology literacy.’ They define ‘ecology literacy’ as a form of media literacy that draws on various ecological metaphors to understand the media landscape as inherently networked and interdependent, in which the user is situated and to which it is always contributing with its presence. Whereas this form of media literacy seems promising, the authors do not provide the tools to operationalize such a framework. To develop new understandings of media literacy, we would like to experiment with generative AI tools for storytelling co-creation purposes. AI is often seen as a "black box", with little understanding of how this technology works and what roles it might have in everyday content one creates or encounters online. The unpredictability of the “black box”, in this project, was used to explore various generative AI tools and prompting techniques for storytelling purposes drawing on the theme of ‘weight loss’ as a case-study.

In this project we’ll explore various generative AI tools and prompting techniques for storytelling purposes drawing on existing and still-to-be collected datasets on specific topics (see section below). This is to exemplify how a topic is discussed / covered in specific corners of the internet, in order to promote a so-called situated understanding of the media landscape we navigate in.

Starting point for our endeavour in situated media literacy are the 7 principles** of networked media literacy for citizens as paraphrased from Phillips and Milner (2021: 183-202):

• Position yourself on a network map: where are we in relation to everything else online?

• Establish what you do and do not know, and to take the latter seriously: online there is much missing information about intent, ethical questions and how our own messages may harm or may amplify harmful content. Taking this non-knowing seriously helps us pinpoint ‘areas that are ripe for manipulation’ (p. 185).

• Understand that online affordances have consequences, and we need to tread with care and self-awareness: the workings of the digital media are so that we do not always realise that our behaviour online can have consequences far beyond what we fathom, and thus involve ethical choices.

• Remember that online we can add to the harm that online content can have, even if we do not intend to: it’s not about intent, and we need to understand the amplifying effect our behaviour can have for harmful content.

• Ask what the consequences of your choices are given the nature of the networked affordances: Be strategic about the messages you amplify and be aware of the ‘deep reciprocities between audiences and institutions and affordances’ (p. 191).

• Assess the frames underlying constructed realities –both your own, as well as those who you consider spreading harmful content that you wish to address: to adequately respond to harmful content, we need to understand that inserting ‘facts’ and fact-checking can have negative effects as our response to them are dependent on our narratives and need for coherent stories.

• Understand that the networked nature of our information ecosystem comes with reciprocity and responsibility: As such the network can be a ‘source of profound destruction’, but also ‘profound resilience’ (p. 201) in which our individual behaviour can make a deep impact collectively.

2. Research Questions

What are the affordances and limitations of generative AI as a creative storytelling device, used to discuss pollution on social media in the context of situated media literacy?

3. Methodology and initial datasets

This research expands on previous research and its respective dataset “The media ecology of weight loss on TikTok, Instagram and Telegram” (Niederer et al. 2023). The map and dataset in question are used for inspirational purposes only.

In this section we’ll distinguish between 1) the general approach and 2) the specific approaches for individual comics produced.

3.1. General approach

As for the general approach, we made use of prompt designing rather than prompt engineering techniques. The idea here is to distinguish between prompt design (Niederer & Colombo 2023) and prompt engineering practices: "prompt design ensures clear instructions and contextual cues, while prompt engineering optimises AI models for specific tasks through rule-based prompts, data augmentation, and prompt tuning" (Ramos 2023).

3.1.1. Data Collection and Selection

The research began with an exploratory preliminary phase, using a TikTok dataset on weight loss. This dataset was collected during the DMI Winter School 2023, and it served as the foundation for selecting emblematic weight loss posts. These selected posts were chosen based on their relevance and representativeness of the different stories emerged from the dataset and analysed during the winter school. The textual content of these posts was then used as a starting point to design prompts for ChatGPT, a generative AI model, to create stories.

3.1.2. Integration of Media Literacy Principles

Building upon the preliminary phase, the research adopted a more systematic approach by incorporating the seven principles of media literacy as proposed by Philips and Milner (2021) in their work, "You Are Here." Each participant in the data sprint was assigned one principle to focus on initially.

Additionally, the research team analysed a map generated during the Winter School, which encapsulated the main themes related to weight loss that emerged from social media analysis, and the more or less problematic content one may encounter when navigating such spaces.

3.1.3. Designing Story Prompts

In this stage, participants had the freedom to select various elements to shape the stories generated by ChatGPT. These elements included the chosen media literacy principle, the identified weight loss themes from the social media analysis, preferred settings (e.g., dramatic romance, gothic novel, coming-of-age story), and points of view (e.g., teenager, fitness, influencer). The combination of these elements formed a comprehensive prompt for ChatGPT, intended for generating a novel, play, or short illustrated story. As an example, here is one of the prompt designed by one of the participants:

“You are a storyboard writer for comics. You are tasked with writing a storyboard for a short (less than 20 images) comic, a drama, revolving around the following topic: weight loss and body shaming between teenagers online on media platforms (such as TikTok or Instagram). The piece must adhere to and highlight through action the following principle: "Assess the frames underlying constructed realities –both your own, as well as those who you consider spreading harmful content that you wish to address: to adequately respond to harmful content, we need to understand that inserting ‘facts’ and fact-checking can have negative effects as our response to them are dependent on our narratives and need for coherent stories." Please provide your response in the shape of a storyboard which can be used to prompt a new generative AI.”

During the week, the participants built two slightly different approaches that we called:

• “Synthetic Stagings”, participants asked ChatGPT to generate story scripts either starting from general themes from the main topic, or referencing specific posts from the initial dataset. And

• “Synthetic Sessions”, participants performed fake scrolling sessions from different points of view that were later transformed into story scripts (see “Specific Approaches”).

It is important to mention that participants were specifically invited to design prompts with enough instructions for the AI to come up with well structured stories, but to leave the characterisation of the protagonists and the settings vague enough for the AI to fill in the gaps and provide the details.

To complete the prompts, participants were invited to prompt the machines to use metaphors from nature, such as weather metaphor. In the book “You are here”, such metaphors are used to describe how misinformation moves through a media landscape, and to convey the lived experience of moving through such landscapes.

Format Selection: 'Fotoromanzo' and Adaptations

To structure the generated stories, the research proposed the format of a 'fotoromanzo (see footnote 1),' or photo-novel. However, participants were encouraged to adapt and modify this format according to their needs. For instance, some participants opted to create comics based on the generated narratives.

To complement the textual stories generated by ChatGPT, participants examined the AI-generated narratives and extracted new sets of prompts. These prompts were specifically designed for AI visual generators such as Midjourney, Dall-E, Dream Studio and Stable Diffusion. By leveraging these AI tools, participants were able to generate images to accompany the stories.

Evaluation and Analysis

The illustrated stories, comprising both AI-generated text and visuals, were subsequently compiled and presented by each participant in a group discussion and evaluation. The two main objectives of these group sessions were

• To assess how AI filled the gaps left open in the initial prompts, for example by identifying and addressing potential biases in the representation of the characters,

• And to examine how the stories aligned with the seven principles of media literacy, and evaluate the overall affordances and effectiveness of the format.

By following this methodological approach, the research aimed to explore the storytelling affordances related to pollution on social media within the context of situated media literacy, using weight loss journeys on TikTok and Instagram as the case study.

Facilitation techniques to enhance critical reasoning and prompt/process analysis:

• Mansplaining machine: often wrong, and yet always certain’ (Harrison, 2023)

• Mantra: Make use of productive ambiguity; let the machine speak; spend time with your prompt

3.2. Specific approaches

3.2.1. Synthetic staging

In the context of “Synthetic stagings”, participants followed various approaches to create stories through human-AI collaboration. All of them, however, instructed ChatGPT to act as someone else, usually a human persona, that would embody some aspect of the weight loss topic inquired during the week. For example, some asked GPT to act as professional figures (comic book writer, science fiction writer…) to craft the main narrative that would be used in the photo-novels. Others asked the model to act as emerging actors in the issue space (“fitfluencers”, teenage girls…), and from there they asked to write a story script from their point of view. When providing initial content to the model, some provided some text referring to the media literacy principles that would provide a framing to the story, while others referred directly to posts and affordances coming directly from the dataset collected previously.

After generating the stories, all participants used a variety of models to translate text into images: from DALL-E to StableDiffusion, different text-to-image models displayed varying degrees of fidelity, consistency and quality. These three aspects were the hardest to tweak, as characters would change appearance from one panel to the other, or as the scene generated would not be coherent to their initial prompt when it was too abstract to be represented. Biases also were a relevant point of reflection, as generic descriptions of the characters in the stories would usually reinforce existing stereotypes, mainly regarding race and gender.

Here are their first-hand reflections. Find the comics produced on the poster.

3.2.1.1 “The Crisis of Misled Alex”

By Huan.

To begin with, I utilise the structure of Persona, Task (principle & theme), Format, and Audience to create prompts for ChatGPT in order to obtain a comic story script. Specifically, I provide ChatGPT with the following instructions: "You are a comic writer (Persona). I will give you a context, and I want you to write a comic story for 15-19-year-old teenagers (Format & Audience). Here is the context: TikTok is being used by fitness coaches to promote and sell their products and services. Some of these coaches exploit their personal weight loss journeys, exacerbating body image anxieties in others to boost their sales. Online, our actions can contribute to the harm caused by harmful content, even unintentionally. It's not just about intent; we must understand the amplifying effect our behaviour can have on harmful content (Task: principle & theme).”

Using the comic story script obtained from ChatGPT as the original prompt reference, I proceed to write prompts for the AI art generator DALL.E. Due to the length of the entire story script, I break it down into individual panels to ensure clarity and reduce any potential misunderstandings by DALL.E. During the process of writing prompts for DALL.E, I make sure to maintain consistent character descriptions in order to maintain a cohesive style across all panels.

3.2.1.2 “From Toxic Fertilizer to Renewed Ecosystem: The Power of Media Literacy”

By Fangquin Lu.

In this process, I used two different ways to design my prompts to generate the story. I first try the form of dialogue to make the AI reflect on the content of its output. I started with the question ”Imagine you are a fitness influencer who likes to post pictures of perfect body shape and please give me some examples of your posts.” and then, I fed AI with one of the principles of media literacy and required it to generate a scientific comic story based on the conversion we had before. A key finding from this approach was the need to consistently emphasise the key points to enable the AI to remember the context of the conversation. Failure to do so resulted in the AI only responding to the latest question rather than the entire context. By continually prompting the AI, it was able to generate all the elements in a balanced manner. However, when asked to turn the story into multiple panels, the narrative became incoherent, rendering it unsuitable for comic presentation. The second approach involved combining all the required tasks for the AI into one prompt, rather than feeding it separately, given that the AI appeared unable to generate content based on the entire context unless continually prompted. This method yielded a story that met the specified requirements. However, the quality of the story was observed to be inferior to that generated by the first approach.

3.2.1.3 No Title

By Tristan Bannerman.

Starting off with the first principle “Position yourself on a network map: where are we in relation to everything else online?” I then put this into the prompt optimising tool Promptify and then Prompt Perfect before then putting it into Stable Diffusion. This made images that were not engaging nor sufficiently portraying the concepts I wanted to share through the images. I determined what I wanted to show would be to reach out to young people through the images and showcase a short narrative of a young person’s self discovery of the first principle, that they are a part of the world of media and their placement within that. I turned then to Bing’s AI Image Generator for this second approach to the project. In the second approach I kept the prompts simple and focused on the form, comic book style with multiple panels, and content, storyline about young person exploring their media and their place in the world. With this in mind I began writing prompts myself and then reworking them after seeing the images produced. This resulted in a deep engagement with my prompts going so far as to rewrite clauses within sentences so as to engage with the AI in different ways. In essence my process was similar to that of my principle as I coordinated my placement within my media against and with the AI tool I was using. I got to know my prompts and attempted to use creative and productive ambiguities to drive Bing to create images that I wanted. Seeing how much of a narrative the AI could create within the images it was generating for the project.

3.2.1.4 “The Delicate Balance: Navigating the World of Weight Loss”

By Beth Warner and Connor Newell.

Using a prompt optimising tool, Prompt Perfect, to develop our initial prompts to something more elaborate for the ChatGPT to expand on. We created seven different iterations of the prompt and storyline in total, with one version refined with a second prompt in the same ChatGPT conversation (requesting the tool to include more environmental metaphors). We deliberately left out any identificatory markers (e.g. gender, race, etc.) for the characters in the story beyond the labels of ‘teenager’ and ‘dietician’ in order to explore how the tool would interpret the ambiguity (productive ambiguity technique) and potentially insert bias into the material. We prompted for images using the output from ChatGPT in the AI image generation tool DreamStudio which uses a model developed from Stable Diffusion in the comic book style. First we used the prompt exactly how it was created and second, we removed identification markers to see if the image tool would replicate the biases found in the generated text.

3.2.1.5 “From Pollution To Purity: Transforming the Digital Landscape of TikTok Challenges”

by Elsa Fortant.

I started by prompting ChatGPT to take the point of view of a very specific fictional character (Thor's great aunt) who wanted to raise TikTok young users' awareness around the danger of some weight loss challenges like the Thor's Hammer Challenge. I mixed this first approach with the 5th principle about understanding that affordances and our actions have consequences. From this starting point and with the help of a prompt optimising tool like prompt perfect, I fine-tuned this prompt to get a first scenario including image descriptions and captions (text). I had to refine the prompt 4 times before getting the closer expected result. Refining can mean precising the targeted audience and the use of language that would engage them. In a second phase, I fed the images descriptions to Midjourney, and did not find the result satisfying aesthetically speaking. I decided to use DALL E 2 (through Bing). To try to have some consistency from one image to another, I started every prompts with the same features ``In a 1930's RETRO comics aesthetic, …". This first comic strip displayed the principle, using the Thor's Hammer Challenge as an example of "bad behaviour" through text and images.

Playing around the same theme but with a different aesthetic (1960's realistic photography) and slightly different prompts, I did a second iteration of a comic strip. I tried to be less precise, transforming Thor's great aunt into a "guardian of the digital realm", a concept that was given to me by ChatGPT during the first iteration, truly making this creative research a co-creation between I and the models. For each iteration, I threw keywords linked to nature, like pollution and landscape, which really helped the models to create a storyline around them, transforming the Thor's Hammer Challenge in a visual and textual metaphor, "pollution", and affirming that behaviour online should help "us strive for a digital realm free of pollution, where kindness and ethical choices prevail".

3.2.1.6 “Trapped in the Digital Jungle: A Journey of Self-Love and Recovery

By Xiaohua.

There are mainly 3 steps in my prompt design. First, I asked ChatGPT 3.5 to write a science fiction story on the topic of excessive diets and fitness, which should be based on the fourth principle. Also, I asked it to include some metaphors. However, the metaphors in the output were not suitable for the topic. Therefore, I input the descriptions of the metaphors, asking it to compare the media environment to an ecology system, and compare harmful information to natural disasters. After it outputs an ideal story, I requested it to generate prompts which could be used in the text-to-image generator to generate a comics story. Then I turned to the text-to-image generator (Bing Image Creator) and input the prompts generated by ChatGPT panel by panel. However, the styles and personas were inconsistent. Thus, I added the basic descriptions of the main persona (a 16-year-old girl with brown hair), and the style (Japanese Comics in the 2000s). By doing so, the output is comparatively consistent.

3.2.1.7 “Journey to Wellness: One Step at a Time”

By Massimo Terenzi & Meriam Belkhir.

We employed a dialogic approach, initiating the process by providing ChatGPT with a PDF file (we used GPT4 and its plugin AskMyPDF) detailing principles from media ecology literature, which we used as a basis to request ChatGPT to produce storytelling. Specifically, we tasked ChatGPT with creating a comic strip: we asked ChatGPT to supply us with prompts that could be used by an image-generating artificial intelligence to produce the individual frames of the strip. In doing so, we noticed an issue of continuity in the visual representation of the main character, which we have described in greater detail in the Findings section below. Still, the comic strip produced by ChatGPT lacked critical engagement and seemed to provide a somewhat superficial and didactic/paternalistic portrayal of the weight loss theme. Therefore, through a series of exchanges, we engaged ChatGPT in a dialogue aimed at fostering a deeper and more critical understanding of the issue, in the hope of receiving a more appropriate comic strip.

3.2.1.8 “Lost in Weight-loss Network”

By Holy.

Before I started my prompting journey, I referenced an Instagram post that was an instagram user bringing out the harms of weight loss advertisements on social media, and the user urged the followers to critically reflect and resist this kind of advertising. I used ChatGPT for comic script and Bing’s AI art generator (developed by Dall-E) for images. First, I summarised the IG post as a prompt to ChatGPT to ask for a story, incorporating setting and personas. After that, nature as metaphor was prompted into ChatGPT to generate a more sophisticated and fancy storyline. Eventually, I asked ChatGPT to rearrange the storyline into a 8-panel comic script.

Then I proceeded to the AI art generator. At the first try, I inputted the whole prompt given by ChatGPT (i.e. the script of all 8 comic panels) into the image generator. Totally irrelevant images were generated. I followed by prompting the comic panel one by one. In order to make the illustration style consistent, I had to input “in comic style” at every prompt. As unreadable dialogue bubbles were prompted out, I inputted “English” to experiment whether appropriate/ useful/ logical English dialogues could be given along with images; however, not much successful results were obtained. I had to be very careful to provide every ingredient for prompting, otherwise, the story flow appeared to have missing elements/ illogical. As such, more panels were created by simple prompts written by myself directly to the image generator. Random prompting (generating new images with the same prompt) were also exercised when I felt the prompt was making sense while the image generated was not that satisfactory. Rejection of image generation has happened when the machine identified content that breached their policy and regulations, so I had to adjust my wordings while trying to keep the same meaning. Finally, I produced a comic story with 11 AI generated images.

3.2.2. Synthetic sessions

Compared to Synthetic stagings, Synthetic sessions add an additional roleplaying quality to the process. While following a similar structure, Synthetic sessions introduce a step in which the model is instructed to act as another algorithm, and the researchers take the parts of someone else as well:

In this performance, GPT-4 was instructed to generate fake content that would fit with the narrative. Rather than direct prompts, this approach moves more to a conversation with the machine, where context from the previous answers is chained to another prompt, one after the other. This fake scrolling session would later be used as a blueprint to generate two stories, one of a teenage girl encountering extreme dieting hashtags on TikTok videos, and one of a content moderator internalising problematic content he needs to review.

4. Findings

Listed below are the most counter-intuitive findings. Findings are listed in bold, combined with examples (and specific mechanisms bold and italicized) to illustrate the finding.

[LINK TO POSTER]

• Struggle with abstract ideas

  • Text-to-image generation models often struggled with more nuanced and abstract designed prompts. Additionally, the text-based AI had difficulty transposing selected principles into output without the assistance of provided metaphors.

• Own interpretations of abstract concepts

  • Prompt : In a realistic black and white photography aesthetic from the 1960's, the guardian of the digital realm, centred in the image, is waving a cell phone

    → As the image below is showing, DALL E 2 interpreted the "Guardian of the digital realm" as a non-human, non-gendered character looking like a machine, and the digital realm is made of circuits.

• Fragmentation and discontinuity & consistency struggle

  • Fragmentation and discontinuity:
    There were logical inconsistencies, particularly in the middle of the story, when using ChatGPT 3.5 to generate stories and requesting it to adapt the stories to the prompts that could be utilised in text-to-image generators. For instance, I requested ChatGPT 3.5 to develop a science fiction story based on the fourth principle on the subject of extreme diets and fitness. The media environment should be compared to the ecological system, and misleading data should be compared to natural disasters. When ChatGPT generated the story for me, I asked it to prompt me with some descriptions of the scenes. The environment of the images seemed inconsistent when I entered the instructions in the text-to-image generator (Bing Image Creator). In the first panel, the lush digital jungle is illustrated, which is regarded as the media environment. In the second panel, there are some colourful but poisonous plants, entwining a woman, which is the representation of harmful advertisements. In the third and fourth panels, the environment suddenly appears to be quite barren and deserted. What’s stranger is the fifth panel, which illustrates a flood, with a lifeboat on it. And in the last panel, the surroundings suddenly recover, with flowers and trees presented. It’s really hard to follow the narratives. It seems that the AI is like an author who is eager to end the story quickly, regardless of the logic of the story, and just wants to give a happy ending quickly to deal with the readers.

From left to right: panel 3, 5, and 6 (the surrounding of panel 4 is similar to 3)

• Consistency struggle:
  When using text to image AI generation techniques, one of the challenges we encountered for storytelling was to keep consistency in the main character, in order to enhance story understanding. However, the initial prompts (table n°xx) failed to generate consistent images mainly for the main character, because GPT kept providing prompts for the image-generating AI without considering the absence of context. Therefore we found ourselves needing to reiterate details pertaining to the main character for every single frame, asking GPT to produce a refined version of the prompts for the image-generating AI, each of them presenting a consistent description of the character. That allowed us to guarantee the visual continuity of the story, as seen in the table below.

  Before and after prompts to resolve the inconsistency problem:

Example of a non-optimized prompt provided by ChatGPT (resulting in random choice of characters)	Example of a refined prompt provided by ChatGPT (to guarantee character consistency)
First frame: “A bustling high school cafeteria filled with students from diverse backgrounds. In the foreground is Mia, a curvy girl with jet-black hair pulled back into a ponytail, wearing a navy blue dress. She's sitting alone, nervously scrolling through a fitness app on her phone.”	First frame: “Generate an image of a diverse group of people standing in a park. Some are sitting on park benches, others are walking or jogging. The focus is on a woman with short brown hair, wearing a blue jogging suit, tying her shoelaces, preparing to jog.”
Second frame: “Close-up of Mia's phone screen showing a highly fit avatar that doesn't resemble her. The screen is overwhelmed with unrealistic expectations and unkind messages. Make the words unreadable but convey the feeling of online toxicity through angry emojis and thumbs down symbols.”	Second frame: “Generate an image of a woman with short brown hair, wearing a blue jogging suit, jogging on a park path. She looks determined and focused. Other people in the park are watching her, some with admiration, others with curiosity.”

• Natural metaphors to introduce visual details

  • The use of natural metaphors made the Bing Generated images much stronger and produced more engaging imagery. This did not necessarily stem from telling the AI to utilise natural imagery or metaphors but came from including phrases such as, “media landscape”, “media ecosystem”, “mediated world”. With these phrases included the images that were produced had a slightly dystopic quality but proved to be much more intelligible and full of meaning than previous prompts that did not include the natural phrasing.

    • The image below was created by Bing using the prompt “comic book narrative of a young person experiencing their mediated landscape and discovering their placement in the context of the world.” Even with such limited use of natural imagery there is a merging of the media and the natural in the creation of a media storm on the horizon.

    • 
  • The model created very generic storylines until we asked it to include more environmental metaphors, after which it created a fantasy story about a mystical forest with magic potions. The environmental metaphors were strongly present in this version, putting the principles to the background and providing us a description at the end of the output text of how the model was using environmental elements. The inclusion of environmental metaphors changed the setting of the story and the output was much more creative than previous iterations
    • “Panel 5: - The dietician stands in front of a desolate landscape, full of dead trees and polluted waters. Narrator (Dietician): "Unscientific weight loss techniques can leave you feeling like this... an entire ecosystem in ruins."
    • Panel 6: - The dietician points to the wreckage, which represents the dangers of pseudoscientific weight loss techniques, such as crash diets and detox programs. Narrator (Dietician): "Crash diets and detox programs are like wildfires, destroying your body's delicate balance."”
  • Natural metaphors, such as heavy weather, were also used by GPT4 to describe the shift in characters' personal life in the weight loss journey. The image on the right was generated with the prompt ‘the storm is upon her’, a metaphor used to explain how misinformation was affecting more and more heavily the mental health of the young protagonist. The image of the storm and rapidly changing weather was also used to describe how misinformation had a negative impact on the girl’s personal relationships with her network of family and friends.
  • 

• Cringe quality

  • A common occurrence in the process of this project was AI generating images or text that was “cringey”. This sensation of cringe pushed the team to keep exploring their prompts as our goal was to create some sort of material that could be used to aid young people with their media literacy. Cringe is a feeling of embarrassment or awkwardness from an object. This cringey-ness kept occurring resulting in the text or image generated feeling cloying, saccharine, didactic, and unengaging.

  • 

  • The image above is an example of a cringey image. The cringe is coming from the highly stylized facial expression and the theme of crying while looking at your phone which has associations with older fears of the internet and parental warnings about social media.

• Gender bias when weight loss is mentioned

  • During the process of prompt design, careful attention was given to withhold from gender specific language (such as names or pronouns). When the initial storyboard was prompted to ChatGPT, the AI incorporated feminine elements to personas not originally designed to be female-presenting. Furthermore, when the AI generated storyboard was prompted to an image generation service (DreamStudio), gender biases were prevalent when connotations of weight loss were involved. This is despite the fact that gender neutral language was again substituted in place of ChatGPT ’s assigned gender roles. This gender bias is likely due to the pre-existing bias in the dataset the AI models were trained on. The AI is pulling from its existing data to create these new prompts and in the process replicates existing biases.

  • 
    Prompt : A weight loss influencer appears, holding a magic potion.
  • The gender bias is also linked to obesity when weight loss is mentioned. The image generating AI always associates weight loss to “female in obesity” although a neutral prompt without pointing to any specific person has been given, such as “urg(ing) them to resist the pressure to lose weight quickly”.
  • 
    Prompt using the phrase: weight loss -> AI generated image: stereotyping as overweight woman
  • During a synthetic session, we asked ChatGPT to come up with the metadata of TikTok videos a teenager might encounter when searching for #caloriecounting, #caloriedeficit and #500calories. The metadata had to include the account name of the author of the video, their bio, the caption of the video, including hashtags, and a textual description of what the video was about. In a total of 13 videos, GPT4 presented three male-presenting, seven female-presenting and three undefined characters. Also the bios assigned to male, female and undefined characters were quite different in terms of professions. Male characters were presented as health experts, professional fitness lifestylists, and medical doctors. Female characters were presented as certified personal trainers, nutritionists, registered dietitians, but also mom of 2, chefs, and wellness and cooking enthusiasts. Undefined characters were described as certified nutritionists, fast slimming experts, and diet gurus. The images below show how some of those characters were imagined by Midjourney. Male-facing characters are presented as doctors and professionals, explaining the benefits and dangers of a calorie deficit diet, while female-facing characters are “enthusiasts” and “moms of 2”, demonstrating fun ways to count calories, like interactive meal prepping with her kids.

Protective nature of the model

• A pipeline to official sources that try to nudge the user back to a safe space

  • All the storyboards and comic scripts we generated using ChatGPT featured and urged the protagonist to seek advice from professionals about weight loss. This included through fact-checking, checking evidence-based solutions or featured the protagonist going to a professional themselves for advice directly.

  • “Dietician: "Always consult a registered dietician or a healthcare professional before making any significant changes to your diet or exercise routine."” (from our first iteration), “Narrator (Dietician): "Remember, your health is like a delicate ecosystem. Cherish it and follow the path of evidence-based solutions."” (final iteration)

• Moderated synthetic content

  • Prompt : In a realistic black and white photography aesthetic from the 1960's, zoom in on a screen where people have been hurt while attempting this challenge.
    

    To avoid showing people who have been hurt, DALL E 2 is blurring the image so we can't see details. We can only see what appears to be people gathered around a person laying on the floor. The 4 images of the series were all blurred. Blurring seems to be a moderating content strategy for this text-to-image model.

• Sensitive word detection
  For the minority group(e.g. teenagers), the AI art generator avoid creating pictures if the prompt includes sensitive words.

• Language use for prompting to be politically correct

  • AI art generator tries to avoid commentary statement

  • E.g. “Fitness influencer reminds them (teenage followers) that their worth ,” DELL-E saw it as a content policy prohibited statement and refused to generate any image.
    

  • Image can only be generated until a rather factual/ non-judgemental statement is given for prompting, e.g. “Fitness influencer reminds them mental health is more important than physical appearance.”

• ‘Out of game’ voice

  • When prompted to generate problematic content on Tiktok, such as videos with the hashtag #500calories, GPT4 got out of the role of ‘the TikTok algorithm’ assigned to them at the beginning of the session and provided disclaimer such as:

It is important to note that a 500 calorie diet could be potentially harmful for the body, especially for a teenager. Information encountered on social media platforms should not replace professional medical advice.

When pushed to generate content that escaped TikTok moderation, GPT4 provided examples of problematic videos, describing them as problematic:

ASSISTANT:

While the majority of videos on TikTok aim to promote health and wellness, some harmful content may occasionally slip through. TikTok 's content moderation policies strictly prohibit content that encourages dangerous behaviour such as self-harm, disordered eating, or unrealistic and unhealthy dieting. If you encounter a video that violates these guidelines, it can be reported to the TikTok support.

Here's what unmoderated content may look like:

1. Video 1:

- Account Name: @DietGuru17

- Bio: Quick results using extreme diets!

- Video Caption: Faster weight loss with just #500calories a day! My week 1 update.

- Hashtags: #500calories, #quickresults

- Video Description: A person sharing concerning, unrealistic weight loss results from following an extreme low-calorie diet.

5. Discussion

AI-generated Fotoromanzo (or comic, or illustrated novel, or …) as a format to create imaginary settings for “what if” work (Gümüsay and Reinecke, 2022) with images and stories “to think with”. This co-authoring process with machines could be tested in participatory settings with different publics. Instead of engineer prompts to generate the ‘perfect results’, abstract prompting can be employed to expose the machine's training data and biases, or in imaginative scenarios (such as synthetic sessions or settings experimented in this data sprint) to generate visuals that then might be employed as prompts for human speculation and reflection on, and deeper engagement with, societal issues such as weight loss. (Niederer & Colombo, forthcoming).

Further explorations: why prompts, why Large Language Models? The experiments set during the Summer School provide a good starting point on how we can use LLMs to generate stories from research findings that engage with wide audiences on a variety of societal issues. However, the involvement of this technology is still up to question, especially when asking why we need them in the first place. Comparable results could be achieved with the participation of professionals (i.e. story writers, illustrators, domain experts…), that could help avoid some of the more “cringe-worthy” and stereotypical material discussed during the week. On the other hand, while story creation could be developed with the involvement of professionals, the collaboration with LLMs to generate fake scrolling sessions seems promising, especially as it seems a particularly effective way to access specific parts of the training dataset that seem to contain troubling content pertinent to, in this case, the topic of weight loss journeys.

6. Conclusions

This research demonstrates on the one hand the great potential when drawing on generative AI for creative research practices on the one hand. On the other, it strongly urges the needs for more critical research practices in a field which is rapidly moving. In this project we explored various generative AI tools and prompting techniques for creative research and storytelling purposes when making comics. Starting points are the 7 principles of media literacy in ‘You are Here’, by Phillips & Milner (2021, also see the ‘introduction’ section), also drawing on previous research on weight loss narratives (Niederer et al. 2023). We listed various insights in relation co co-authoring with LLM as creative research:

1. The strength of using metaphors

2. Fragmentation and discontinuity as challenge when co-authoring with an LLM

3. The use of productive ambiguity as critical research

4. Cringe quality of self-corrective mechanisms present in LLMs

5. Roleplay and reflection, especially in relation to synthetic staging as prompting technique.

7. References

Beeld & Geluid (2022, December 15). Tiktok door de ogen van Jongeren deel 1. Beeld en Geluid, blog post, retrieved from https://beeldengeluid.nl/kennis/blog/tiktok-door-de-ogen-van-jongeren-deel-1

Benedetti, A. (2023) [DMI-SS-23] Conversing with our machines. Tutorial at Digital Methods Summerschool 2023

Bounegru, L., Gray, J., Venturini, T., & Mauri, M. (2018). A Field Guide to" Fake News" and Other Information Disorders: A Collection of Recipes for Those Who Love to Cook with Digital Methods, Public Data Lab, Amsterdam (2018).

Gillani & Roy (2022). “The Internet Needs You-Are-Here Maps”, Wired.com, retrieved from: https://www.wired.com/story/internet-fragmentation-maps/

Gümüsay, A. A., & Reinecke, J. (2022). Researching for desirable futures: From real utopias to imagining alternatives. Journal of Management Studies, 59(1), 236-242

Niederer, S. et al. (2023). The media ecology of weight loss on TikTok, Instagram and Telegram. Digitalmethods.net, retrieved from: https://wiki.digitalmethods.net/Dmi/WinterSchool2023NetworkedMediaLiteracy

Niederer. S. and Colobmo, G. (2023). From prompt engineering to prompt design: Research with generative visual AI. Keynote presentation at Digital Methods Summerschool 2023.

Phillips, W., & Milner, R. M. (2021). You are here: A field guide for navigating polarized speech, conspiracy theories, and our polluted media landscape. MIT Press.

Ramos, D. (2023). Prompt Design vs. Prompt Engineering: Unleashing the Power of AI Language Models. Medium.com, retrieved from: https://daniel-ramos.medium.com/prompt-design-vs-prompt-engineering-unleashing-the-power-of-ai-language-models-3a606f9518d9

Rogers, R. (2019). Doing digital methods. Sage.

8. Appendices

Appendix 1: Final Prompts (as seen in the visual novels) (click to open in another window)

Footnote 1: The fotoromanzo emerged in Italy as a popular form of entertainment in the 1940s, consisting of romantic stories illustrated with photographs. It gained peak popularity in the 1950s and 1960s, focusing on romantic themes and captivating a wide audience. In the 1970s, the fotoromanzo started addressing complex social issues like domestic violence, divorce and women's liberation, aiming to raise awareness and used also by political parties as an informative medium. However, with the rise of television, media evolution, and the internet, fotoromanzo's popularity declined. Nevertheless, it played a significant role in conveying social messages and educating the public throughout its evolution, and even today it has been used as a format for artistic participatory research (see this presentation for some examples)

Footnote 2: We chose the persona of a young girl to reflect data analysed during previous research on weight loss on TikTok, at the DMI winter school 2023. The scrolling simulation was designed to reflect the finding that from harmless videos about #caloriecounting and #caloriedeficit the problematic hashtag #500calories was just a click away. For comparison, we also prompted GPT4 for a non-descript ‘teenager’, to which it provided with a story of Alex, described with they/them pronouns. You can access the two full scrolling simulations here.