Mapping Affect: exploring new protocols for visual analysis

What can AI mean for visual analysis in social networks?

Team Members

Marloes Geboers, Victoria Andelsman, Lucia Bainotti, Matteo Banal, Evy Brans, Tommaso Campagna, Ketaki Chand, Lucie Chateau, Regina Dashkina, Jenny Fan, Sohini Goswami, Judith Schossboeck, Jing Shi

Contents

Summary of Key Findings

Because of Google Cloud Vision API's positivity bias, one has to be careful using object labels as signifiers of affect. While the visual content fed into the API were clearly not always showing positive events, the dominant affective labels were 'Fun', 'Happiness' and 'Smile'.

Hashtags seem to give us a better starting point to analyze visual affect in hashtag publics, providing more contextual information. From there, it might be useful to annotate categorized images within certain hashtag clusters, using the Cloud Vision API.

The over time element is important to include in the analysis as hashtag publics bond and disbond through emotions in a very ad hoc manner. To detect the formation and disbanding of publics based on the visual content they circulate and/or respond to, we need to organize and anlayze our data using separate time slots.

Differences and similarities in the platforms were detected using qualitative interpretation of the clusters of both object labels and images that cluster around such labels. When comparing #prayforsyria on both Twitter and Instagram we see recurring visual themes, of which depictions of children and text-based imagery are significant labels on both platforms, in terms of their label frequency and visual language, although the compositional modality of images of children and textbased images differs between platforms: children on Instagram are often set in a more 'aesthetic' way than on Twitter. Resonating child images on Twitter show more explicit details of war and more contextual information where Instagram images that resonate within the public, focus more on faces, showing less context. We must note though, that the images of faces on Instagram, much like children images on Twitter, depict sad emotions. Diverging themes in resonating visual clusters are the pope on Twitter and tourism/travel on Instagram. This might be related to the large follower base of the pope account on Twitter.

Although tourism is a significant label on Instagram, the context of war is often provided within the image through showing 'before and after bombings' photos or textual information. The API seems to not recognize the added layers of war information and proceeds to annotate such images with the much more positive 'tourism'. This also relates to the formerly mentioned positivity bias.

Based on a text mining analysis of the Twitter hashtag #prayforsyria we explored the query contexts of positive and negative words of affect for this network. This lead to the identification of context-specific timeframes in which solidarity networks were appearing or disappearing, for instance #hope in week 49/2018, confirming their rather ad-hoc and quick formation, particularly in relation to critical events, f.i. the bombings (which happened in week 48 starting with Monday 26th of November). Visualizing the word contexts of both positive and negative queries allowed for the identification and presentation of positively or negatively connotated affective publics (positive: children after the bombings, negative: idlib, assad, bombings) and their contextual appearance over time. Solidarity hashtags can thus be seen as reactive to already existing Twitter conversations or publics. Political happenings or critical events and the related solidarity networks are clearly detectable, for instance with quantitatively less word associations found in week 48/2018. Analyzing the specific word contexts of hashtags might thus provide further insight into the affective nature of solidarity networks and their development over time.

1. Introduction

The diffusion of networked images and their discursive signification opens up new possibilities for understanding affect. By studying how ad hoc hashtag publics (Bruns & Burgess, 2015) distribute, negotiate, or creatively re-appropriate images (e.g., memes), and the way these publics react to images through networked behaviors such as tagging, replying, liking, or retweeting, the dominant visual language within and across networked publics may be mapped. Through the analysis of such activities, we may reveal what affect means within particular discursive online (hashtag) spaces.

Thus, we locate affect both in the visual content of the tweet — focusing on the visual modality — as well as in the technicity of its content; the latter is, in this case, largely constituted by likes, retweets, and hashtags.To arrive at specific visual themes that resonate over time we included annotated object labels, generated by Google’s Cloud Vision API, in network analyses which are further discussed in the Methods section.

Alongside applying the Vision API, we also executed visual analyses using textual pointers, such as hashtags in co-occurrence networks that show both the tags and their images and words in Rank flows of negative and positive word associations. In the latter we did not get to sensibly connect such word associations with the images that go with these words.

Emotional engagement through sharing and liking affective visual content is extensively researched, often with a clear focus on the types of interaction that platforms allow its users to have, however, this project is part of a study that aims to go beyond mapping interactions of users by studying the visual and textual content of tweets and Instagram posts as stimuli for affective interactions and emotional responses of users. While user activity metrics are repurposed to sort and categorize visual content, we also wanted to trace or map the visual characteristics that might account for these responds.

In this endeavor, we need to advance visual methods within a framework of digital methods, which meant we had to 'test' new protocols that allow us to arrive at viable findings about visual affect. The project proved to be more a test case for the performance of the Cloud Vision API (Google) in comparison to textual protocols, which was insightful.

2. Initial Data Sets

We worked with

1) a subset (18-11-2018 to 18-12-2018) of Tweet data extracted through DMI-TCAT:
The query bin consists of the following tags and key words: Syria, #Syria, Sirya, #Sirya, #Syrianrefugees, #Eastghouta, #Idleb, #assadmustgo, #PrayforSyria, #IranOutofSyria, Khan Sheikhoun, #KhanSheikhoun, Khan Shaykhun and ['الغوطة الشرقية'] [#إدلب] [إدلب] [#الغوطة_الشرقية] ['اللاجئ السوريون'] [سوريا‎] [سوريا‎#] [#خان_شيخون] ['خان شيخون']

We queried most frequent solidarity hashtags for that subset which were #savesyrianchildren, #eyesonidlib and #prayforsyria.

We also worked with

2) Instagram scraped data using the DMI Instagram scraper, queried with the abovementioned most frequent twitter hashtags (for comparative reasons) and queried for more Instagram specific hashtags which are #prayforsyria, and #syrianchildren.

3. Research Questions

Clearly state your research questions and any hypotheses / expectations.

Overarching Q:
What might resonance and visual content reveal about affect in Syrian hashtag spaces on Instagram and Twitter?

Methodological questions:

• Can we repurpose the Google Vision Api for researching affect in hashtag publics?

• Or are we better off performing text/image analyses using hashtags or word associations?

4. Methodology

Figure 1: A general overview of the research protocol in flow chart style

Twitter object labels/images networks protocol:

We set off, establishing most frequent hashtags of 18-11 to 18-12 2018 in Twitter though using the hashtag frequency module in DMI-TCAT. Note that one has to rearrange dates in the output file to get to a AtoZ sorting of frequencies.

To get to the object labels networks, we needed to isolate image URLs within the frequent hashtags data. Starting off with demarcating the TCAT bin through querying the hashtags separately, we then used the media frequency module in DMI-TCAT to determine most tweeted and retweeted media.

Now we arrive at a list of URLs to feed to the Google Cloud Vision API. We used the Python version of the Memespector tool to apply the Vision API. See link to Git for instructions.

(Pro tip: just feed CSVs with merely one column containing URLs, add a second column with whatever content to the right of your URL column, for some reason one gets delimiter errors otherwise, even when configured the delimiter in the correct way, you're welcome).

For analysis we used the object labels output, however, the Vision API gives you other output too: such as web entities and face recognition scores. Working on images within a platformed context, web entities do not seem to give very meaningful output as this feature grabs text surrounding images in the space that they are in, for 'platformed images' this means it often grabs tags. When using URLs to images on the web, one would get more meaningful contextual information.

On top of the so called annotated input file (which is the ouput actually), the Cloud Vision API provides gexf files as output, too. These gexf files are opened in Gephi as a new workspace (undirected).
To connect retweet (or like in the case of Insta data) frequencies to the image nodes:

In overview mode of Gephi: run ForceAtlas2.

In the data laboratory, from the nodes table: export table. Import data in a new sheet that you add to the corresponding annotated input file that you got as output from the Vision API tool.

In this new sheet:
sort a/z ID column
sort a/z the column with g_labels and images (which is often the type column) so that images are put together
add a new column next to the type column and name it frequency.

Copy paste frequencies from the annotated input file next to the type column in the new sheet. To make sure image IDs and frequencies actually correspond: use the function VLOOKUP. Be sure to have frequencies to the RIGHT of your image IDs in the annotated input sheet where you make the range connecting to image id and frequency columns.

Export the 'VLOOKUPPED' spreadsheet as a csv. Import this in your data lab as comma/nodes table, ONLY keep the boxes IDs and frequencies checked, uncheck the rest!
Pro tip: import frequency as “integer” (not string!!!).

If you want to add label frequencies: go to open refine, let it run. Import your sheet and split your object labels from one cell into multi valued cells, to get the labels into rows while still connecting to the image.

Using the Gephi plugin Image Preview

Install the Gephi plugin called “Image Preview” (Tools/Plugins/Available Plugins/Image Preview)
Go in Data Laboratory and create a New Column, call it image.
Use the function “copy data to other column” to copy the file names from the column “file” to the column “image.
In preview, check “Node Images” section, specify the image path, and click refresh, images should appear on the nodes.

Tweaks of the protocol for working with Instagram data and the Memespector:

From the scraped Instagram .csv file (output): extract URLs and download the images with a browser plugin like: TabSave /DownThemAll.
Create a folder with your images and place it in the Memespector folder where your inputfile.csv is.
Change the Input section (memespector) in the config.txt with the path of the images folder. Example:Input:/Users/user/Desktop/memespector-python-master/memespector-outputs/img
Insert the image folder name in the section “InputImageFolder”
Run in terminal the call for your Memespector.

Making co-tag/image networks (Instagram)

In this part of the project we decided to focus on hashtags as a starting point for mapping visual affect in hashtag publics. In so doing, it is possible to frame the analysis of the visual discourses connected to affect within the grammar of hashtag, providing some textual and contextual information. In particular, we focused on the hashtag #syrianchildren, which is the most popular hashtag on Instagram during the time period considered (966 occurrences in the time stamp 18/11/2018 – 18/12/2018).

Once the dataset was selected, we used Table2net (http://tools.medialab.sciences-po.fr/table2net/) to get a Gephi-friendly hashtag co-occurrency network. Such a co-hashtag network allows for network and cluster analysis based on hashtag usage.

Secondly, in order to set a relationship between images and hashtags, we created a bi-partite image-hashtag network. It allows for network and cluster analysis based on hashtag and image usage together. Again, the network has been constructed with Table2net and then visualized with Gephi. To replace nodes with images in the graph, we used the Gephi plugin Image Preview (see above).

LineMiner protocol

Lineminer provides a search feature counting the number of lines a query or queries appear in. In this case we focused on the word context for a specific query (Lineminer shows the word context with 15 words, which can be analyzed f.i. with the RankFlow visualization tool. In an experimental attempt using emotionally relevant words related to positive or negative affect (affective or emotional processes), we queried 12 words each (positive query: happy joy love smile happiness fun adore like hope relief admire enjoy, negative query: hate kill worthless fear nervous afraid fear sad cry depressed grief anger). Showing the word context of both queries, we used the CSV file for further visual and qualitative analysis of the quantitative occurance of words we queried.

5. Findings

We networked object labels and images in a bi-partite way for all the abovementioned frequent or significant solidarity hashtags. After this we zoom in on specific ways of doing analysis.

Figure 2: collection of networks made for most frequent (Twitter) and significant (Instagram) solidarity tags.

Figure 3: Instagram network for PrayforSyria. Showing text/font, child and building/tourism as frequent labels and resonating (likes) images.

Figure 4: Twitter network for #PrayforSyria, showing similar and different (pope, religion) clusters than the Instagram network of this tag. Note how fun is quite significant. Although mainly positive, here one may find the label 'agression' as well.

Figure 5: showing similar and different most resonating images across both networks for #prayforsyria.

Figure 6: Hashtag usage for #syrianchildren on Instagram over time

Figure 7: co-tag network for #syrianchildren Instagram

Figure 8: Image clusters in co-tag network.

Figure 9 shows a comparison between on the left: a co tag network of #eyesonidlib (Twitter) and the same space but then using the Google Cloud Vision API object labels.

Figure 10 shows rank flows on the basis of LineMiner and RankFlow, of word assocations related to a positive query for #prayforsyria (Twitter)

Figure 11 shows rank flows made with LineMiner and RankFlow, of word assocations related to a negative query for #prayforsyria (Twitter)

6. Discussion

Limitations:

• Some images did not get labels: Google Vision API does not pick up on them. Did not have to do with their extensional properties. Seem to be a bug.

• Lack of contextual information via object labels. For analysis we used the object labels output, however, the Vision API gives you other output too: such as web entities and face recognition scores. Working on images within a platformed context, web entities do not seem to give very meaningful output as this feature grabs text surrounding images in the space that they are in, for 'platformed images' this means it often grabs tags. When using URLs to images on the web, one would get more meaningful contextual information.

• Model is not trained to detect effect. Even among these semantically similar networks, there are several clusters.

Further research:

Positivity bias of Vision API: Researchers should feed in more war/conflict context to better train the algorithm.

Hashtag analysis should be a starting point, detecting visual patterns in a more contextual manner and then run the Vision API, being aware of it’s bias.

A fruitful direction would be connecting the word content of the overtime rank flows from LineMiner to images that associate with positive and negative words (images over time that is) as hashtag publics are known to often have a strong ad hoc component.

Using Clarifai might be a good idea too: one can build their own datasets, feeding your own relevant labels into it. This way we would enhance labelling through training the machine.

Another idea is to isolating the 'affective' hashtags through taking resonating images as a starting point and seeing what affective/solidarity hashtags recurr throughout. In a second stage we would then use either Google or Clarifai to execute computational coding of our affective hashtag images.

7. Conclusions

The diffusion of networked images and their discursive signification opens up new possibilities for understanding affect. By studying how ad hoc hashtag publics (Bruns & Burgess, 2015) distribute, negotiate, or creatively re-appropriate images (e.g., memes), and the way these publics react to images through networked behaviors such as tagging, replying, liking, or retweeting, the dominant visual language within and across networked publics may be mapped. Through the analysis of such activities, we may reveal what affect means within particular discursive online (hashtag) spaces.

Because of Google Cloud Vision API's positivity bias, one has to be careful using object labels as signifiers of affect. While the visual content fed into the API were clearly not always showing positive events, the dominant affective labels were 'Fun', 'Happiness' and 'Smile'.

Hashtags seem to give us a better starting point to analyze visual affect in hashtag publics, providing more contextual information. From there, it might be useful to annotate categorized images within certain hashtag clusters, using the Cloud Vision API.

The over time element is important to include in the analysis as hashtag publics bond and disbond through emotions in a very ad hoc manner. To detect the formation and disbanding of publics based on the visual content they circulate and/or respond to, we need to organize and anlayze our data using separate time slots.

Differences and similarities in the platforms were detected using qualitative interpretation of the clusters of both object labels and images that cluster around such labels. When comparing #prayforsyria on both Twitter and Instagram we see recurring visual themes, of which depictions of children and text-based imagery are significant labels on both platforms, in terms of their label frequency and visual language, although the compositional modality of images of children and textbased images differs between platforms: children on Instagram are often set in a more 'aesthetic' way than on Twitter. Resonating child images on Twitter show more explicit details of war and more contextual information where Instagram images that resonate within the public, focus more on faces, showing less context. We must note though, that the images of faces on Instagram, much like children images on Twitter, depict sad emotions. Diverging themes in resonating visual clusters are the pope on Twitter and tourism/travel on Instagram. This might be related to the large follower base of the pope account on Twitter.

Although tourism is a significant label on Instagram, the context of war is often provided within the image through showing 'before and after bombings' photos or textual information. The API seems to not recognize the added layers of war information and proceeds to annotate such images with the much more positive 'tourism'. This also relates to the formerly mentioned positivity bias.

In terms of analyzing the word context of certain hashtag publics, it is important to note that hashtags are often used in concurrence with others, sometimes making it difficult to relate affect (or certain affects) directly to a specific hashtag. In our analysis, we found that Twitter was more useful for an analyses of word content than Instagram and we suspect that this might have to do with the stronger use of multiple hashtags on Instagram (or the fact that Instragram is a complex ecosystem of social media platforms, see Highfield & Leaver 2015), making it difficult to distill distinctive contexts. However, this could provide a good starting point for identifying related hashtags public, smaller solidarity networks, or a further semantic analysis, f.i. regarding the public perception, affect and general connotations of solidarity networks and their elements. Lineminer also proved useful for identifying certain timestamps related to the formation of hashtag publics, confirming their rather ad-hoc and quick formation, but also the changing sentiment around their elements. For instance, while the bombings appear two times in our weekly analysis of negative word context (last appearance in week 49/2018, other terms like assad or #idlib seem to appear in this query more continiously or persistently, while some appear in week 49/2018 only, see countryside). On a side note, while in semantic analysis it is sometimes not so useful to detach hashtags from the general terms, this was useful for our research interest, as it allows for the identification of certain (contextual) timeframes in which solidarity networks appear (or disappear), as well as the presentation of the positive or negative context of specific affective publics (f.i. assad, idlib, countryside).

8. References

Bruns, A., & Burgess, G. (2015). Twitter hashtags from ad hoc to calculated publics. In N. Rambukkana (Ed.), Hashtag publics: The power and politics of discursive networks (pp. 13-27). New York, NY: Peter Lang.

Bucher, T. (2012). A Technicity of Attention: How Software ‘Makes Sense’. Culture Machine 13. pp. 1-13.

Dennis, J. (2018). Beyond Slacktivism, Political participation on social media. Palgrave MacMillan

Highfield, T., Leaver, T. (2015). A methodology for mapping Instagram hashtags. First Monday, Volume 20, Number 1, 5 January 2015. Available at https://firstmonday.org/ojs/index.php/fm/article/view/5563/4195. Date accessed: 25 jan. 2019. DOI: http://dx.doi.org/10.5210/fm.v20i1.5563.

Papacharissi, Z. (2016). Affective publics and structures of storytelling: sentiment, events and mediality. Information, Communication & Society, 19(3), 307–324. https://doi.org/10.1080/1369118X.2015.1109697.

Papacharissi, Z. (2015). Affective publics: Sentiment, technology, and politics. Oxford, UK: Oxford University Press.

Papacharissi, Z., & de Fatima Oliveira, M. (2012). Affective news and networked publics: The rhythms of news storytelling on #egypt. Journal of Communication, 62(2), 266–282.

Rose, G. (2016). Visual methodologies: An introduction to researching with visual materials. (4th ed.) Thousand Oaks, CA: Sage.

Wahl-Jorgensen (2019). Emotions, Media and Politics. Cambridge, UK: Polity Press

You may find the presentation slides here:

SLIDES