-- FrederikkeChristiansen - 15 Jan 2021

A Truth Ecology: (Mis)Information Sources on the Deep Vernacular Web

Team Members

Yentel Boot, Frederikke Christiansen, David Dijkhuis, Morris Nieuwenhuis, Alejandra O’Connor

Facilitator: Anthony Burton

Summary of Key Findings

• In the runup towards the 2020 United States Presidential Election, we found news on both Reddit and 4chan have become more partisan and contain a higher quantity of misinformative sources.
• Whereas political 4chan has moved increasingly towards the right, political reddit moved towards the left compared to the first half of the campaign season (Burton & Koehorst, 2020).
• There was a significant shift in the role of YouTube as a news source, as the influence of traditional alternative influences declined rapidly.
• It will become more difficult to counter the effects of misinformation on the conception of truth among members of subcultural communities.

Introduction

The 2020 Presidential Election was going to be chaotic. After the shocking amounts of disinformation that tainted the 2016 election, pundits and scholars predicted that interference and deception would be widespread on social media (Pearce, 2019). In 2020, a similar study was conducted by Burton and Koehorst, to determine the status of misinformation on the Deep Vernacular Web, focusing primarily on Reddit and 4chan. As the election drew closer, its aftermath took a turbulent turn, and the Deep Vernacular Web is increasingly being cited as a source for the conspiracy theories that led to the storming of the United States (U.S.) Capitol (Talley & Levy, 2021; Heilweil & Ghaffary, 2021). Thus, it is critical that we re-evaluate the status of misinformation on these areas of the Internet.

The platforms initially used in the 2020 research will remain the focus of our investigation. Both Reddit and 4chan are renowned for their oppositional subcultures and perpetuation of political misinformation and can serve as breeding grounds for extremism - as became clear in the aforementioned Capitol storming (Burton & Koehorst, 2020; Tuters, 2019; Zannetou et al., 2017). How partisan is the news consumption and news presence in these subcultural spaces? Who holds the authority in terms of what kind of content is being distributed and that the people in those communities are being exposed to? Can the sources cited be an explanation for the hyper partisan points of view towards major events like COVID-19 and election integrity? This research aims to provide a deeper understanding into these matters, to better understand the situation of a post-Trump information ecology, and broadly speaking, the presence, or lack thereof, of a universal approach to the truth.

Initial Data Sets

To determine an increase or decrease in misinformation, we extracted the political sphere on the Deep Vernacular Web, focusing solely on Reddit and 4chan. To gain a more updated understanding of this shift in misinformation, we built upon prior research published in March 2020, focusing our timeframe to between the 1st of April 2020 and the 31st of December 2020. Ultimately, we narrowed our list down to subreddits that were primarily focused on American politics. In total, we identified and examined 167 political subreddits pertaining to US politics. In addition, on 4chan, we demarcated the American political sphere by extracting posts from users with an American flag on /pol/, the largest political board. In order to create similarly sized sample sets we only tagged and categorized websites with over 200 mentions on 4chan and 100 on Reddit. This approach yielded 220 results from Reddit and 219 from 4chan.

Research Questions

• What effect, if any, did the lead-up to - and the aftermath of - the 2020 U.S. election have on the spread of disinformation in 4chan and Reddit’s political spaces?
• How is YouTube being used as a news tool in both Reddit and 4chan, and what is the role of the ‘Alternative Influencer Network’ within the platform?

Methodology

To accumulate our initial dataset, we used the 4CAT: Capture and Analysis Toolset created by the Digital Methods Initiative (DMI) at the University of Amsterdam and the Amsterdam-based collective OILab. The tool allowed us to gather all posts on the /pol/, also known as “Politically Incorrect,” on 4chan, within our timeframe. We gathered all posts containing URLs by users with the American flag, since the board automatically assigns a country flag to a post according to their IP address, unless manually changed to a non-country specific flag (Burton & Koehorst, 2020).

Similarly, we used the 4CAT tool to create our Reddit dataset, relying on a Reddit bot named ‘userleansbot’ to identify the key Reddit political forums. The bot was “created in order to provide more information and transparency to the users engaged in political communities across Reddit who are interested in meaningful and honest political discussions” (userleansbot, 2020). The bot determines the political stance/leaning of users and forum boards, “providing an honest representation of the users activity on other subreddits… without having to look through their user history manually” (userleansbot, 2020). We examined the list of subreddits that the bot relies on and how they are categorized in terms of partisanship.

Through this process, we applied the “haystack-to-needle” method by first gathering a substantial amount of potentially relevant information, then cutting down that information to highlight one or more trends in our data (Hagen & Jokubauskaite, 2018). For this research we queried for hostnames within each of our population datasets. A python script was written to scrape each post from the Pushshift Reddit API, filtering out hostnames that appeared and counted them. This list was then manually categorized by site content using a qualitative tagging system. For the categorization of the websites, we used the tagging system laid out in Table 1.

Table 1. Explanations of the categorization system used for the tagging of Reddit and 4chan.
Type	Subcategory	Definition
News	Neutral or centrist	News reported in an unbiased, even-handed, and impartial manner
	Democratic	News reported with leftist, democratic intent.
	Republican	News reported with right-wing, republican intent.
Junk “news”	Propaganda	Content from biased sources, typically with a political agenda.
	Sensationalist	Content presented to provoke interest and excitement without a focus on accuracy.
	Conspiracy	Content that highlights a belief that an unknown, covert organization is responsible for an inexplicable event.
Irrelevant	Non-news	Links to other platforms and social media platforms (Facebook, Twitter, Wikipedia, archive.is).
	Campaigns	Content that is directly linked to the official campaign of one of the Presidential runners.
	N/A	Any content that is not identifiable as the above.

With YouTube growing as a “news” source, we wanted to investigate this further. On Reddit, approximately 4.2% of links were to YouTube, while 55.9% of links on 4chan were to YouTube. To gain a better understanding of the types of videos shared on the platforms, we used the DMI’s YouTube data tools to collect a video dataset: including the video IDs, uploader, share frequency, and if there are any AIN’s present. We decided to narrow our dataset for easier categorization. We did this by focusing on the top videos with over 50 shares on Reddit and over 100 on 4chan - creating a dataset of 26 from Reddit and 76 videos from 4chan. For the categorization of the YouTube URLs, we used the tagging system displayed in Table 2.

Table 2. Description of the categorization system used to tag the top YouTube videos shared on 4chan and Reddit.
Category	Description
Clips	A short, stand-alone clip without context.
Discussion	Videos featuring a conversation or debate surrounding a particular topic.
Press Conference	Videos of full (or partial) press conferences held by officials.
Interview	Videos featuring complete interviews between two individuals.
Reportage	The reporting of an event by the press or individuals.
Campaign Video	Videos created by political consultants or staff for a direct purpose.
Compilation	An edited collection of videos, clips, reports, etc.
Livestream	The live filming and transmission of an event.
Audio Book	Videos containing the recording of a book.
Shitposting	Videos containing deliberately posted low-quality content to provoke attention and to distract from the main conversation.
Meme	Videos typically of humorous nature, which are easily spreadable.
Other	Any content that is not identifiable as the above.

Findings

Finding 1: “Neutral” news sources have collapsed on both platforms.

Mainstream news sources in both the 4chan “/pol/- Politically Incorrect” board and the political communities on Reddit constitute a large proportion of the sources linked. On Reddit, news makeup 83% of the sources shared, and on 4chan, 58%.

Interestingly, compared to previous research where the news sources shared were predominantly neutral in their factual accuracy and reporting (Burton & Koehorst, 2020), the presence of such sources has decreased, and the frequency of hyper partisan sources has increased instead. In this research, we found that news sources on Reddit were mostly Democratic and accounted for 42% of all the sources shared, whereas there were only 30% neutral sources and 11% Republican sources. On 4chan, both Democratic and Republican news sources were equally present, but only 18% of the content shared included neutral news (figure 1.). This finding indicates the polarizing viewpoints of both platforms. Reddit leaned towards a more liberal perspective and 4chan towards a more conservative one, which we further analyzed in finding 3, with the appearance of campaign sources as primary sources of information.

Figure 1. Total news sources shared on Reddit (left) and 4chan (right).

Finding 2: “Junk” news comprises 42.4% of the information sources on 4chan and 15.9% of the information sources on Reddit.

When conducting our analysis, we categorized ‘junk’ news as sources that fell under ‘propaganda,’ ‘sensationalist,’ and ‘conspiracy’ content. When compared, the numbers between the two platforms differed considerably. On 4chan, over a third of the sources were made up of “junk” news. The category that appeared most frequently among the “junk” news, and made up 9% of all 4chan information, were conspiracy theories. Meanwhile, on Reddit, that number was below 4% (figure 2.). The disparity among the platforms is partially attributable to 4chan’s harboring of fringe communities that serve as vehicles for the spread of conspiratorial content (Tuters et al., 2018), particularly in 2020, with such content being bred due to President Trump’s remarks of COVID-19 and the presidential elections.

Figure 2. Unique news sources shared on Reddit (left) and 4chan (right)

Finding 3: Campaign sources are primary sources of information on Reddit and 4chan, especially as the election came closer. Whereas Democrats dominate Reddit, Republicans dominate 4chan.

When tagging the categories of the sources shared on Reddit and 4chan, we found campaign websites' appearance as sources of information, endorsing opposing political figures on both platforms. Especially as the election drew closer, the share of campaign sources increased (figure 3.), interestingly propaganda and conspiracy sources also increased leading up to the election.

On Reddit, posts mainly linked to Democratic political actors' campaign websites, including 'berniesanders.com,' 'secure.joebiden.com,' and 'dralgrossak.com.' In contrast, on 4chan, most political campaign website links shared were Republican and involved President Donald Trump. These included 'armyfortrump.com,' 'youaintblack.com,' and 'barelytherebiden.com.’

The polarized political affiliations on the platforms are explicable through the different affordances offered. The Republican domination on 4chan strongly relies on the affordance of anonymity that allows users to share their views, which tend to be more radical, without repercussions. Many of the campaign sources shared in favor of Donald Trump were not solely focusing on Trump's successes and benefits, they also actively attacked his democratic opponent, especially Joe Biden - accusing him of racism ('youaintblack.com') and being an inadequate candidate to lead the country (such as 'barelytherebiden.com'). The conservative dominance on 4chan is also a consequence of the absence of left-leaning views within the platform's culture, which leads to the creation of a vacuum in the image boards, which is ultimately filled by right-wing 'humor' (Nagle, 2017).

In comparison, Reddit lacks anonymity, and users can be held accountable for their posts. The tighter regulations may be why the Democratic campaigns shared were focused on the political figures themselves and less so on slandering their opponents.

Figure 3. Sourcetype volume between April and December 2020 on 4chan /pol/.

Finding 4: YouTube remains a critical news source, but the influence of the traditional ‘Alternative Influencer Network’ has crumbled.

The data shows that YouTube continues to play an important role as a news source on 4Chan and Reddit. The video streaming platform is especially popular on 4Chan, where the videos of youtube.com and youtu.be combined are shared over 445.000 times, making it the number one shared website. YouTube also appeared in the top 10 URL frequencies on Reddit, where the videos of both sources contain around 6700 shares. The growing importance of videos as a news source imply the preference of audio-visual material over text. In addition, the audio-visuality of news sources fits the aesthetic of the imageboard nature of both of the platforms.

In December 2020, the creators of the videos were significantly more mainstream than originally affiliated to the AIN, what Rebecca Lewis (2018) describes to be “a group of YouTube channels that blur the lines between news and personality-centric vlogging, spreading misinformation-laden commentary on social and political issues,” whose influence was way more visible in the data from March 2020, where in particular Joe Rogan and The Thinkery reigned supreme on the pseudonymous platforms. In March 2020, there were 3989 YouTube URLs shared on Reddit connected to the AIN, which in December 2020 only counted for 458 instances. In addition, the 4chan link frequency reduced from 596 to 337 shares. It is therefore safe to say that the influence of the AIN has decreased heavily over the past year on these platforms. The influence that remains is more shared on 4Chan than on Reddit.

The most important AIN related shared creators on 4Chan are ComputerFever (61), Destiny (58), and PragerU (49). Interestingly, Mike Cernovich’ “Un/Convention: Exposing Fake News at the RNC and DNC” video is shared 378 times on Reddit, in contrast to the second most popular creators, namely: TheDailyWire, TheRubinReport and TimPool, whose videos only have been shared 15 times on the platform (figure 4.). Cernovich’s video gained particular traction in a long post claiming Joe Biden’s Super Tuesday upset was reminiscent of the victory of Hillary Clinton over Bernie Sanders in the 2016 Democratic Primary and reinforces a narrative of a cheating and fraudulent DNC. This proves that both the anti-establishment left and the establishment right are susceptible to sharing and consuming the alternative news sources.

Figure 4. The Alternative Influence Network on both 4chan (blue) and Reddit (red).

Finding 5: About the alternative influencers and Amy Adams?

There has been an evident shift from direct, factual evidence and political commentary to interpretable, manipulated content through YouTube. The most shared YouTube videos on 4chan are short clips that have been taken out of context to form a new narrative that suits alt-right ideologies. One example of this is ‘SHOCKING: Joe Biden discusses the left's globalist agenda’, which is a 35 second clip, as well as ‘Kamala Harris is literally campaigning at polling stations as people stand in line to vote…,’ which is a 19 second clip. Both these videos were uploaded by user ‘Amy Adams’ as a tool to instill fear in alt-right viewers who are oblivious to the video’s full contents and context. Similarly, on 4chan a significant number of videos shared were categorized as shitposting, referring to “deliberately post[ing] low-quality content on social media to provoke attention” (Colley & Moore, 2020, p. 22). These videos include ‘Do Not Eat the Babies’, ‘Donald Trump - Never Come Down’ (a 10-hour video of Trump dancing), and ‘remember us,’ with most of the shitposting content coming from one YouTube channel: ‘Realfake Newsource.’

The top YouTube video categories shared through 4chan were shitposting (16), discussion (13), and clips (11). Whereas on Reddit, the top categories were audio books (7), discussion (7), and Interview (5) (figure 5.). Which could indicate the notion that Reddit users are more media literate than users on 4chan, further highlighting that while Democrats dominate Reddit, Republicans (alt-right) dominate 4chan.

Figure 5. Types of YouTube news sources shared on Reddit and 4chan.

Discussion

Is there a bubble to burst?

Days after the storming of the capitol, Remesh Srinivasan, professor at the University of California Los Angeles’ Department of Information studies, stated:

"When conspiracy theories spread on forums such as Discord, 4chan, or even Reddit, violent discussions about retribution and insurrectionist rhetoric around taking back the country emerge on the platform. People see their experiences on those platforms as the reality” (Mosley & Hagan, 2021).

The ideas of a closed construction of reality and filter bubbles has generated enormous debate without clear consensus (Chitra & Musco, 2020; Pariser, 2011; Geschke et. al 2019; Flaxman et al., 2016) Nonetheless, our findings do support this remark in the sense that non-partisan sources are eerily absent. Only a quarter and a fifth of 4chan and Reddit’s information ecology is composed of neutral or moderately partisan sources. However, even partisan news sources are to an even further rise in authority of YouTube, which on 4chan alone contributes to over half of the URLs shared on /pol/. This indicates a crisis, or at least a shift in authority, as users increasingly sideline all forms of print media.

Polarization, deplatforming, and new influences

Reddit and 4chan have developed in two diverging ways. As r/the_Donald was de-platformed in June 2020, many Redditors found new homes on different sites, such as TheDonald.win, which has also engulfed other far-right subreddits, including r/ConsumeProduct, r/MetaCanda, and r/Gavin McInnes. Many right-wingers have also moved to the alt-right platform Parler (Emerson, 2020; Demsas, 2021). These events left a prominent place for left-wing political philosophy on Reddit's political ecology, where links to writings and explanations of prominent socialist and communist thinkers like Karl Marx and Friedrich Engels are very prevalent. The left-wing political subreddits, especially r/Communism and r/Communism101, focus on established theory rather than on newsy sources to explain contemporary events.

Whereas Reddit has seen a sharp decline in Trumpist activity and misinformation in general, on 4chan, it is still powerful. However, the sources of misinformation were different. In March 2020, after cross-referencing Lewis' (2018) AIN with the kinds of YouTube videos shared to determine the influence of alt-right commentators, Burton & Koehorsfound that the most popular alt-right videos on Reddit are those who deal in 'alternate debate style.' In contrast, those on 4chan are more characterized by videos that use a direct-to-audience style of address that invites audiences to join them in antagonizing mainstream liberal culture (Burton & Koehorst, 2020).

After the removal of the r/the_donald subreddit by the platform in June 2020 (Timberg & Dwoskin, 2020) and the more recent banning of President Trump on the majority of mainstream social platforms (Crichton, 2021), the conversation surrounding deplatforming has quickly escalated. Similarly, there has been an increase in awareness regarding alt-right pages, forums, and accounts on social platforms to avoid the spread of misinformation or extremist ideologies. This has created a shift in alternative influencers, with some well-known alt-right names who faced permanent removal, such as Baked Alaska and The Thinkery, but instead has created space for new alternative influencers.

The calls-to-antagonizing of last year have been replaced by videos reinforcing prevalent right-wing allegations and conspiracies. These videos seem to prove election irregularities, COVID hoax theories, and the Hunter Biden-Ukraine allegations. One primary example is ‘Amy Adams’ whose videos appeared 8 times in the top 76 YouTube URLs with over 100+ shares on the platform. For ideas and ‘evidence’ to spread and take root, the talking face is seemingly less important than when Lewis designed the AIN or when this research was conducted in 2020.

Conclusion

Our findings suggest that the closed off spaces of the Deep Vernacular Web have indeed seen an increase in misinformation closer towards the U.S. presidential election and its aftermath. Whereas traditional and neutral news sources have lost ground, conspiracies and partisan news have gained it. Continuing the trend pinpointed by Burton & Koehorst, “video is the new print news” (2020, p. 3), as the authority of traditional news is especially lost towards new and anonymous publishers of short snippets of citizen journalism and compilations of political ‘evidence’ on YouTube. From Joe Biden’s Ukraine allegations, to videos of empty hospitals supporting that COVID is not real, to election fraud. The creators of these videos have an immense influence on how reality is construed on both Reddit and 4chan, given the closed-off nature of the Deep Vernacular Web.

Whereas on Reddit politics have generally shifted left, and on 4chan politics have generally shifted right, our research indicates that it will be troublesome to attempt to dislodge people in these communities from ways of thinking and conceptions of the truth. The content presented in these communities seek to reaffirm ways of thinking, not to change or challenge an established point of view. The dominance of non-influencer YouTube channels, such as ‘Amy Adams’, especially on 4chan, suggests that the big alternative influencer names are increasingly less necessary to root people in a news ecology or misinformation, enabling them to share completely unedited or uncommented videos as serving their own narrative. It brings special significance to the famous phrase that you can kill a man, but not his ideas.

That said, this research could never encompass the full news ecology of political communities on Reddit and 4chan. Although we were able to establish the dominant sources shared in all these communities, there remain many little shared sources that might, despite being less popular in quantitative terms, be just as influential in a qualitative sense. Similarly, the use of various other popular websites shared, such as Facebook, Wikipedia, and Archive.is, goes beyond the scope of this research, but remains incredibly relevant to ascertain how the ‘truth’ is construed for members of these communities. Thus, this topic knows an immense depth that we hoped to have contributed further to understanding, and we strongly encourage other researchers to expand, challenge, and deepen our findings.

References

Burton, A. G.; Koehorst, D. (2020). The spread of political misinformation on online subcultural platforms. The Harvard Kennedy School (HKS) Misinformation Review, 1(6). https://misinforeview.hks.harvard.edu/article/research-note-the-spread-of-political-misinformation-on-online-subcultural-platforms/.

Chitra, U. & Musco, C. (2020). Analyzing the impact of filter bubbles on social network polarization. WSDM '20: Proceedings of the 13th International Conference on Web Search and Data Mining, P. 115–123. https://doi-org.proxy.uba.uva.nl:2443/10.1145/3336191.3371825.

Colley, T., & Moore, M. (2020). The challenges of studying 4chan and the Alt-Right: ‘Come on in the water’s fine.’ New Media & Society. https://doi.org/10.1177/1461444820948803.

Crichton, D. (2021, January 9). The deplatforming of President Trump. TechCrunch. https://techcrunch.com/2021/01/09/the-deplatforming-of-a-president/?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_sig=AQAAAAfAeekzgoye4wXF25g9Qfmlhhjq-VhpYt9G_58im89J6JXPKaSqUjG5BIc3J_7PDppEtzDeuub1axZVlGAl81oD0Yu1heawGn8_89kfGSUQxDVKH2eau96AXbFD4DM-FdVrAkO1mZhLQnbQw-OJ5S9sLCPfldSucHxAhUbs1ewF.

Demsas, J. (2021, January 11). The online far right is angry, exultant, and ready for more. Vox. https://www.vox.com/2021/1/9/22220716/antifa-capitol-storming-far-right-trump-biden-election-stop-the-steal-hawley-cruz.

Emerson, S. (2020, July 2). Months before Reddit purge, The_Donald users created a new home. One Zero. https://onezero.medium.com/months-before-reddit-purge-the-donald-users-created-a-new-home-a732f79e4f04.

Flaxman, S., Goel. S., & Rao, J. M. (2016). Filter bubble, echo chambers, and online news consumption. Public Opinion Quarterly, Vol. 80, p. 298–320. 10.1093/poq/nfw006.

Geschke, D., Lorenz, J., &Holtz, P. (2019). The triple-filter bubble: Using agent-based modelling to test a meta-theoretical framework for the emergence of filter bubbles and echo chambers. Br J Soc Psychol, 58(1): 129-149.10.1111/bjso.12286.

Hagen, S., & Jokubauskaite, E. (2019). Dutch junk news on 4chan and Reddit /pol/. In R. Rogers & S. Niederer (Eds.), The Politics of Social Media Manipulation (pp. 115–151). Amsterdam: Dutch Ministry of the Interior and Kingdom Relations.

Heilwell, R., & Ghaffary, S. (2021, January 8). How Trump’s internet built and broadcast the Capitol insurrection. Vox. https://www.vox.com/recode/22221285/trump-online-capitol-riot-far-right-parler-twitter-facebook.

Lewis, R. (2018, September 18). Alternative influence: Broadcasting the reactionary right on YouTube. Data & Society Research Institution. https://datasociety.net/output/alternative-influence/.

Mosley, T., & Hagan, A. (2021, January 7). How social media fueled the insurrection at the U.S. Capitol. WBUR-FM. https://www.wbur.org/hereandnow/2021/01/07/social-media-capitol-mob.

Nagle, A. (2017). Introduction: The Online Politics of Transgression; Gramscians of the alt-light. Kill All Normies: Online Culture Wars From 4chan and Tumblr to Trump and the Alt-Right. Ch. 2-3. London: Zero Books.

Talley, I. & Levy, R. (2021, January 11). Extremists posted plans of Capitol attack online. The Wall Street Journal on the Internet. https://www.wsj.com/livecoverage/biden-trump-electoral-college-certification-congress/card/x1dwwPqnJM1XfQh5LaUj.

Timberg, C., & Dwoskin, E. (2020, June 29). Reddit closes long-running forum supporting President Trump after years of policy violations. The Washington Post on the Internet. https://www.washingtonpost.com/technology/2020/06/29/reddit-closes-long-running-forum-supporting-president-trump-after-years-policy-violations/.

Tuters, M., Jokubauskaitė, E., & Bach, D. (2018). Post-truth protest: How 4chan cooked up the Pizzagate bullshit. M/C Journal, 21(3). https://doi.org/10.5204/mcj.1422.

Tuters, M. (2019). LARPing & liberal tears: Irony, idiocy & belief in the deep vernacular web. In Political Science: Vol.Volume 71. Post-digital cultures of the far right: Online actions and offline consequences in Europe and the US, p. 37–48. Bielefeld: Transcript-Verlag.

Pariser, E. (2011). The Filter Bubble: What the Internet is hiding from you. London: Viking/Penguin Press.

Pearce, K. (2019, November 14). Digital disinformation and the 2020 election: 'It's going to be a complete mess'. John’s Hopkins University HUB. https://hub.jhu.edu/2019/11/14/digital-media-2020-elections/.

userleansbot. (2020). List of political subreddits used by userleansbot. Reddit. https://www.reddit.com/user/userleansbot/comments/au1pva/faq_about_userleansbot/.

Zannettou, S., Caulfield, T., De Cristofaro, E., Kourtelris, N., Leontiadis, I., Sirivianos, M., Stringhini, G., Blackburn, J. (2017). The web centipede: Understanding how web communities influence each other through the lens of mainstream and alternative news sources. Proceedings of the 2017 Internet Measurement Conference on – IMC ’17, 405–417. https://doi.org/10.1145/3131365.3131390.