Open issues in the automatic interpretation of text-image relations

In this seminar, I will discuss the limits of current AI-powered systems for the automatic or semiautomatic interpretation of problematic text-image clusters (shortened as TICs in Sindoni and Polli 2022). TICs cannot be interpreted by considering language or visuals independently from each other, since they produce meanings through mutual interaction between verbal and visual components (Baldry 2004; Dynel and Poppi 2021). The pragmatic, intercultural and inferential skills of the human brain can be only partially reproduced by machines. A current problem in this respect is that algorithms cannot always successfully identify dangerous or harmful contents and meanings not conveyed by language-based alone resources.
Memes are basic examples of TICs, as they consist of basic text/s plus image/s, but this standard configuration typically incorporates a pragmatic conflict between the text’s and the image’s meanings, thus producing a humorous effect that may be harmless and playful, but potentially convey hateful or dangerous contents. This case study will focus on the problems of automatic identification of harmful digital texts and will provide empirical indications about how a multimodal critical discourse approach (Machin 2013; Ledin and Machin 2017) can contribute to reducing online hate speech and discrimination  (Sindoni and Polli 2022).

MS TEAMS LINK