Abstract
This lecture addresses the very purpose of language, (effective) communication, and its limits. For cases of a medical emergency, first-aid kits and even defibrillators (AEDs) are available in many public places, and they come with instructions combining pictograms and text. However, while this may be a matter of life and death, there are currently no guidelines for the design of such multi-modal instructions (MIs) that effectively instruct people to operate an AED, produce an arm sling, remove ticks etc., the variation in pictorial and verbal means employed in these instructions is huge. This lecture will present an ongoing project that investigates the use, effects and optimisation of documents that contain pictures and text (PAT). This project innovatively combines approaches from Information Design Research and Computational Linguistics employing corpus collection and analysis, (automatic) annotation, experimentation, and natural language generation. Besides theoretical results in terms of empirically validated models for effective MIs, this project has some very practical values, too, including new annotated multimodal corpora, software to automatically annotate potentially effective properties of MIs, algorithms to automatically generate effective text-picture combinations and authoring guidelines to produce good quality instructions.
Mini-bio
Dr. Ielka van der Sluis is a computational linguist, specialising in the areas of human communication, multimodal information presentation, human-computer interaction, natural language generation, affective computing and multimodal interaction. She researches the production and processing of the visual and verbal means that people employ to communicate as well as the contexts of use in which this happens through collection and annotation of corpora and empirical experimentation, with the aim of improving in computational models for use in real-life applications. She is currently a member of the Center for Language and Cognition Groningen (http://www.rug.nl/research/clcg/) and an Assistant Professor at the Department of Communication and Information Science, at the University of Groningen. In the past she has worked as a Research Fellow at the Computational Linguistics Group at the Department of Computer Science at the Trinity College Dublin in Ireland, the Natural Language Generation Group at the University of Aberdeen, and the University of Tilburg.