Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology

Brett A. Halperin and Stephanie M. Lukin. 2023. Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ‘23). Association for Computing Machinery, New York, NY, USA, Article 240, 1–21. https://doi.org/10.1145/3544548.3580744

Example Data Instance:

• Source Images:

What problem does this paper address?

Creative visual storytelling based on image sequences.

What is the motivation of this paper?

• Modern works on visual storytelling systems mainly focused on description of events or scenery, but did not address the problem of “Where should the story from the image end? And if it ends, what is there beyond?”. Bridging these creative leaps can potentially break the barriers that separate images from a larger story world occurring “off-screen”.
• The storyteller’s biased perceptions of the image can relay stereotypical character archetypes or harmful messages.
• Training a system to automatically generate stories based on human-authored stories will reproduce human biases.
• The paradox of generating harmful narratives from biased data and inevitable biases being source of creativity when explored critically and creatively.

What are the research questions?

• What are the varied ways in which human authors approach the same systematic creative process of improvised story-building based on image sequences?
• How can the envisioned narrative intelligence criteria for computational storytelling guide the development of more equitable training datasets and facilitate the design of robust algorithmic models for visual storytelling?

What is the objective of this research:

The paper explores human-authored visual storytelling to establish a foundational understanding for developing computational narrative intelligence systems. It collects a curated anthology of 100 stories derived from image sequences, analyzing creative storytelling processes. [page 1]

What are the main contributions of this paper?

• This paper introduces a data collection paradigm of the systematic, creative process of improvised story-building from image sequences.
• This paper releases an anthology of visual storytelling.
• This paper expands the debates on visual storytelling and computational models, and the avenues of increasing creative expression beyond literal descriptions and surface-level commentary of images.
• This paper exposes how narrative biases are encoded in the process of visual storytelling and offers suggestions for responsibly recognizing them.
• This paper suggests ways to derive plausible storylines from images by characterizing objects / entities.
• This paper contributes considerations for narrative intelligence criteria for computational visual storytelling.

Which type of qualitative research methods was adapted to conduct the thematic analysis?

Qualitative research methods for narrative knowledge engineering.

Which five themes are identified in the thematic analysis?

The analysis reveals five recurring themes in human storytelling:

• Narrating “What is in Vision” versus “Envisioning” beyond images.
• Dynamically characterizing entities/objects.
• Sensing experiential information about the scenery.
• Modulating the mood of the narrative.
• Encoding narrative biases. [page 4]

Which narrative intelligence criteria for computational visual storytelling was envisioned?

1. creative
2. reliable
3. expressive
4. grounded
5. responsible

Narrative Creativity:

Stories showcased variations between literal image descriptions, inferred deductions, and imaginative deviations. For instance, authors either captioned the visuals, inferred plausible scenarios, or invented entirely new narratives beyond what the images depicted. [page 7]

Characterization of Entities / Objects:

Objects and entities in the stories were characterized in diverse ways, from being overlooked or treated as static props to becoming interactive or even personified elements like team members with human traits. [page 8]

Multisensory Experiences:

Some stories utilized multiple senses (sight, smell, sound, touch) to enrich the narrative and immerse readers in the depicted scenes. This contrasted with unisensory stories focused solely on visual elements.

Mood Modulation:

Authors employed two main narrative trajectories for mood modulation:

• “Gloom to Doom” intensified an initially somber mood into darker outcomes.
• “Gloom to Bloom” reversed bleak beginnings into uplifting, hopeful conclusions.

Narrative Biases:

Stories reflected inherent biases, such as stereotypes and cultural assumptions, which have implications for training computational storytelling systems to avoid perpetuating these biases irresponsibly. [page 11]

Experimental Setup:

Stories were derived through an improvisational process involving sequences of three images. Authors responded to structured prompts, progressing through four facets: [page 6]

1. Entity
   • “What is here?”
   • mimics the visual activation that draws attention when first viewing an image
   • list visible entities, describe visual characteristics, and report their degree of confidence in their assessments
2. Scene
   • “What happens here?”
   • treat the scene as a spatiotemporal snapshot and provide information about depicted actors and actions
   • literal explanation and factual description of the image, identifying location, time and activities
3. Narrative
4. Title

Dataset Composition:

The dataset includes two distinct image sources:

• High-quality images from Flickr.
• Search-and-rescue (SAR) scenarios with low-resolution and atypical perspectives, fostering diversity in storytelling. [page 6]

Data Statistics:

Future Implications:

The findings emphasize the importance of grounding computational storytelling systems in human creativity, ensuring outputs are expressive, grounded, and responsible. The anthology serves as a benchmark for understanding and addressing challenges in narrative intelligence. [page 5]