FABLES is an application that gives new form to spoken word by visualizing the users’ voice as real-time reactive animation.
FABLES presents poetry and narrative as text against a white screen. As the reader speaks the words out loud, the application produces imagery and animation. The animation will take on a variety of forms depending on the reader's pitch and volume. Furthermore, the reader may improvise rhyming words as they speak. The reader’s improvisations will be reflected in the imagery revealed. The core concept is to explore the intimate connections between the experience of speaking, modulating and comprehension. By providing a rich environment to interact with the text, Fables hopes to advocate for active and engaged reading and listening practices. I hope the content will inspire dialogue on a wide range of issues between generations (culture, class, race, love, aging, integrity, family). While we are thinking of this version of the project primarily for readers of English, we will explore the blending of two or more languages.
A short clip of a demos in LAUSD
FABLES is about giving new forms and movements to the spoken word. In order to inspire a feeling of intimacy the software will articulate the “character” of the speech. Beyond this technological innovation, there is a great depth of creative opportunities to be mined within the realm of speech and meaning. Sometimes the imagery associated with the spoken words will follow closely, other times the imagery will subvert expectation. The project’s intention is to provide users with a starting point for new perspectives on language, speaking and expression.
We hope readers will engage with big questions about how knowledge is transferred, how images are invoked, and how the concept of “reading” has evolved over time. We would like users to think creatively and critically while engaging with the application.
I spent some time demoing FABLES this year in Los Angeles and Pasadena. About 200 students have tested the app over the course of 10 45-minute class sessions. On 2/23/2023 I gave my first in-class demo to 20 6th-graders at Eliot Arts Magnet in the Pasadena Unified School District. The students in this classroom all had individualized education plans and were reading at or below a 4th-grade reading level. On 4/21/2023 I conducted a demo over 7 consecutive periods in a 7th-grade English classroom across the hall. A total of about 140 7th-grade students tested the app. These 7th-grade students had a range of reading abilities: some of them were more comfortable reading out loud than others. On 5/31/2023 I gave a demo at Octavia Elementary to 40 2nd-grade students over 2 class periods. Some students at Octavia are ESL learners with Spanish as a first language.
I found these demos to be very encouraging. You can see from the short video below that 2nd graders, 6th graders, and 7th graders were all excited to play with FABLES. Even students struggling with reading were curious to try it out in a group setting.
Survey and Feedback
I submitted a survey to the 7th graders for feedback.
The Questions:
On a scale of 1-5, how easy is it to use FABLES? 1= It’s confusing, 5= Easy to understand
Would you have used FABLES to learn to read? Yes/No
Would you like to read more content for FABLES on your phone or computer? Yes/No
What was your favorite content?
What could be improved about the content or experience?
Out of 140 7th grade students, 79 students returned their surveys. I included some scanned surveys here, and the results are represented in pie-chart form below:
7th Grade Student Survey Results
Ad-Lib Games
I distributed an ad-lib style game at the end of the demos to let the kids draw their own version of “There Was An Old Man With a Beard” by Edward Leary. The instructions: ask your neighbor to fill in the blank words and then draw what they described! All of the kids were very eager to make art after playing with the FABLES app. I want to encourage kids to use their imagination and draw, create.
Next Steps
Below are my thoughts about next steps.
Publish a web app. I would like to publish a functional web application as soon as possible to send to friends, family, and schools for testing. This will help collect more data about use. I can adapt my existing content and application for WebGL.
Create material designed for literacy curricula. I would like to work with educators and experts to create content designed to support literacy education. My preference is to focus on Spanish/English bilingual content for now. I have a long-term dream to create a bilingual Pashtun/English FABLE with the Afghani refugee population in San Diego. This would all be made accessible on the website.
Develop the technology. The students’ favorite FABLE was “One Foot Two Feet” because the character of the visual output changes based on how you say the words. They were endlessly entertained by the variations! I’d like to incorporate this into more content. This will give me the opportunity to also explore the nuances of emphasis and punctuation in language. For example the meaning of “eats shoots and leaves” (describes a panda) is very different from “eats, shoots and leaves” (describes a criminal). Which word(s) are emphasized in a sentence also changes meaning. For example, consider the following sentences: “I didn’t steal your wallet,” “I didn’t steal your wallet,” and “I didn’t steal your wallet.” Even within a single word, placement of emphasis changes meaning. For example: ref-use vs re-fuse, con-tract vs con-tract, en-trance vs. en-trance, etc.
FABLES received fiscal sponsorship from NYFA to help build the project into something bigger.