Final Documentation of “Lingo Gizmo”

The “Lingo Gizmo” is a fabricated interface that lets people invent words for a missing dictionary. People collaborate over time to submit meanings that don’t have words yet, and invent original words for those meanings.

At the ITP Winter Show, I shared a prototype in which people could invent a word and assign an original meaning to it all in one interaction. I learned many things from this two-day user test that I can apply to a later version.

Check out this short video of the interaction below. 



You can play yourself online!

The code is hosted on Github.

Here are some fun examples of words and meanings that people created at the show.

  • ‘mlava, food for volcano monsters
  • lajayae, a person of Latin American origin wishing to live in New York City
  • juhnaheewah, the feeling that someone behind you is having a better time than you are.
  • dahvahai, good times, good mood
  • dayayay, to tell a donkey to calm down
  • erouhhtahfaher, a food that tastes like cement in a pail
  • fapaneuh, to strike fear into the enemy’s heart
  • kadaveaux, too many knishes
  • nabahoo, that feeling when you’re just not going to get out of the house today
  • payvowa, a special kind of grapefruit
  • Quaeingiwoo, a town in Western Connecticut


Inspirations & References

I created this project because I’m interested in how people build culture through shared experiences, and the ways language acts as a tool for naming and codifying that culture.  In some ways, this project is a slang maker, allowing people to name a feeling  that others may have, and give it a new status with its own word.

I also love creative word games such as Balderdash, which is a parlor game based on voting on the best-guessed or faked definitions of words for a chosen obscure word in the dictionary.

Lastly, I think of words as physically related to their meanings. The shape a word creates in one’s mouth can inform their meaning.  Therefore, it wasn’t a stretch for me to think to ask users to create words by physically interacting with a mouth. Interestingly, there is a theory called the Bouba Kiki Effect.  The theory suggests that people across different cultures and languages are very likely to associate the shapes below as “kiki” on the left, “bouba” on the right.  This phenomena suggests there may be a non-arbitrary, or purposeful, mapping between speech sounds and the visual shape of objects.

500px-Booba-Kiki.svg.pngOne last great reference suggested to me by Allison Parrish, on faculty at ITP, is the Pink Trombone.  It’s an online interface lets you manipulate the inside of a mouth to generate accurate sounds. Very fun to play with.


How It Works

Many skills and strategies went into this project. See below for a summary.


The face, teeth and tongue are designed and sewn by myself, using patterns I developed with paper prototypes. I did not know much about sewing before starting the project!

The inside of the face includes a cardboard structure with a hinge and rubber band to allow the top of the mouth to move down for consonants like “ma” “ba” and “wa”.


In my 500+ lines of code, I’ve used the p5.js libraries to play the opening animation, cycle through sound files, add chosen sounds to the user’s word, and save the user’s inputs into the file name of the recording, which is created with the p5’s recorder function.

Physical Computing

I used an Arduino Mega microcontroller because it offers enough inputs to accommodate the project’s nine sensors.  Five sensors are made up of conductive fabric and velostat circuits. Three are force sensing resistors. The last sensor is an accelerometer to measure the x axis movement of the top of the mouth. I used a ADXL326 chip.

All nine input values are processed by a sketch I’ve saved on the microcontroller. The sketch takes the analog values and turns them into a changing list of 0s or 1s to signify whether they are turned “off” or “on” by the user. The p5.serialport library allows me to send that list from my microcontroller to my online browser. My browser is running a local server to serve up my code along with the serial data so that the user can interact with the fabricated mouth interface.

Design and User Feedback

Many rounds of design, branding, and user feedback informed this project. I used lots of paper and pen to map out my ideas, and used Illustrator to finalize the design and system logic of my project.  Over time I had several formal user feedback sessions with classmates, and quickly asked for input at critical moments in the process.


Next Time

The ITP Winter Show confirmed that if I had another couple days, my list of additional features were in the correct order. Here they are!


1. Get rid of the mouse by creating a box with three buttons to let the user press “Add” “Start Over” and “Done” while interacting with the mouth interfaces. This would simplify what the user has to touch in order to complete the first interaction.


2. Create a user flow of three pages, each with a distinct purpose. First, an opening animation to attract people to play. Second, a page to create one’s word. Third, a page to add their word to the dictionary.  Currently, it’s all in one page which is a little too much for someone to get through at once.

Physical Computing

3. While I learned a lot using different sensors, next time I would use only one kind of sensor for the same kind of human gesture of pressing down. I was asking people to press each “fabric button”, but underneath were different sensors which each required a different kind of touch.

Overall Design

4. On a larger scale, my first prototype demonstrated that people understand and are interested in the concept, feel ownership over their words, have a lot of fun with the current interaction, and are very surprised and delighted by the end. However, the definitions don’t have as much variety in mood or tone as I’d like to encourage in a final version of the project. As of now, people add definitions that match the playful and slightly absurd interaction that I’ve created (strange mouth parts anyone??) . Very few are introspective, although two people did submit definitions about wishing to move to NYC or worrying that someone else is having a better time than they are.

One thing I want to do is rerecord the audio recordings to include more variety in phonemes. Right now they all end “ah” because they are taken from instructional resources online. Including “ack” and not only “kah” will give people more choice.

Considering my recordings all end in “ah”, any word people make sounds pretty silly. Therefore, the inviting but strange branding and design that I created for the show fit that experience. Next time, I can change the design to set a different tone for people’s interactions, in hopes of giving people the room to submit definitions that have more variety to them.



System Diagrams

Here are a few diagrams of my work.

Phonemes Chart


My circuits as a schematic and illustrations.

Meaning Maker Circuit Illustrations and Schematics


These webpage layouts are close to what I would finish creating with more time.

Webpage Design_Build A Word-01Webpage Design_Build A Word-02Webpage Design_Build A Word-03Webpage Design_Share a Meaning-01Webpage Design_New Meanings and Words-01






Intro to Computational Media – Update on Final Project “Meaning Maker”

I’ve decided to combine my Intro to Physical Computing and Intro to Computational Media final projects. I’ll save by “What’s Your Character” project for winter break or next semester! I’m disappointed to not work on it now, but want to focus on completing more of one project than a little less of two projects.

You can read more about my “Meaning Maker” project over here.  I’ll do as much of my larger idea as I can. At a minimum, I hope to complete the interaction of building words by using the mouth enclosure. I may have to save the “game” element of people interacting with each other through the interfaces for another time.

Here are my interaction diagrams, schematics, and interface designs.

I’m sure these will continue to change and I’ll update them with my final post.




A chart assigning phonemes to different interactions.

Phonemes Chart


My circuits as a schematic and illustrations.

Meaning Maker Circuit Illustrations and Schematics


My webpage design so far, created in Illustrator.

Webpage Design_Build A Word-01Webpage Design_Build A Word-02Webpage Design_Build A Word-03Webpage Design_Share a Meaning-01Webpage Design_New Meanings and Words-01






Intro to Computational Media – Notes on Github

To help myself, here is a cheat sheet for sending updates to Github using the terminal.

  1. Stage the file for commit to your local repository.
    git add .
    # Adds the file to your local repository and stages it for commit. To unstage a file, use 'git reset HEAD YOUR-FILE'.
  2. Commit the file that you’ve staged in your local repository.
    git commit -m "Add existing file"
    # Commits the tracked changes and prepares them to be pushed to a remote repository. To remove this commit and modify the file, use 'git reset --soft HEAD~1' and commit and add the file again.
  3. Push the changes in your local repository to GitHub.
    git push origin your-branch
    # Pushes the changes in your local repository up to the remote repository you specified as the origin


If I’m given a message that I can’t escape out of because my master repo is one or more commits ahead of my local repo, then I use these commands in the Terminal.

  1. press “i”
  2. write your merge message
  3. press “esc”
  4. write “:wq”
  5. then press enter

Then enter a “git pull” command into Terminal to get the latest master repo onto my computer.

Then enter a “git status” and “git add .” any files that aren’t uploaded to the staging area.

Then enter a “git commit -m “…” command

Then enter a “git push” command

Class 10 Intro to Comp Media – Proposal “Collage Character Portrait”

Project Summary 

Ideal Scenario

I’m hoping to eventually design a large projected interaction. People can playfully answer “What main character are you in a novel?” People would drag images of minor characters, similes and metaphors to the outline of their figure, as a way of defining themselves by what surrounds them. The interaction would take a few steps:

  • Enter a photobooth-like environment
  • Snap a photo of their whole body, which is turned into a silhouette. This is projected on a wall.
  • Drag from a “pile” of various characters/objects at the base of the projection to surround their figure’s outline.
  • At the end of the interaction, people can print out their picture and take it home with them. Or post online. Etc.

Prototype for Now

But for now, I’ll make a simple prototype for my Intro to Computational Media final…. This will include being able to use a laptop’s camera to capture a portrait and turn it into black and white (or other colors), drag objects to to your portrait’s outline with the mouse or possibly with touch, and then finally email yourself the photo or post it online.

Context and Audience

This is definitely for fun, is interactive, and is meant to give people a moment to consider how what is outside their bodies can define their identities. (Which is in great contrast to my Intro to Physical Computing project, which externalizes what’s going on in people’s heads. You can see more about that project over here.)


Looking forward to redrawing these sketches next time…!

This is my “Prototype for Now”



Here is my Ideal Scenario

Later on, I hope to build it further into this type of interaction:




Why? And Inspirations

This idea is based on a digital humanities project I contributed to this summer. You can learn more about it below. I’m interested in how people or characters are defined in creative ways by objects or the people around them.

Background (not necessary to read unless you’re really interested! Mostly for me to reference later.)

This digital humanities project was led by my friends Sarah Berkowitz and James Ascher at University of Virginia. They used textual analysis and github to explore the nature of character in 19th century literature and new practices of digital transcription in the 21st century. Their project focused on Characters, the second volume of a book called Genuine Remains by Samuel Butler, a 19th century author in England.  Each chapter in the book is a brief series of “jokes” about a stereotypical person, such as “A Wooer, ” “An Astrologer,” and a “Corrupt Judge”. The descriptions are biting, witty, and act a bit like a dictionary of people. You can see online the transcription and analysis here on this website and over here on Github.

While working on the project, I was struck that only men were featured as main characters, which is unsurprisingly for the time.  But the absence of women made us wonder even further about ALL the “invisible ink” minor characters mentioned in each chapter. How do these passing characters add definition and meaning to the main character?   Can these “invisible” characters be made more visible?

Sarah did some amazing analysis and a group of chapters to categorize “non-specific humans,” “proper names,” “mythological creatures,” and “animals”.  I’ve been inspired to take this type of analysis and let people play with it in a visual and fun way.



Source Material

I need more references, so please suggest them! I did find:

  • This interactive window projection by NuFormer, a group in the Netherlands, came up in my google search. I like how it turns your body into another texture and outline.


  • This pinterest collection of “interactive wall installations” is helpful.

  • There I found this interesting advocacy intallation against child abuse.  It seems to just use projection from the back to turn your body into a black shadow.


  • This is also cool:



For my simple prototype, I’ll use:

  • The Coding Train videos on how to use a laptop’s camera to create portraits and modify pixels to become black and white, or whatever colors I choose.
  • A library called matter.js that Dan Shiffman recommended, which is a 2D physics engine for the web. I can use it to mimic the effect of a “pile of trash” of objects on the ground, that people can “pick up” and attach to the outside of their portrait.


Collecting ideas for a title and 1-sentence description

I am literally collecting ideas. Let me know!

Project Planning

I will be making a spread sheet to give myself a certain number of hours each week until the finals, so that I’m forced to keep this manageable.


November 15

November 22 (no class because of Thanksgiving)

November 29

December 6


User Testing of Interface/Environment

I’ll also be doing a few user tests with just paper and pencils, to understand if there’s anything elegantly simple I can do to make the interaction more compelling and easier to understand.

I also need to ask people what metaphors or similes they would describe themselves as, so I can be sure to have a variety of


I need more code references for the behavior of snapping something to the outline of a shape. I didn’t get far online. Otherwise I’ll just use the mouse to move the collaged objects around, and some very suggestive user interface to insist the user put them around their outline…

I also need to find out how to drag a shape next to another, so that the one is partially hidden behind the other. In my mind, the outline of the figure is the first layer, and shapes are a little behind the figure as a second layer.


Class 10 Intro to Phys Comp – Final Proposal “Meaning Maker: By the Mouthful” OR TBD NEW NAME

I’ll be turning my midterm into my final, which means I’ll have more time to build my project out further. This is a game that lets people create new words to fit meanings that don’t have words in the English language… yet.

See below for my plans!

Project Summary 

To start playing this game, people can decide where they’d like to start. They can either:

  • Share a meaning, feeling, or observation in their life that could use a word in the English language.
  • Create a new word, record and spell it.
    • They can match this new word with a meaning from someone else.
    • Or if they wish, they can go right ahead and define the new word they just created.
  • Vote on the top words
  • View/listen to the best ranked words.
  • Also, because I’m curious whether people will use these new words… it would be great to let people share whether they used a word recently, too. Something like “Let us all know if you heard a word used! What was the sentence?”


I’m interested in creating a moment for people to:

  • Have fun finding out what other people are thinking about, but not talking about.
  • Build culture by creating new words a community can use together.
  • Pause for a moment to think about how the mouth forms sounds.
  • Consider whether language allows for a full expression of how they’re feeling on the inside, and give people some agency to think of language (and therefore their world-view?) as not fixed.
  • If people are especially theoretical or grounded in linguistics… they might think about how the actual sounds of words may intentionally describe objects and experiences (see more about this Kiki and Bouba effect below)


Context and Audience

I’d like this game to be installed in a shared place for a week or longer, in which people spend time together whether consistently or as they come and go.

Obviously, the ITP Floor is a great place to start. Placing it in a library, a coffee shop, museum, or the subway platform would all lead to different interactions, too.

I do wonder how “sticky” the interaction will be. Will people want to come back and see what others have created? Or will it be a one-time interaction?

I also wonder how people who speak multiple languages will interact with the game. Will they be inclined to add meanings that are in their language, but not English? That might mean they try to create actual words in another language with the mouth components.

Another consideration is how clumsy or expressive do I want the action of creating words to be? Right now its very playful but fairly clumsy. Linguistic and phonetic experts would find it to not be expressive as something like this Pink Trombone vocalizer. How important is that at this point?


As for my interest in words, I’ve always liked the game Fictionary, which lets you create fake definitions for obscure words in the dictionary. When I play the game Bananas, I enjoy turning the last round into a competition of made-up words that you must defend with fake definitions at the end.  And who doesn’t enjoy learning about words in other languages that don’t exit in English? Sometimes other languages have better words for describing feelings. Other times a word can describe a feeling I’ve never felt before.

Source Material

I met with Allison Parrish, a faculty member here at ITP.  She offered a lot of sources and examples to consider as I decide what is most important to emphasize in my project. They include:

  • Sniglets – A 1980’s American TV game about “sniglets” which can be “any word that doesn’t appear in the dictionary, but should.” The monthly TV episode as part of the show “Not Necessarily The News” became a series of books.
  • Fictionary and Balderdash – A parlor game and then board game that involves voting on best-guessed or faked-definitions of words for obscure words in the dictionary.
  • The Bouba Kiki Effect –  As an interesting example of synesthesia, people across different cultures and languages are very likely to associate the shapes below as “kiki” on the left, “bouba” on the right.  This phenomena suggests there may be a non-arbitrary mapping between speech sounds and the visual shape of objects.  Even further, some believe that the evolution of language might have to do with the neurological feedback of the shapes that the mouth creates while speaking, in that humans might use sound symbolism to non-arbitrarily map sounds to objects in the world.  It’s possible that this extends even to ideasthesia, in which people may or may not sense concepts or sense ideas as perception-like experiences. ***In short, I may need to experiment with my game design to lead users towards expressive combinations of consonants and vowels that better match the emotional “shape” of other people’s submitted meanings.***


  • Pink Trombone – Have fun playing creating sounds with different parts of the mouth! This is a game changer for music teachers of voice and wind and brass instrumentalists… you can show your student how to shape the inside of one’s mouth! Key to shaping tone and articulating notes.
  • ITP graduate thesis project – I’m still looking for a link to a previous student’s thesis which used physical actuators to create a voice synthesis for vowels.
  • Greg Borenstein – is another ITP graduate who has a Twitter Bot account @fantasticvocab that generates new words with new meanings “out of the atoms of English”.
  • Suzette Haden Elgin – Elgin is a writer of science fiction, including a book series called Native Tongue. The series uses a new language she created, called Láadan.

Project Planning

I do need to get lots of advice to actually make this thing! These are the questions I have:

  • User interface of Mouth components
    • How to better construct my mouth parts out of fabric, especially when they become larger and might have wireless chips inside. I might be able to ask someone in the Tisch costume department.
    • How to make the mouth parts wireless, possibly using XBee wireless chips.
  • User interface of Screens
    • How to best strategize using screens, whether just one or multiple, and what combination of keyboards, mouses, and touch gesture is best.
    • How to design the environment surrounding the screen(s), and what materials are best to use.
  • Database
    • How to create a database that stores people’s words, meanings and votes, that can be accessed over time.
    • How to create a voting mechanism as well.

Bill of Materials

  • User interface: Mouth parts
    • Fabric
    • Fiber fill
    • Thread
    • Misc. other materials as needed
  • User interface: Screens
    • Can I borrow screens from the ER?
    • Do I need to buy my own?
    • Keyboards? Mice?
    • My own buttons to reduce need for keyboards?
  • Database
    • What do I need to build a database accessible online? Do I need to pay for this or can I do this for free?


User Testing of Interface/Environment

I have three options for how to display my project.

Most Simple

I can have just a simple set up with one or two monitors, the mouth components, and perhaps a separate enclosure for special buttons.


More Work Required

Ideally, I would be able to divide up the different entry points of the game into “stations”. I’m a little concerned that the activity of coming up with meanings/definitions is too introspective to stand up to the absurd sounds of the vowels and consonants playing back while other users might be creating words.


Lots of Work Required

Even more of an ideal, it would be cool to have a column with three faces for each entry point into the game.



That’s it for now! Looking forward to doing a user test in class tomorrow.

Class 9 Intro to Comp Media – Sound

I’ve been using sound in my Intro to Physical Computing midterm. Below is a video of the interaction. The game lets you create new words to fit meanings that don’t have words yet. You build words by pressing different combinations of a mouth, teeth and tongue to unlock various consonants and vowels. There’s more to read about it over here!


Here’s an older iteration that shows the screen up close. If you hear a consonant or vowel you like, you can save it to a growing word at the bottom of the screen.

Class 9 Intro to Physical Computing Midterm – “Making Meaning: By the Mouthful”

Here is the final iteration of my midterm project. This is a game that lets people create new words to fit meanings that don’t have words in the English language… yet.


The interaction so far:


To start playing, people can decide where they’d like to start. They can either:

  • Request a new word for a meaning they’ve always wished there was a word for.
  • Create a new word, record and spell it.
  • Define the meaning of a new word.
  • Vote on the top words and view the best ranked words.

Currently I’m focusing on designing for the second starting point of the game. To do so, people take apart a human mouth, and use a mouth, tongue and teeth in different combinations to unlock consonants and vowels. They can combine these syllables into words. There’s more to the game described in my last post.

For now, here are some more photos, the sketch online that you can play with, and my code.

The Enclosure



The Online Sketches

Below is a link to the latest code I’m working on. However, this only works if you have the mouth with you!

But here’s an example of the sketch you can play online without the mouth.


The code

Thanks to Leon, Chino and Jen!!