Vanessa The Robot
How she works
In 2021 I had the idea use an old self driving food delivery robot I’d been given to support an animatronic piano playing puppet and drive it around the city, a la Vanessa Carlton’s famous music video.
The project ballooned in scope a little bit and tested my sanity, but I ultimately ended up with something that I’m pretty proud of. Here’s how it works.
Animatronics Design
Mechanically, the puppet consists of three assemblies - the puppet body and two hand movers which manipulate the hands via rods (like a typical muppet).
The puppet body contains servo motors controlling 6 axes of motion - body forward / back lean, shoulder rotation, head rotation, head tilt, and upper / lower jaws. The jaws are moved in tandem to create head pitch, or opposite to open and close the mouth.
Each hand mover (below) is an azimuth / elevation mounted servo pair connected to a carbon fiber rod affixed to the puppet hand. Horizontally, the puppet hand moves in a slight arc rather than linearly down the keyboard.
Thermal
Most of the servos are under load (and therefore drawing current) only when the are accelerating a part of the puppet. However the jaw servos must overcome the spring force of the foam to close the jaw fully. Originally, medium torque but high speed compact servos were used there, but even with the addition of a fan to evacuate hot air from the head they overheated and broke. They were replaced with stronger but slower servos which did the trick.
Textiles
The puppet was sewn / glued together based on a pattern from Adam Kreutinger. The clothes are all custom made for the puppet out of what seem to be some of the most annoying fabrics to sew (stretch fabric, lace, and stretch velvet). The necklace was custom made by Lane Powell.
The ornaments on the outside were sculpted digitally using an iPad app, 3D printed flat in PLA, and then thermoformed onto the piano by heating them to 140 F in an oven and pressing them while malleable against the piano contour. They were then sanded, primed, painted gold, and affixed with screws.
Lights
There are a number of pretty standard individually addressable LEDs under the piano and under the lid, a red / green laser projector pointing down from under the piano, and a disco ball with two custom pinspots mounted under the lid. The disco ball is made from a disco ball planter (hollow) within which is a motor and 3D printed interface to the disco ball. The piano lid has its own linear actuator so that it can be opened mid performance to reveal the disco ball.
In-head exhaust fan shroud
Piano
Originally I was going to just buy a toy piano and be done with it however I decided that if I was putting in all this effort I might as well make something that would look nice in my apartment and could actually be played (via integrated electronic keyboard).
The piano ended up custom designed to be about the largest thing that could reasonably fit on the robot, tilt upwards, and fit through a door. It was designed to overhang the robot significantly to look as comical as possible. To enable this it needed to be cantilevered on two legs quite strong in bending. Since I decided at some point to add a linear actuator to tilt the piano up (for navigating narrow sidewalks) it made the most sense for the legs to be welded metal box structures with the actuator inside one of them. The piano itself is made of CNC cut plywood and primarily consists of a base plate, an upper rim (to which the lid is affixed) and many ribs which connect the two. A flexible plywood cover was then glued over the ribs to form the curve.
For the non-mouth puppet movements, I have a bunch of control surfaces set up on my iPad via TouchOSC. I record puppet movements a few axes at a time by moving the puppet with the iPad and recording the movements with Logic Pro as the song plays in the background. I do an initial pass with the head pitch / rotation, a second pass with shoulders / lean, a third with head tilt, and fourth and fifth with left and right hands (attempting to roughly mimic the movements that an actual piano player would be making).
Robot Base
The robot base is a heavily modified mobility scooter originally used for food delivery. However it proved unreasonably large to navigate sidewalks, so this older generation of robot was handed down to friends of the company. They get turned into strange art projects on a regular basis.
Electrical
Motion commands are all made through MIDI control change messages between daisy chained microcontrollers. MIDI was used due to the ease of using existing software to send messages synced with audio, and the readily available optoisolated MIDI breakouts for microcontroller boards. In retrospect, I regret not just making custom PCBs with servo headers broken out, rather than needing to wire things manually. In addition to puppet movement, lighting and some additional actuators are also controlled via MIDI. Audio playback is via a Sparkfun Tsunami board, and puppet motion playback is via Seeed Wio Terminal (configured and programmed by my homie, Noah Klugman)
Performance
The puppet performance motion is programmed using two methods — motion capture from video for the mouth open/close, and recording of control surface movements for the rest of the axes.
For the mouth open/close, rather than attempting to manually program the lip synced performance I found it easiest to record myself lip syncing with tracking dots on my lips. It looks very dumb, but it was quick to do once I had a process established. I first put small pieces of green masking tape with black dots on my upper and lower lips and record a 60 fps video of myself lip syncing a (very exaggerated) performance with very bright light and a short exposure duration on an iPhone. I then use After Effects to track the position of each dot in the video and copy and past the graph output into a text file. I have a Python script written that I use to subtract the bottom lip position from the upper lip position that then outputs a MIDI file with a control change message corresponding to how open the mouth is. I can then import this into Logic Pro where all the puppet motion tracks are aligned with the song.
That’s it! I’m in the process of making a few upgrades to make the performance even more dramatic, and am planning on taking her out for spins in the city periodically. Follow on twitter @bendhoward for updates on when she’s making her way downtown!