What Robot Makers Must Learn from Dogs, Animators and Video Game Designers: A Conversation with Bruce Blumberg

DogAs robots become more ubiquitous, the interaction between humans and machines becomes more interesting. Understanding how we as people engage with robots or virtual characters is at the heart of Bruce Blumberg’s passion and mission, shaping a career that starts in the earliest days of Apple and NeXT, Inc. and moves on to creating World of Zoo, a video game that ultimately informed the user interface on the earliest collaborative robots. I recently sat down with Bruce to talk about his ideas about the evolving nature of the relationship between human and machine.

Q: What did the path you took from leading product marketing and development at Apple and Next, Inc. with Steve Jobs to working on human interaction with autonomous characters look like?

Thinking about the work I’ve done, on the whole, I’ve always been engaged in ways to make the user experience better. After getting my MBA from MIT’s Sloan School of Management, I worked at Apple – as a product manager for the Lisa, one of the first PCs with a graphical user interface, and later as the product manager for the Apple LaserWriter. I was the first employee at NeXT after the founders and while there I got interested in Rodney Brook’s work on autonomous systems. So, I returned to MIT to get my Ph.D. in Media Arts and Sciences, and taught for several years at the Media Lab.

My work at the Media Lab drew on the fields of Animal Behavior, Artificial Intelligence, Animation and Machine Learning. Interestingly enough, it all started because my son got interested in show dogs. Diving deep into the ways that dogs engage with people, I realized that every interaction – whether human-human, human-animal or human-machine – could be viewed as a conversation. That then led to looking at ways to create virtual characters based on the kinds of non-verbal cues and cognitive processes that scaffold interspecies communication. From there working on the user experience with collaborative robots was a very natural evolution.

Q: You’ve built your career in technology and seen several tipping points in the way technology makes our lives better. What are some of the things you’ve learned along the way?

Of course, working closely with Steve Jobs taught me the importance of design, user experience and the philosophy that no detail is too small. He also taught me to trust the technology: do the right thing in software and ride the cost curve down. I have also learned the power of taking ideas from one discipline and applying them in another: whether it was high-quality typography and helping to create the Desktop Publishing industry, or ideas from animal behavior informing the AI for computer characters or UX design for industrial robots. Finally, much of my work has been in developing tools for others, and here I am guided by the philosophy that simple things must be simple and complex things possible. Hopefully, all of this has allowed me to see ways to harness technology in ways to truly solve problems and improve our lives

Q: You worked at a video gaming company that paved the way for much more interactive engagement between players and the characters on the screen. What got you interested in gaming?

I’ve always loved building things, making things move and creating characters where perceptions are perceivable. They all came together in the pursuit of ways to make video games more interactive and engaging.

For me, it all goes back to that idea of a conversation. Humans, dogs and many other animals use non-verbal cues to scaffold their understanding of the interaction. For example, give a command and if the dog doesn’t understand what you want, it will cock its head to the side – who knows why they do this, but we use it as a reliable cue that they don’t understand us. Or think about a rattlesnake that communicates a warning, by coiling up and shaking its rattle before it strikes. Much of this is rooted in the physics of movement or the biology of attention but we use these cues to help construct a story about the mental state of the other, and guide our subsequent interaction. And trust me, animals are doing the same to us.

Classical animators have known this instinctively. Think about a Disney character pitching a ball: the animators focus on exaggerating the motion of the wind-up so that the audience can anticipate what comes next. The anticipation prepares the viewer’s eye and mind for what is to follow. I call this ‘playing to the amygdala’. The amygdala is a primitive, but critical, structure in the brain that makes snap decisions based on coarse sensory data and which in turn guides more sophisticated cognitive processes.

Another important understanding that came out of my work with dogs, and my reading, especially the work of Winifred Gallagher, was the importance of solicitation and attention. Dogs are “man’s best friend”, in large part because they are biased to attend to us, and most importantly they actively solicit our attention and care, and by doing so, they make us feel better about ourselves. And for dogs, it has turned out to be a terrific evolutionary strategy.

So at Blue Fang, we set out to create a video game that took these ideas and delivered on a child’s fantasy of being a zookeeper where the fun of the game was based on the child’s moment-to-moment interaction with the animals

Q: How did you translate that experience into developing the user interface for collaborative robots?

As robots are increasingly deployed in close proximity to people they need to interact with the real world. Something that computer engineers and scientists consistently underestimate is how hard it is to build technology that incorporates anticipation, expectation, and solicitation.

Here again, the concept of thinking about the interaction as a conversation between the person and the robot comes into play. Human cognitive processes lag behind immediate perception by some number of milliseconds and so getting the anticipatory clues right is critical to prepping person for what comes next. That’s in large part why the Baxter cobot we built at Rethink Robotics had a moveable screen that through its movement and images communicates to the person it is working with that it “understands” – or not – what it is being asked to do. For example, the screen would ‘nod’ as an acknowledgment of a command. The noise of the nod and the movement of the screen, seen out of the corner of the eye, was an incredibly efficient way to communicate, “yup, got that”. Not only does the interface make the robot approachable to workers on the floor, it communicates cues that are recognizable by those workers and allows for continued “conversation” that makes for a positive experience for both parties.

Q: We’re seeing and reading a lot about machines and AI taking over. Thoughts?

Rodney Brook’s blog does a very good job of covering AI cogently. Personally, I think there are a lot of misconceptions about artificial intelligence and we are overestimating the power of what’s been done to date.

More important than a level-set on the progress that’s been made though, is recognizing that no matter how great the technology might be, infrastructure can dramatically slow adoption. Technology is always adopted quickest when it doesn’t have to fit into an existing infrastructure. We have to always go back to what has to be true in order for something to work. The Segway is a great illustration of why: here was technology that was going to revolutionize the way people moved, but, cities are full of curbs and stairs and sidewalks are full of pedestrians who don’t want to share the sidewalk with a mechanized vehicle. I am very pessimistic about the rate of adoption of autonomous cars.

I think we’re better spending our time on augmented technology. These are not fully autonomous systems but are designed to collaborate with people to help them do their jobs better, increase safety and so on. Careful attention will need to be paid to transparency and extensibility: transparency so the user has confidence in the how the system works, and extensibility so that if the system only gets it 95% right as far as the user is concerned, the user can add the 5% that makes it perfect for that user. One of the challenges for deep learning is that these systems create representations that are neither transparent nor extensible.


Bruce’s journey has certainly taken some interesting twists and turns. For me, his focus on creating a user experience that’s positive and transparent and how that informs the way machines and humans work together is compelling. Is it possible for humans and machines to truly have a dialogue and move work forward through that conversation? Share your thoughts on that connection with me. Where do you find positive experiences in working with machines? Tweet me @jim_lawton.

Originally published on Forbes.