In 2009, Patti Maes, a professor at MIT’s Media Lab, delivered a TED talk in which she presented a wearable human-computer interface which interpreted real world gestures in real time. It was called SixthSense, and her student Pranav Mistry, whose idea it was, became an instant star.
In March this year, another of her grad students made a splash with an HCI device. Called AlterEgo, and in its first public prototype, it was created under Dr. Maes’s guidance by Arnav Kapur and his brother Shreyas, an undergrad student. Some of the attention was because of headlines that said it read minds.
In a video chat with The Hindu , Mr. Kapur admitted to being dismayed by some of the coverage. Aside from the clickbait oversimplification, he says, “That would also scare people. You already have technology right now that can be misused.” He refers to the current worry about Facebook’s access to user information and says that there was a conscious decision not read directly from the brain, because a user would not want their stray thoughts, sometimes involuntary, to be available to the world. “Our approach from the start was to make something that the user has absolute control over what information to transmit to another person or computing device. We chose this modality, which is somewhere in the sweet spot between thinking and speaking.”
Minimal disruption
How would AlterEgo be best described? “A silent-speech device would be more accurate,” he says. “An intelligence-augmentation device is a higher metaphor.When you say intelligence, it’s a lot of things: memory, decision-making, thinking. We have applications that actually do that.”
In an email, Dr. Maes set the background: “Today’s digital devices are very disruptive. They require our complete attention.” Like SixthSense, she says, AlterEgo experiments with a radically different form factor, with the goal being to “enable a person to make seamless use of digital information and services in a minimally disruptive way.”
“I am very excited about AlterEgo because it offers a solution that is wearable, that is private and that is very quick, convenient and seamless.”
AlterEgo interprets ‘silent speaking’ and when necessary, sends feedback to the user via audio others can’t hear. Silent speaking, Mr. Kapur says, is what we do when we consciously say a word in our minds and try to speak it internally; while we seemingly don’t move our mouths, our brains send tiny electrical signals to our speaking muscles, which AlterEgo’s sensors on the face and jaw detect.
It sends this information to a server where a machine-learning model translates it into words. “Then we have another software environment which makes sense out of the words, understands the task, whether it’s to turn off a light switch or solve a math problem, and gives you an output, which could be turning off the light, or if it warrants a reply, sending the answer to your device, which transmits it to your inner ear via text-to-speech.” The transmission is through bone conduction, vibrations to the skull no one else can hear, via the part that fits around the ear. They chose this, he says, because they did not want to block the user’s normal auditory experience.
Deep learning
Creating a device like AlterEgo requires more than casual knowledge of a variety of fields. Mr. Kapur describes himself as “not that good a student, bad in a classroom setting though extremely fond of learning,” someone who enjoys wrestling with problems. In their Noida home, he remembers, “Shreyas and I had our own workshop room where we would discuss ideas and build things together. Our learning has been outside the classroom, from real India, from observation from the world around us; we’re a product of that workshop-room learning, for better or worse.” That still defines his approach now.
“My methodology is to go deep into concepts, read a lot about concepts I haven’t encountered. I think I’ve developed the ability to sit in seemingly fragmented fields, connecting them in different ways.” The Media Lab, which brings together practitioners in wildly disparate disciplines and encourages collaboration and cross-fertilisation, is a perfect environment for him.
The fields in play so far, he says, are “Neuroscience, electrophysiology; which overlaps with biotech in a way, making the signals accessible; AI, which is computer science and machine learning; hardware design; computer-aided design, how to build hardware, miniaturise all of these components; electronics for design implementation; software application design, which is when you have the platform ready, how do you develop more applications on top of that.”
Further refinement
Dr. Maes and Mr. Kapur both stress that there is a lot of work to be done before the device can get to the market. This would include refinement of the form factor. Mr. Kapur mentions materials science research in progress, which could lead to a transparent device. Multiple languages, and within those, accents, may be problems to solve. “Yes, there is a different characterization of neuromuscular signals for each person, but we don’t know if these correlate with speech accents,” Mr Kapur says. “There might be an influence of accent on silent speech, but we’re treating this as a new form of communication, and then working towards making the platform a multi-lingual multi-user one.”
What uses do they foresee? Mr. Kapur says that they have experimented with a number of use cases, including controlling devices, solving math problems, assisting chess and go players, assisting memory by pulling up stored information.
Other important applications he and Dr. Maes see are the enabling of people with speech or hearing disabilities. She says, “We have yet to start testing for these use cases, but I am hopeful that we will be able to help some people, and that will be incredibly gratifying.”
Mr. Kapur permits himself one large statement amidst the disclaimers about work still to be done: “It gives you superpowers.”
