Test person with EarFS prototype controlling the mobile phone using facial gestures. Credit: © Photo Fraunhofer IGD Mobile devices play an increasingly important role in our lives; in some situations, though, they cannot be adequately managed and even accepting a call is a real challenge. In a study, researchers at the Fraunhofer Institute for Computer Graphics Research IGD in Rostock, Germany have evaluated which alternative control concepts are suitable to supplement the conventional use of mobile devices. EarFieldSensing (EarFS), the proprietary development that recognizes facial gestures via a special ear plug, has a lot of potential and offers further development possibilities besides its use on mobile devices.
Modern mobile devices are usually controlled by means of a touch screen. In everyday life, however, there are many situations in which this type of operation cannot be implemented. If you are wearing gloves or have your hands full with purchases, the use of smartphones, etc. is difficult, at best. Scientists from the Fraunhofer IGD are therefore looking for alternative concepts for managing mobile devices. Operation via language is an obvious choice. However, challenges such as environmental noise as well as social acceptance impose narrow limits on language-based operation. The Fraunhofer experts' solution: operation via head and face gestures, such as winking, smiling or nodding.
EarFS measures a smile in a person's ear
In the context of researching contactless operation for mobile scenarios, the Rostock researchers have evaluated various technologies with which the movements of a person's head and face can be interpreted. In this process, great importance was placed on practical use in everyday life. For example, systems that read gestures using sensors directly on one's face are very accurate and able to recognize a large number of gestures. They are so conspicuous and unpleasant to wear, though, that they are not suitable for daily use in public. Instead, systems are needed which are as inconspicuous as possible, such as EarFS, a proprietary development of the Fraunhofer IGD. This is a special ear plug that measures the muscular currents and distortions of the ear canal which occur during facial movements. The sensor detects even the smallest movements in the face through the way the shape of the ear canal changes and measures muscle currents that arise during the movement of the face or head.
"The challenge was that these currents and movements are sometimes very small and need to be intensified," explains Denys Matthies, scientist at the Fraunhofer IGD. "In addition, the sensors cannot be interfered with by other movements of the body, such as vibrations during walking or external interferences. To solve this problem, an additional reference electrode was applied to the earlobe which records the signals coming from outside." The signals detected inside the ear are matched with the signals coming from the outside – the remaining useful signal enables clear facial gesture identification, even if the person wearing the EarFS is in motion.
Numerous application and further development possibilities
EarFS not only allows microinteractions with a smartphone, such as accepting and rejecting phone calls or operating the music player. The evaluation of facial movements also allows conclusions about weariness, exertion and other emotional states of the user. As a result, smartphones could warn drivers when signs of weariness and fatigue are registered or could switch automatically to mute when drivers are concentrating attentively. The use of the technology is also imaginable in the medical field. For example, it could help people with locked-in syndrome to communicate more easily by allowing them to operate computers with facial movements. However, this is not the end of the potential for EarFS. "Differential amplification sensing, or, in other words, the amplification of muscle currents and ear canal distortions while simultaneously filtering out external signals, fills a gap in research," says Matthies. "With the technology, we can also read activities at other parts of the body and separate them from external signals: this opens up further possibilities for us, including the complementary operation of machines in Industry 4.0."
Explore further: Battery-free technology brings gesture recognition to all devices
