Yiyu Shi, associate professor of computer science and engineering, has received one of the eight Towards On-Device AI Research awards presented to university faculty. Facebook received more than 160 proposals from 111 universities on a wide range of topics related to efficient on-device AI systems.
“Artificial intelligence has the potential to change the way people interact with the objects around them. Everyone has heard of smartphones, smart watches, and smart homes,” says Shi, who directs Notre Dame’s Sustainable Computing Laboratory and the National Science Foundation’s Industry/University Cooperative Research Center on Alternative and Sustainable Intelligent Computing.
In this project, Hardware/Software Co-Exploration of Multi-Modal Neural Architectures Targeting AR/VR Glasses, Shi and his team — Weiwen Jiang, a postdoctoral researcher, and Qing Lu, a Ph.D. candidate in computer science and engineering — are working to optimize the execution of AI applications on edge devices.
“Edge devices connect local networks to the Internet and Cloud,” says Shi. “They include things such as notebook computers, smartphones, security cameras, even ‘smart’ microwaves.
“We are exploring how to co-design neural networks and the associated hardware — augmented reality/virtual reality glasses in this case — necessary to grab the depth of information available from the environment and use it to increase task efficiency and accuracy while using less energy.”
Shi and his team are focusing their efforts on incorporating audio captured from the glasses with the normal visual information acquired from the device. For example, an image captured by a device may clearly be a cat. However, if the visibility is too low to identify the animal, then hearing a “meow” would allow the network to combine the audio and visual cues and more quickly identify it as a cat.
The project builds on the existing collaboration with Facebook, in which Shi and his team developed the first co-design framework on Application-Specific Integrated Circuits. He also has extensive experience in the co-design of neural networks for platforms, such as field programmable gate arrays and network-on-chips.
— Nina Welding, College of Engineering