Augmenting Reality with MEMS Sensors

insights from industryJack McCauley Innovator in ResidenceJacobs Institute for Design Innovation

Jack McCauley understands the interplay between video game hardware and human interaction like few others in the industry. He designed the guitar and drums for Red Octane’s (later Activision’s) Guitar Hero video game series. As co-founder and chief engineer of Oculus VR, he designed the Oculus DK1 and DK2 virtual reality (VR) headsets and helped guide the company through its acquisition by Facebook in 2014.

Image Credit: SEMI

Now active in automotive technology, he builds cars at Black Lab, his private R&D facility and hardware incubator in Livermore, Calif. And, in no small feat, he thinks he’s solved the head-tracking problems in augmented reality (AR)/VR headsets – which he’ll demonstrate during his keynote presentation, MEMS Applications in Augmented Reality, October 6 at MSEC 2020, SEMI’s first virtual MEMS & Sensors Executive Congress. Register now for MSEC 2020.

Image Credit: SEMI

What inspired you to become the first person to use a MEMS sensor in a gaming device?

When I started designing the Guitar Hero peripherals, I had intermittent problems with motion tracking. I switched to a Freescale single-axis accelerometer, developed some IP around it, and that fixed the problem. That’s how I became an early customer of MEMS.

When you pioneered immersive VR gaming experiences at Oculus VR, tech industry analysts predicted the widespread adoption of VR for gaming. What do you think happened?

There are a lot of reasons why VR hasn’t become the standard-bearer for gaming. Gaming used to be a solitary activity, but as companies like Microsoft and Sony got behind multiplayer gaming, we realized many gamers found the social aspect more important than the visual aspect. Many gamers are content to play on a 2D screen or on multiple monitors because they’re playing against many people. The proliferation of internet connections worldwide has also promoted the kinship and social aspect of gaming.

Do you think VR has a place in other applications?

I think it has a lot of potential in real estate, VR movies, and engineering and design, among other areas. The automotive designer Henrik Fisker, for example, created whole vehicles in a game-engine model. If you wanted to buy one of his cars, let’s say, you could change the color and upholstery, for example, and then view it in a VR environment.  

One of the biggest obstacles to VR adoption is the motion sickness some people experience during gameplay. What would you do to fix that?

The vestibular system in the brain, which uses the inner ear, is crucial to helping you balance. If there’s a mismatch between what your eyes see and your brain is perceiving, you’re likely to feel dizzy. I’ve built a VR headset that uses a MEMS pico projector with micromirrors and a small laser for position tracking as well as for facial tracking and modeling. However, the platform’s not for sale.

Still, many of the technical advances that we’ve made in VR are helping us with AR development. The increasing power of mobile chipsets and GPUs, the decreasing geometry for individual transistors, and the way specific chips are processed, screen interfaces that will drive a 4K panel at a high frame rate, plus MEMS devices inside the eyewear for rotations and tracking are all helpful innovations.

When designing cars in your own lab, you’re doing a lot of work with AR. What do you think of AR’s commercial viability?

 I know there are well-funded AR programs in place at major companies. That’s because mobile-device companies want an omnipresent phone in front of your face. I thought Google Glass, for example, was brilliant, but it was way too early for that product, and there was too much hype behind it.

Jack’s latest R&D project is a vehicle that incorporates augmented features and a computerized display. The vehicle is a custom-built, environmentally friendly super-car with enhanced driver safety and high vehicle performance.

AR is appealing because it lets people see through a screen – and have objects appear on that screen – while they are moving through space. My son actually came up with one of the ideas I’m implementing in a car I’m designing.

We were driving in Spain, and he suggested that instead of using Google Maps to show me driving directions – which would force me to look down at an infotainment display – a sign could appear on AR glasses that would instruct me how to drive to Italy. That’s just an example of how we’ll use AR. 

After you sold Oculus VR to Facebook, you began investing time and resources into engineering education. Why did you make that choice?

I’m originally from a blue-collar family, and then I got an education at Berkeley. That made a major difference in my life. When I sold Oculus, I donated to education-focused charities primarily, because education can lift an entire family out of poverty. Let’s say your family are farmworkers, but you get a degree in engineering and land a job at Apple. That could produce a ripple effect. As other members of your family and people in your community see the benefits of your education, they’re more likely to get an education, too.

What would you like MSEC attendees to take away from your presentation?

I appreciate what the MEMS industry has done for VR because if Oculus didn’t have nine degrees of freedom (9DoF) IMU, no one would have bought our company.

A new application will come along sooner or later that will require a different type of MEMS technology, and I have total confidence that the MEMS industry will deliver what’s needed.  

For more information on McCauley’s R&D projects or on his position as Innovator in Residence at UC Berkeley’s Jacobs Institute for Design Innovation, visit his website.  

MEMS & Sensors Industry Group® (MSIG), a SEMI technology community that connects the MEMS and sensors supply network in established and emerging markets, enables members to grow and prosper. Visit us today.

About Jack McCauley

Jack McCauley is an Innovator in Residence at the Jacobs Institute for Design Innovation, where he mentors students, lectures in courses focused on product design and design for manufacturing, and leads research and development projects focused on applications of augmented, virtual, and mixed reality for design professionals and students.

McCauley graduated from Berkeley Engineering with a B.S. in Electrical Engineering and Computer Science in 1986 and credits the time he spent at Berkeley as an undergraduate with helping to ignite his career.

McCauley’s inventions, intellectual property, and patents bridge diverse technologies. He was one of the inventors of the Universal Serial Bus (USB) specification. He created the original scrolling feature for a computer mouse. As director of research for the electronic entertainment company, RedOctane, he was the chief engineer behind the Guitar Hero game series, as well as at least eight other highly successful digital games. In 2012, McCauley co-founded Oculus VR and manufactured Oculus Rift, the head-mounted virtual-reality display. Oculus was later acquired by Facebook.