Applications of Thermal Cameras in the Automotive Industry

In this interview, AZoSensors talks to Mike Walters, VP of Product Management in FLIR's Automotive division, about the benefits and applications of thermal cameras in the automotive industry. 

Could you give our readers an overview of FLIR and the automotive solutions that you offer?

FLIR is the world's leader in thermal cameras. We have about 2 billion in annual revenue and about 3,000 employees worldwide. We have been in business for over 40 years and about 60 percent of our business is directly related to thermal cameras.

Over the last 15 years, FLIR has shipped over 700,000 thermal cameras into automotive applications. To date, these cameras have been used for driver-enhanced night vision systems and they have been provided in conjunction with FLIR's tier-one partner, Veoneer. The thermal camera systems have been an option on higher-end vehicles that allow drivers to see pedestrians or wildlife, such as deer, while driving their vehicle at night.

What are the main challenges associated with using autonomous vehicle systems?

Having an accurate perception of the environment around the vehicle and using that perception to do path planning and operate the vehicle safely is one of the major challenges. Having a sensor suite that is capable of perceiving the environment in all lighting and weather conditions is critical. Thermal cameras, since we're looking at heat and not reflected light, do a particularly good job of detecting pedestrians and other living things that you'd like to avoid, especially at night.

Image Credit: Zapp2Photo/Shutterstock.com

They're also very good at seeing through many types of bad weather, such as fog, and seeing in difficult lighting situations that challenge other sensors such as visible cameras. When you're driving directly into the sunrise or sunset, a human driver has to squint their eyes, giving them a less complete picture of the road ahead and causing potentially dangerous driving situations. A visible camera can be blinded in these situations as well, from the massive "blooming" effect caused by the sun or bright oncoming headlights. But because a thermal camera sees heat, the sun remains a small disc in the sky and we're able to determine objects ahead of the vehicle very easily. Oncoming headlights are not very warm and similarly cause no blooming issues for FLIR thermal cameras.

How do your thermal cameras perform in difficult environmental conditions such as low light or harsh weather?

Since a thermal camera is looking at heat and not reflected light, we need absolutely no light to see. The technology has been developed originally for military applications to provide the ability to see at night. That's one of the chief benefits that a thermal camera brings to vehicle perception: the scene self-illuminates since everything in the world around us is emitting heat. A thermal camera is able to detect its surroundings even in absolute darkness, so we have very good visibility at night.

We're also able to see well through many types of fog. If there are pedestrians approaching an intersection or walking along a sidewalk, you might not be able to see them with your eyes when you're driving in very foggy weather, but a thermal camera will be able to see those pedestrians quite well.

How does FLIR's thermal camera increase the safety of advanced driver assistance systems?

FLIR's thermal cameras can improve advanced driver assistance systems and we think there's a particular opportunity for thermal cameras to improve automatic emergency braking systems. In the fall of 2019, the American Automobile Association (AAA) published an evaluation of four cars with current automatic emergency braking systems and they found that most of the current automatic emergency braking systems implemented in vehicles today fail in common driving scenarios about 60 percent of the time in daylight and 100 percent of the time at night.

They published some videos and you can see the vehicles that are out on the street crashing into test dummies and shockingly, how poorly these automatic emergency braking systems (that typically combine data from a visible camera and radar platform) perform.

FLIR recently contraced VSI Labs to develop a fused-AEB algorithm on their test vehicle that combined data from radar and a thermal camera and repeated some of these Euro NCAP or new car assessment program tests. In their preliminary testing at day and night, VSI Labs was able to successfully stop the vehicle in 100% of tests performed. These are scenarios that are common in the real world but difficult and challenging for current systems, such as a pedestrian emerging in the street from behind a parked car that would block your view of the pedestrian until they're in the street. 

This is preliminary work and VSI Labs is going to continue to develop this test vehicle and repeat the full suite of in-cab testing. But we believe, because a thermal camera is looking at a heat signature, that's what gives it the ability to very readily detect pedestrians at night. We think this will dramatically improve the safety of automatic emergency braking systems.

In the U.S. there are about 6,000 pedestrian fatalities a year and about 75 percent of those fatalities happen at night, which equates to about 4,000 deaths at night per year. This is an area where we think thermal cameras can make a big improvement in the safety of automatic and effectiveness of automatic emergency braking systems.

What benefits do FLIR offer in automotive integration?

FLIR offers several tools to help automotive advanced driver assistance systems designers quickly evaluate and integrate thermal cameras. FLIR offers an automotive development kit, or ADK, that includes FLIR's Boson 640 x 512 resolution thermal camera in an IP67 housing with various electrical interfaces that a developer can quickly attach to their data collection vehicle and begin evaluating thermal cameras for themself.

Boson 640 x 512 resolution thermal camera

In addition, FLIR offers a number of annotated thermal and visible image datasets. We currently have a free dataset from the Santa Barbara area, as well as a dataset of San Francisco that is available for designers to download and begin training their neural networks with thermal images.

Lastly, FLIR announced at CES that we're partnering with ANSYS to develop a thermal camera model, a virtual reality driving simulation environment that is physics accurate. Designers are able to configure a vehicle in this environment with a multitude of sensors, radars, visible cameras, and now, thermal cameras, and create simulation scenarios to supplement data they collect while driving their vehicle, and examine the effectiveness of their system for corner cases or challenging cases that may be hard to collect and may require many miles driven to collect.

About Mike Walters

Mike Walters is Vice President of Product Management for Uncooled Thermal Cameras at FLIR.  He has extensive experience defining, developing and bringing to market high-volume longwave infrared and visible camera modules.  He holds a Masters Degree in Electrical Engineering from Stanford University and BS Electrical Engineering and BA Physics degrees from the University of Southern California.