Thermal Cameras Gain Acceptance For ADAS And Autonomous Cars

Transportation Thermal Cameras Gain Acceptance For ADAS And Autonomous Cars Sabbir Rangwala Senior Contributor Opinions expressed by Forbes Contributors are their own. I write about perception, sensors and LIDAR for autonomous vehicles, drones and robots Following New! Follow this author to stay notified about their latest stories. Got it! Oct 28, 2022, 09:47pm EDT | New! Click on the conversation bubble to join the conversation Got it! Share to Facebook Share to Twitter Share to Linkedin Left – Visible Camera Image, Right – Thermal Camera Image.

Imagery from a FLIR Thermal Camera . . .

[+] Enables Pedestrian Detection and Classification at Extended Distances in Light Starved Conditions Teledyne-FLIR Thermal imaging measures temperature differences in a scene by imaging each resolved point to a corresponding pixel in a camera sensor. Each pixel is sensitive to radiant thermal energy, which changes the electrical resistance and enables measurement of scene temperature. These sensors are typically made using silicon CMOS technologies and are called micro-bolometers.

Bolometers were invented more than a century ago – the rise of semiconductor technologies led to the practical realization of arrays of pixels (µbolometers) that could be used to provide thermal contrast images of a scene. Honeywell invented the technology in the 1970s under a classified contract with the U. S.

Department of Defense, which led to the development of long-wave infrared cameras (LWIR) for ISR (Intelligence, Surveillance, Reconnaissance). Portions of the technology were de-classified in the early 1990s and transferred to multiple commercial entities. Today, thermal imaging serves applications ranging from surveillance (defense and border control), medical diagnostics and remote temperature monitoring (fever detection) to nighttime driving.

Micro-bolometers are sensitive in the 7. 5 – 14 µm wavelength range of thermal radiation. This part of the spectrum is invisible to human eyes and visible cameras (which rely on incident light or photons).

Animals and humans have body temperatures that are typically different from other natural or artificial objects in the scene. Measuring these temperature differences can provide unique perception capabilities in various lighting and obscuration conditions, critical in automotive applications where pedestrian and animal detection at long range is essential for safety and autonomy. Several car companies pioneered the deployment of passive thermal cameras in the early 2000s (General Motors, BMW, Honda, etc.

), primarily to provide safety in nighttime driving due to animal collisions or injury to pedestrians in areas with poor lighting or heavy fog. These were strictly meant to assist the human driver. As the DARPA Grand Challenge kickstarted higher levels of autonomous driving, different sensing technologies gained prominence, with LiDAR (Light Detection and Ranging) attracting the most interest and funding.

Along with radar and visible cameras, this suite of sensors was widely promoted as the ideal perception stack for higher levels of autonomy (L3 or conditional autonomy with a human driver ready to jump in and assume control with a 10-second alert and L4 or full autonomy under a specific ODD (Operational Design Domain). The realization of L4 autonomy is proving to be more difficult than initially projected because of the various corner cases that typical road conditions involve. Some companies are adding thermal cameras to the mix since this is a sensor modality that can address gaps with LiDAR, radar and visible cameras.

These include recognizing animals and humans in light-starved or obscured environments (fog, smoke, steam). Waymo Via and Plus. ai are doing this for trucking autonomy on highways.

Nuro , Cruise and Zoox deploy these on their purpose-built vehicles for last-mile food, grocery delivery, and ride-hailing services in densely populated cities. MORE FOR YOU Meet The Unknown Immigrant Billionaire Betting Her Fortune To Take On Musk In Space Talking Travel With Fox News Sunday Anchor Shannon Bream Today’s ‘Heardle’ Answer And Clues For Saturday, October 29 The U. S.

National Highway Traffic Safety Administration (NHTSA), part of the Department of Transportation (DOT), is also active in this area. It recently issued an RFC (Request for Comments) for the New Car Assessment Program (NCAP) . As part of this RFC, the NHTSA proposes to add four new ADAS (Advanced Driver Assistance Systems) technologies.

These include blind spot detection, blind spot intervention, lane-keeping support, and pedestrian automatic emergency braking (PAEB). For PAEB, in particular, the RFC lists thermal cameras as a potential sensing technology. General Motors responded to this RFC and listed far infrared cameras as a sensing technology that could prevent animal and pedestrian collision accidents.

Interestingly, Tesla, which promotes the “visible cameras” only approach for autonomy, also recommended the evaluation of infrared cameras for PAEB in nighttime conditions. As mentioned above, Zoox is one of the companies that uses a stack of four thermal cameras as part of its sensor suite on its bidirectional, purpose-built ride-hailing vehicle. Jesse Levinson (Zoox CTO) makes the following points about adding this functionality: Advancing L4 autonomy relies on addressing corner cases.

The point is to give the machine-learning stack the best information possible for safe driving. Thermal cameras sense heat, allowing the identification of humans and animals independent of lighting and obscuration conditions. This sensing modality provides the capability to address unique corner cases that other sensors like LiDAR, radar and visible cameras can miss.

Low illumination, smoke, fog and steam are examples of this. Using thermal cameras helps the perception stack to reduce false positives. Highway ranges are not required from the thermal imager.

Wide FoV (Field of View) and lower imaging ranges are acceptable for urban driving. The cost of the sensor stack is always a concern. But Zoox is in the service business, not the car sales business.

Higher sensor stack costs are amortized over time and ride revenues. Figure 1 demonstrates an LWIR camera’s effectiveness in detecting pedestrians and other objects through a layer of steam. Figure 1: Steam or Smoke Obscures Pedestrians (Inset is Visible Camera Image).

LWIR Camera Image . . .

[+] Can See Through Obscurants Zoox The supply chain for thermal cameras is well-developed in the U. S, Europe, Korea, Japan and China. The sensor is a crucial consideration, with Teledyne-FLIR being a leading provider.

This capability grew out of the FLIR entity based in Santa Barbara, California, in the mid-2000s (Indigo, later purchased by FLIR). FLIR provided some of the initial sensor capabilities used for human driver assistance by OEMs like BMW (Veoneer, which was part of Autoliv, the Tier 1 supplier of thermal cameras). Teledyne acquired them in 2021.

Today, Teledyne-FLIR is a dominant provider of LWIR thermal cameras for a wide range of applications from firefighting and security to drones and automobiles. John Eggert is the Head of Automotive Business Development at Teledyne-FLIR. The Boson family of thermal imagers is now in its 4th generation and available in VGA (Video Graphics Array) format of 640×512 pixels or QVGA format of 320×256 pixels.

The pixel pitch for either format is 12 µm. A detailed comparison of pedestrian detection performance in heavy fog is provided across the visible (0. 4-0.

7 µm), SWIR (Short Wave Infrared, 0. 9-1. 6 µm), MWIR (Mid Wave Infrared, 2-3.

5 µm) and LWIR (8-12 µm) imagers are provided along with LiDAR data (at 0. 9 µm). LWIR performs the best across these options for various ranges and fog densities.

Figure 2: Left-Visible Camera Image, Right – FLIR Thermal Camera Image. Thermal) camera clearly . .

. [+] sees a pedestrian in a dark, foggy roadway. Teledyne-FLIR Optics is an essential consideration in thermal imaging.

Given the 8-12 µm wavelength range, Germanium was the traditional choice for military LWIR imagers. This elemental material has excellent optical properties at these wavelengths; however, other alternatives must be explored for high-volume automotive applications. One reason is that over 90% of global Germanium reserves come from China and Russia.

Given the geopolitical environment, this is potentially a supply chain concern. Umicore, a Belgium-based materials company, is a leading Germanium optics and semiconductors provider for lenses, solar cells and infrared VCSELs (Vertical Cavity Surface Emitting Lasers). According to Bendix De Meulemeester, Director of Business Development, PAEB is a significant concern given the high rates of pedestrian fatalities today (exacerbated by increased levels of human driver distraction).

As a result, regulatory and safety rating agencies across the globe are increasingly demanding nighttime and bad-weather pedestrian detection and AEB features for human-driven passenger cars. Thermal cameras are critical for this. Higher levels of autonomy (L3 and L4) will also require this capability for highway and urban driving.

According to Mr. De Meulemeester, the thermal cameras required for human-driven cars will be much more cost-sensitive (<$100/camera) than those deployed on L3 and L4 vehicles. Human-driven cars need lower-range and wider FoV performance and can use a single QVGA imager/car.

For L3 and L4 autonomy, VGA format imagers will likely be used with more expensive optics to support long-range and narrower FoV for highway driving and deploy multiple cameras/car. However, an alternative to Germanium optics is required for all automotive applications. Chalcogenide glass is an avenue Umicore is actively pursuing in terms of material formulation and lens fabrication because it has the following advantages: Volume Scalability : Glass lenses can be molded in large batches, unlike Germanium lenses, which must be machined.

Batch production promotes volume scalability, higher production rates and lower capital and labor costs. Operating Temperature Ranges: are broad for automotive applications, typically -40°C to + 85°C. Germanium properties like transmission losses vary substantially over such ranges, and other materials must be used to athermalize the construction and ensure uniform optical performance.

Chalcogenide glass properties allow for cost-effective athermalization for stable performance over a wider temperature range. Size and Weight: Germanium is denser than glass and, therefore, heavier. The requirement to athermalize the design also increases size and weight.

Cost: Factors 1 – 3 lead to lower cost points for glass lenses vs. Germanium, a critical consideration since, as imager costs reduce with volume, optics become a significant cost item in the Bill of Materials (BOM). Cost efficiencies become crucial, especially if large-scale automotive deployment is to occur.

Supply Chain Concerns with Germanium since, as highlighted earlier, > 90% of Germanium is sourced from China and Russia. Figure 3: Umicore Tessella™ Wafer Level Optics and Lens Assemblies Cover a Variety of FoV . .

. [+] Requirements for Automotive Applications Umicore Other companies like LightPath Technologies (in the U. S.

) and a number of suppliers in Germany and China are active in developing and productizing chalcogenide glass optics for thermal imaging. U. S.

Defense primes who use LWIR cameras for ISR are also transitioning away from Germanium to this material system for some applications. Autonomy is incredibly difficult. Especially in public domains.

Safety is priority #1. If the goal is to replace a human driver (who are generally very good drivers), adding sensors to cover corner cases is important. Otherwise, the autonomy movement (which is actually slowing down now) will never progress.

Although humans are good drivers, sometimes weather and atmosphere do not cooperate. Providing good and affordable driver assistance is also incredibly important. Thermal cameras are a good addition.

Will more sensor modalities be required? Follow me on LinkedIn . Sabbir Rangwala Editorial Standards Print Reprints & Permissions.