Gesture Recognition based User Experience

Enhance interaction between driver and car by detecting driver hand gesture for superior level driver experience with Automotive OEM.

Automotive OEM

2015

Client and Industry

A premier German OEM, renowned as a market leader, prioritizes quality and epitomizes luxury and innovation in the automotive sector. Known for its relentless pursuit of excellence, the brand has consistently introduced ground breaking technologies and luxurious vehicles, establishing the gold standard for safety, performance, and design.

Industry

Automotive OEM

Business Type

Enterprise

Services

Research, Development & Deployment

Our Role

FastCode partnered with a German Automotive OEM to redefine the way drivers interact with their vehicles, through an intelligent system that provides a deeply personalised user experience via body pose and gesture recognition.

Project Goals

01

Install an interior camera system behind the rear view mirror to enhance the user experience, enabling gesture-based controls for toggling reading lights and accessing favourite features through specific hand poses, improving interactions for both drivers and passengers.

02

Foster the OEM's in-house expertise by starting with our consultancy, gradually building a robust 200-member team to achieve self-sufficiency within four years, before our strategic exit.

Business
Challenges

01.

The system needed to be designed for low-power hardware to mitigate heat emission, as the Electronic Control Unit (ECU) was located in a part of the vehicle that could not efficiently dissipate additional heat.

02.

The model was required to recognize a wide range of activities across diverse demographics, locations, and conditions (including driving, weather, and daylight variations) with over Six Sigma accuracy, ensuring reliability in at least 99,999 out of 100,000 instances.

Solution

Utilizing our expertise in Computer Vision, we developed a state-of-the-art solution for multimodal human pose estimation using heatmaps. Our efforts focused on embedding this technology into vehicles, effectively bringing onboard intelligence to life. This achievement required a dramatic reduction in the model's size—compressing it by over 25,000 times—without compromising on accuracy.

The implementation of this technology has revolutionized how drivers and passengers interact with the vehicle, introducing a more intuitive way to control the car's systems, such as using gestures to navigate menus, close doors, and adjust the rear view mirrors by simply looking in the desired direction.

Key Features

Our solution uniquely distinguishes between various gestures made by drivers and passengers, enhancing the overall user experience.

Activating vehicle functions via hand gestures, applying a 3D Convolutional Neural Network (3D-CNN) and Gated Recurrent Unit (GRU) to analyse video frames and extract spatio-temporal features.

Advanced airbag deployment control system, utilizing a processing unit equipped with Long Short Term Memory (LSTM) neural network architecture, analyses images to determine and predict the future position and orientation of the occupant's head.

References

1 - Cholakkal Hisham , Arjun Jain, et al. Method For Identifying a Hand Pose in a Vehicle, WO2020048814, March 2020 View Source

2 - Vikram Gupta, Arjun Jain, et al. Method and System for Triggering an Event in a Vehicle, EP3895064, Jan 2023 PDF

3 - Sai Kumar Dwivedi, Arjun Jain, et al. System And Method For Deployment Of Airbag Based On Head Pose Estimation, 201911039220 A, April 2021 PDF

Impact

01.

The OEM was honoured with four prestigious Automotive INNOVATIONS awards, notably for "Most Innovative Premium Brand - Interface & Connectivity." These awards, presented by PricewaterhouseCoopers (PwC) in collaboration with Prof. Dr. Stefan Bratzel and the Centre of Automotive Management (CAM), underscore the brand’s commitment to pushing boundaries in the automotive industry.

02.

Thanks to our collaboration, the OEM developed one of the best ML teams in India

03.

The OEM integrated advanced deep learning models directly into their vehicles for the first time, setting a precedent in the industry.

04.

This innovation inspired numerous competitors to adopt similar technologies, creating a new industry standard. The game changer was moving away from traditional button controls for rear-view mirror adjustments to adjustments based on the driver's head pose, showcasing the brand's commitment to intuitive design and functionality