Rapid adaptation of deep learning teaches drones to survive any weather

Drones, or autonomous flying machines, must master real-world weather and wind patterns in order to be genuinely helpful.

At the moment, drones are either piloted by humans using remote controls or are flown in controlled environments with no wind. Although drones have been trained to fly in formation in wide spaces, those flights are typically carried out in perfect circumstances.

However, in order for drones to autonomously carry out necessary but commonplace tasks like airlifting injured drivers from a car accident or delivering packages, they must be able to adjust to changing wind conditions in real time. This is known as "rolling with the punches" in meteorological terms.

To address this issue, a team of Caltech engineers created Neural-Fly, a deep-learning technique that enables drones to adapt to novel and unanticipated wind conditions in real time by just altering a few crucial parameters.

A paper that was released on May 4 in Science Robotics describes Neural-Fly. Soon-Jo Chung, a research scientist at the Jet Propulsion Laboratory and Bren Professor of Aerospace, Control, and Dynamical Systems, is the corresponding author. The co-first authors are Michael O'Connell (MS '18) and Guanya Shi, both graduate students at Caltech.

The Real Weather Wind Tunnel, a customized 10-foot-by-10-foot array of more than 1,200 tiny computer-controlled fans that enables engineers to mimic everything from a moderate gust to a storm, was used to test Neural-Fly at Caltech's Center for Autonomous Systems and Technologies (CAST).

The problem, according to Chung, is that the direct and particular impact of different wind conditions on aircraft dynamics, performance, and stability cannot be effectively described by a straightforward mathematical model. "We employ a combined approach of deep learning and adaptive control that allows the aircraft to learn from prior experiences and adapt to new conditions on the fly with stability and robustness guarantees," the authors write. "Rather than trying to qualify and quantify each and every effect of turbulent and unpredictable wind conditions we often experience in air travel, we instead employ this approach."

O'Connell continues: "We have a wide variety of models that are drawn from fluid mechanics, but it can be difficult to get the proper model fidelity and to adjust the model for every vehicle, wind situation, and operating mode. On the other hand, current machine learning techniques need enormous volumes of data to be trained, yet they fall short of the cutting-edge flight performance attained using traditional physics-based techniques. Additionally, real-time adaptation of a whole deep neural network is a difficult, if not currently impossible undertaking."

According to the researchers, Neural-Fly overcomes these difficulties by employing a technique known as separation strategy, which requires that just a small number of the neural network's parameters be changed in real time.

This is accomplished, according to Shi, with the help of a unique meta-learning method that pre-trains the neural network and only requires updates to these crucial parameters in order to accurately capture the environment's changing conditions.

Autonomous quadrotor drones fitted with Neural-Fly learn how to react to severe winds so well that performance is dramatically enhanced after as little as 12 minutes of flying data (as measured by their ability to precisely follow a flight path). In comparison to current state-of-the-art drones outfitted with comparable adaptive control algorithms that recognize and react to aerodynamic effects but without deep neural networks, the error rate following that flight path is roughly 2.5 times to 4 times smaller.

Based on prior systems called Neural-Lander and Neural-Swarm, Neural-Fly was created in partnership with Yisong Yue, professor of computing and mathematical sciences at Caltech, and Anima Anandkumar, the Bren Professor of computing and mathematical sciences. The drone's landing trajectory and rotor speed were adjusted by Neural-Lander using a deep learning technique to account for the rotors' backwash from the ground and ensure the smoothest possible landing; Neural-Swarm trained drones to fly autonomously in close proximity to one another.

Despite the fact that landing may seem more difficult than flying, Neural-Fly can learn in real time, unlike past systems. As a result, it doesn't need to be adjusted after the fact and can react to variations in wind on the spot. In flight tests carried out away from the CAST facility, Neural-Fly performed just as well as it did in the wind tunnel. The team has also demonstrated how flight data collected by one drone can be shared with another, creating a database of information for autonomous cars.

Test drones were required to fly in a predetermined figure-eight pattern while being buffeted by winds as high as 12.1 meters per second, or about 27 miles per hour, or a six on the Beaufort scale of wind speeds, at the CAST Real Weather Wind Tunnel. It would be challenging to use an umbrella in this "strong breeze," according to the classification. It falls just short of a "strong gale," which makes moving challenging and causes entire trees to tremble. Given that this wind speed is twice as fast as the wind speeds the drone experienced during neural network training, Neural-Fly may be able to extrapolate and generalize effectively to harsher conditions.

A typical, off-the-shelf flight control computer that is widely used by the drone research and hobbyist community was installed in the drones. An onboard Raspberry Pi 4 computer, which costs roughly $20 and is about the size of a credit card, was equipped with Neural-Fly.

California Institute of Technology