Duality is a simulation platform that allows engineers to create human-robot collaborative scenarios to speed up deployment time in robotic projects. The goal of this project is to create an end-to-end example demonstrating the capabilities of Duality’s platform. Given a specific GPS location, for example by analyzing satellite imagery, we will use digital twins of consumer off-the-shelf (COTS) Quadcopter and digital camera to determine more precisely the coverage of fire at that location by flying in a low altitude grid pattern.
The goal of this project is to check an area for forest fires, and if one is found, assess the severity and size of the forest fire. To do this, we created a geo-accurate, photo-realistic environment of terrain using satellite imagery of a forest within Duality’s simulator, DuSim. A drone is then placed within the environment, and flies to designated GPS coordinates within the forest. The drone takes pictures (640 x 480 px) and sends these images to Google’s Vision API. Should there be a forest fire, the drone will perform a more thorough investigation of the area to assess how widespread the fire is.
A high-level view of the architecture used in the code is below:
ForestFireDemo.py is the Python script used to control the drone through DuSim. This uses a Python API to send actuator commands and obtain sensor output from the simulation scenario. The Python script receives GPS coordinates as input to scan for forest fires and controls the drone’s orientation and its resulting GPS location through PID loops.
The script has two modes:
A high-level view of the logic within ForestFireDemo.py is below:
Some other notes on the script:
As an example, ForestFireDemo demonstrates how DuSim may be used to create your own control logic, actuate a machine, and receive sensor output from a scenario. DuSim performs the heavy lifting – all the Python process needs to do is send actuator commands and receive sensor output through DuSim. By adapting to a specified hardware controller and camera interface, the logic of ForestFireDemo.py script that controls a “digital twin” can be used with a real drone.
The scenario we used was a simple terrain with forests and shrubs that could be set on fire:
The drone first takes a picture at a high altitude in scan mode to determine whether there is a forest fire or not (clearly, there is!):
Afterwards, the drone enters investigation mode and takes picture in a grid-like fashion:
Here are some improvements we hope to make in the future:
Overall, I had a lot of fun with the project and at Duality! I learned a lot about the fine details of drone movement, path-finding, simulation, control logic, physics, and thread handling to make this project work. Hopefully, this end-to-end example showcases some of DuSim’s is capability. I am thankful that I was able to work at Duality this semester!
Special thanks to Jason’s vision for Duality, Mike for the path-finding and PID loops, Aditya for the fire scenario, and Apurva for being a helpful, insightful, and overall awesome mentor!
Making Unreal Engine accessible to machine learning, robotics and engineering teams.
An end-to-end example of RL leveraging Falcon-gym, training a model for autonomous landing