AITL-WING-HITL: Telemanipulation of autonomous drones using digital twins of aerial traffic interfaced with WING

Abstract

``By touching an instrument placed in the southern gallery, a miniature Spanish cruiser anchored in the fountain lake on the lower floor, 90 feet away, was blown into the air. There was no connection between the transmitter and the vessel in the lake'' reported the New York Times in 1898. Although it was believed as magic by many, Nikola Tesla, by pushing the boundaries once again, became the ancestor of the modern-day remotely controlled wireless drones — ``The first of a race of robots, mechanical men which will do the laborious work of the human race'' — by inventing the first radio-controlled vessel using radio waves — an invention beyond its time. Human-robot dialogue as well as the manipulation of drones has evolved significantly since then in line with the development of autonomy. In this treatise, in this research, an advanced precision control, force-sensitive patented mouse, the so-called WING, as an immersive device, was developed to achieve fine control over Remotely Operated Vehicles (ROVs). Additionally, a collision management system, the so-called DACM — which is tightly coupled with the distributed PilotAware ATOM-GRID Network — was developed for both telemonitoring of aerial vehicles and enabling effective autonomous coordination between independent heterogeneous autonomous unmanned aerial vehicles (A-UAVs) as well as manned aerial vehicles (MAVs). A scalable agent-based platform — AITL-WING-HITL — is designed using the DACM system for the telemanipulation of A-UAVs with distributed operations from the perspective of human-multirobot architecture using Digital Twins (DTs) of aerial traffic when the new driver — AI agent — runs into an unorthodox situation where autonomy is insufficient to cope with. The performance of Human Telemanipulators (HTMs) (i.e., human agents) using the WING, allowing omnipresence, in telemanipulation is evaluated in the developed framework. The evaluation results suggest that AITL (Autonomy-In-The-Loop) A-UAV agents and HITL (Human-In-The-Loop) HTM agents can co-work to achieve a synergistic task performance using a socio-cognitive interaction model. A-UAVs can be telemanipulated by a single HTM with the high-efficacy intervention modes interfaced with the WING causing no significant simulation and cyber sickness (SCS).