XR is getting integrated more and more into work and daily life. This includes safety-critical environments that require to display information accurately as well as precise interaction without overwhelming the user. In this workshop paper, we discuss our lessons learned from using XR in General Aviation (GA) to support pilots during their flight routine. We describe the encountered challenges for safe interaction in safety-critical environments and propose future research directions for XR environments, aiming to improve safety and security.