Aerospace and Electronic Systems Magazine July 2017 - 17

Lim et al.

Figure 11.

CHMI architecture.

The CHMI utilises psychophysiological sensors, which are
integrated into the flight deck, to monitor the pilot in real-time.
Cognition models are used to assess the pilot's cognitive states,
such as fatigue, stress, attention, and mental workload based on the
physiological data collected. Important physiological indicators
include brain (e.g., blood oxygenation levels), cardiac [e.g., heart
rate (HR), heart rate variability (HRV)] and eye (e.g., blink rate,
eye movements, and pupil diameter) activity. Brain activity provides information on cognitive workload, and can be tracked either
electrically [via electroencephalography (EEG)] or optically (via
functional near-infrared (fNIR) spectroscopy). Cardiac activity can
be measured with wearable devices such as wristbands or smart
shirts, and is utilised to assess stress and workload of the pilot.
Eye activity can be tracked remotely with multi-camera systems,
and is also a good indicator of workload-blink rates and duration are inversely correlated and decrease with increasing workload [29]. Additionally, pilot attention can be modelled from gaze
patterns, which are correlated with information sampling [30]. An
inference engine is used to manage task distribution between the
automation systems (e.g., NG-FMS) and human operators (air
based and ground based) based on the external environment and
their cognitive state. If a high single-pilot workload is assessed,
the CHMI provides support by suggesting a transition to higher
levels of system automation, reducing screen clutter, and/or transferring noncritical tasks to the ground crew. On the other hand, if
the system infers that the pilot is losing situational awareness at a
high level of automation, the CHMI either suggests a more suitable level of automation or triggers appropriate alerts to keep the
pilot in the loop. Adaptive alerting is designed to provide cues that
complement the cognitive state of the pilot, based on the system's
assessment of the situation. Alerts are prioritised and are provided
through a combination of visual, auditory, and haptic feedback;
JULY 2017

multi-sensory feedback increases the pilot's perceptual bandwidth
with more channels to process information. Figure 11 illustrates
the architecture of the CHMI.
Based on the SPO concept of operations (Section 4) and current system design-related certification requirements (Figure 10),
the key technical requirements for the design of the cognitive HMI
have been identified in Table 4, primarily revolving around the
reliability/security of the physiological sensors, the operator performance models, decision logics, as well as novel interfaces and
interactions.

Functional Allocation
Function allocation between the pilot and system, as described in
AC 25.1302-1 [28] (Figure 10), is necessary for effective operations. The VPA system achieves allocation of different functions
dynamically by performing high-level information fusion [31]
on sensor data comprising the external environmental condition,
aircraft state, and physiological states of the operator. Based on
the available information, the VPA makes an assessment of the
required automation level and provides two recommended modes
out of six modes for a given application. The pilot can select (but
is not limited to) these two modes. The operating modes for guidance, navigation, and control are depicted in Figure 12: Mode 0
implies that the pilot has full control of the aircraft; in modes 1
and 2, the autopilot provides attitude (roll, pitch, and yaw) and
trajectory control, respectively; in mode 3, the system follows the
flight plan loaded in the NG-FMS; in mode 4, the system handles
autonomous separation from traffic, weather, and terrain hazards;
mode 5 is reserved for emergency and landing in severe weather
conditions, where the system assists the pilot in performing challenging tasks such as CAT III auto landing. As an example, when

IEEE A&E SYSTEMS MAGAZINE

Table of Contents for the Digital Edition of Aerospace and Electronic Systems Magazine July 2017