Aerospace and Electronic Systems Magazine July 2017 - 52

Dynamic, Data-Driven Processing of Multispectral Video Streams

Figure 1.

An example of a single video frame in the employed multispectral data set. Images 1-6 show the 6 visible bands, image 7 corresponds to the near-infrared band, and image 8 is the corresponding foreground result that is derived using LDspectral.

works reviewed in [8]. Exploration of such integration is another
useful direction for further investigation.

DESIGN METHODOLOGY
Compared with traditional imaging methods, multispectral imaging provides increased spectral discrimination, which can exploit
increasing spectral resolution and spectral diversity. Conventional
approaches assume that all of the available bands are employed for
the video processing tasks. When system accuracy is of the greatest importance, it may be desirable to use all bands. However, it
may be most effective to select a proper subset of all the available
bands in situations where resource constraints are critical due to
failures in certain subsystems or a limited energy capacity.
In the DDDAS-driven video processing system design problem, we assume the availability of multispectral data that comes
from a set Z = {B1, B2, ..., BN} of spectral bands, where N denotes the total number of available bands. In resource- or heavily
performance-constrained scenarios where it may not be desirable
or feasible to process all bands, this leads a problem of strategically
selecting a subset S ∈ 2Z, where 2Z is the power set of Z, that is, the
set of all subsets of Z.
We assume that we are given a constraint Cr (in units of time)
on execution time performance for a particular video processing
scenario. Our problem then is to select the set S ∈ 2Z to store and
process, and the associated strategy to process this selected subset
of bands such that video analysis accuracy is maximized subject to
the constraint Cr. In this article, we focus on the former aspect of
this problem-the selection of S ∈ 2Z-while laying a foundation
for incorporating the second aspect as a useful direction for future
work.
Figure 2 illustrates our first version system design for LDspectral, which is designed to address the design optimization
problem described above. Here, video processing configurations
are reevaluated periodically with the period of reevaluation being equal to the value of the reconfiguration interval parameter
52

Tr. Lower values of Tr correspond to the possibility for more frequent reconfiguration at the expense of increased overhead due
to more frequent operations for reconfiguration management. The
reconfiguration management overhead includes computations for
dynamically determining whether or not to reconfigure the system, and determining and applying the new operational parameters, including the band subset S, when reconfiguration is to be
performed.
The block in Figure 2 labeled band subset selection (BSS) is invoked at time intervals determined by the reconfiguration interval
parameter Tr, subject to application specifications. The BSS block
attempts to optimize the subset of bands that is to be employed
during the next interval of video processing. In this optimization
process, offline data (subset selection profiles) pertaining to the
effectiveness of selected subsets of bands is considered along with
recent results from performance evaluation, and the current operational constraint Cr.
The output of BSS is a vector indicating the bands S = {Bs1,
Bs2, ..., Bsm} (m ≤ N or equivalently, S ⊂ Z) that are to be processed
during the next video processing interval.
LDSpectral performs pixel-level fusion, where the selected
bands in a given multispectral image are combined pixel-by-pixel
into a single image. In the image fusion approach, each pixel in
the combined image is derived from a weighted sum of the corresponding pixels in the individual bands. Compared with featurelevel fusion, pixel-level fusion can have significantly reduced
computational cost since features are extracted from the combined
image rather than separately from each individual band (e.g., see
[12], [20]). On the other hand, feature-level fusion allows for optimization of feature extraction algorithms for each band [6]. Extension of the LDspectral framework to include feature-level fusion
and adaptive selection between pixel- and feature-level fusion is a
useful direction for future work.
The video processing functionality performed on the selected
bands is represented by the block in Figure 2 labeled band subset
processing.

IEEE A&E SYSTEMS MAGAZINE

JULY 2017

Table of Contents for the Digital Edition of Aerospace and Electronic Systems Magazine July 2017