Image acquisition

Hyperspectral Imaging

The acquisition of hyperspectral image data is becoming increasingly important for applications in image and video signal processing. For example, modern biometric security systems use hyperspectral recordings to verify the identity of people. Also in agriculture and medicine, new approaches are being published on an ongoing basis, in which, for example, hyperspectral images can be used to record the health status of plants and humans.

Among other things, we deal with innovative systems for recording hyperspectral images and videos at the chair. In addition, the reconstruction of hyperspectral data and its efficient coding are being investigated in detail. The following pictures show hidden information that can be made visible with the help of hyperspectral pictures (please click for enlarged view). The recording system and the associated algorithms were developed at the chair.

Multispectral Camera Arrays

Multispectral Imaging

Camera arrays are increasingly used in fields such as industrial quality control, remote sensing, or medical imaging. A particularly innovative system, which is shown in the figure below, is a newly developed array consisting of nine cameras and an additional short-wave infrared (SWIR) unit. Through upstream interchangeable filters, the array enables the detection of wavelengths in the infrared range that are invisible to the human eye. If three arbitrary wavelengths are overlaid as red, green, and blue channels, a so-called false-color image is created, revealing structures that are not perceptible to the human eye. This principle thus opens up a wide range of new applications, for example in identifying plastics, differentiating tissue, or detecting pressure marks on fruits and vegetables. In this way, the high sensor density and extended spectral sensitivity provide a basis for more precise and efficient analyses.

Image reconstruction

Reconstruction of Non-Regularly Sampled Data

Using the frequency selective reconstruction (FSR), images can be reconstructed on the basis of a few, irregularly measured pixels. This is a task related to compressed sensing. Compared to regular sampling, there is reduced aliasing, which means that, after suitable reconstruction, a higher image quality is possible.

One possible application of irregular sampling is so-called 1/4 sampling. Here, a camera sensor with regularly placed pixels is partially masked, so that an irregular scanning pattern is created. Using a sparse representation, the FSR can be used to reconstruct a high-resolution image.

Example of the reconstruction of irregularly sampled image data with 1/4 sampling

High-resolution reference image (218 pixels)	Low-resolution sensor (216 pixels)	1/4 sampling sensor (216 pixels)	Reconstructed, high-resolution image
Please view the enlarged version of the images, as additional aliasing may occur due to scaling.

To be able to test the performance of the frequency selective reconstruction, we provide the algorithm here as a GitLab project.

Image resampling

Digital images can be assumed to be regular two-dimensional grids of pixels. Transforming a digital image in any way, the pixel positions are transformed as well. Assuming an arbitrary image transform, pixels located on a regular integer grid before transform will be lying on arbitrary non-integer positions after transform. Pixel coordinates at arbitrary positions can neither be stored efficiently nor displayed on a digital screen. Therefore, image resampling is used to interpolate the scattered data onto a regular grid of pixel positions. We use Frequency-Selective Mesh-to-Grid Resampling (FSMR). As well as above-mentioned Frequency-Selective Reconstruction, FSMR takes advantage of spatially sparse matrices. Selecting and overlying suitable basis functions, an image can be resampled onto arbitrary positions and hence, can be displayed on a digital screen. FSMR can be used for affine and projective transforms, see the rotation below, for Super-Resolution, Frame-Rate Up-Conversion and many more.

Image enhancement

Error Concealment of Image Data

If images or video sequences are transmitted over wireless channels or the internet, the risk of transmission errors is ubiquitous. This results in the problem that individual regions cannot be decoded and displayed correctly. But it is possible to estimate these lost areas from the other correctly received regions. To achieve this, we developed the Selective Extrapolation. This algorithm is able to reconstruct arbitrary image contents and can be applied to images as well as video sequences.

Examples for concealment of distorted image data

Original image	Distorted image	Concealed image

Image restauration by Selective Extrapolation

Besides concealment of distortions resulting from transmission errors, Selective Extrapolation can also be used for image restauration. In doing so, defects or disturbing objects can be removed from images. To achieve this, the regions to be extrapolated are marked manually in a first step. For this a binary mask with zero at the regions to replace is generated. Then, the image is divided into blocks and the 2D Selective Extrapolation is applied to all blocks that contain regions to be extrapolated.

Two examples for image restauration by Selective Extrapolation:

In order to evaluate the performance of the Selective Reconstruction yourself, we provide the algorithm here as a GitLab project.