Advanced Image Sensing Systems

Advanced Image Sensing Systems

Before images and videos can be processed and analyzed, the necessary data must be recorded, enhanced, or reconstructed in a first step. For this purpose, the topics at the chair for image acquisition are collected in the first section. In particular, the acquisition and processing of hyperspectral data and the acquisition are discussed. By measuring spectral components, e.g., UV or IR, hidden information can be made visible to the human eye. If multiple cameras are used to record images and videos, also stereoscopic applications can be developed.
In the third section, the image reconstruction is examined in more detail. On the one hand, a new type of sensor concept (non-regular sampling) is presented and on the other hand, the resampling of images is discussed.
Finally, the last section focuses on image enhancement. In particular, the concealment and reconstruction of missing image areas is shown there.

Image acquisition

Hyperspectral Imaging

Contact
Frank Sippel, M.Sc.
E-Mail: frank.sippel@fau.de
Link to person
PD Dr.-Ing. habil. Jürgen Seiler
E-Mail: juergen.seiler@fau.de
Link to Person

The acquisition of hyperspectral image data is becoming increasingly important for applications in image and video signal processing. For example, modern biometric security systems use hyperspectral recordings to verify the identity of people. Also in agriculture and medicine, new approaches are being published on an ongoing basis, in which, for example, hyperspectral images can be used to record the health status of plants and humans.

Among other things, we deal with innovative systems for recording hyperspectral images and videos at the chair. In addition, the reconstruction of hyperspectral data and its efficient coding are being investigated in detail. The following pictures show hidden information that can be made visible with the help of hyperspectral pictures (please click for enlarged view). The recording system and the associated algorithms were developed at the chair.

Spectral reconstruction

Contact
Frank Sippel, M.Sc.
E-Mail: frank.sippel@fau.de
Link to person
PD Dr.-Ing. habil. Jürgen Seiler
E-Mail: juergen.seiler@fau.de
Link to Person

Using multi- or hyperspectral imaging system, as shown above, leads to further tasks. For a lot of applications, the reflectance spectra are an essential part for classification tasks. Therefore, reconstructing the emitted spectrum from multispectral images is a fundamental operation. This operation leads to several challenges. First of all, one typically estimates more variables than observations, which emphasizes the need for prior information. Furthermore, multispectral videos, e.g., imagine a drone flying over agricultural fields while measuring the plant health, are usually heavily affected by noise due to the limited exposure time of each frame. Consequently, spectral reconstruction techniques, which are robust to noise, need to be developed. The applications of such a technique vary heavily. For example, different types of plastic can be discriminated, drug counterfeits can be detected, or security features of bills can be examined:

Reconstructed spectra (the colors of markers and spectra match) Reconstructed images at specific wavelengths

 

Image reconstruction

Reconstruction of Non-Regularly Sampled Data

Contact
Teresa Stürzenhofäcker, M.Sc.
E-Mail: teresa.stuerzenhofaecker@fau.de
Link to Person
PD Dr.-Ing. habil. Jürgen Seiler
E-Mail: juergen.seiler@fau.de
Link to Person

Using the frequency selective reconstruction (FSR), images can be reconstructed on the basis of a few, irregularly measured pixels. This is a task related to compressed sensing. Compared to regular sampling, there is reduced aliasing, which means that, after suitable reconstruction, a higher image quality is possible.

One possible application of irregular sampling is so-called 1/4 sampling. Here, a camera sensor with regularly placed pixels is partially masked, so that an irregular scanning pattern is created. Using a sparse representation, the FSR can be used to reconstruct a high-resolution image.

Diagramm zum 1/4-Sampling

 

Example of the reconstruction of irregularly sampled image data with 1/4 sampling

High-resolution reference image
(218 pixels)		 Low-resolution sensor
(216 pixels)		 1/4 sampling sensor
(216 pixels)		 Reconstructed,
high-resolution
image
Please view the enlarged version of the images, as additional aliasing may occur due to scaling.

To be able to test the performance of the frequency selective reconstruction, we provide the algorithm here as a GitLab project.

 

Image resampling

Contact
Viktoria Heimann, M.Sc.
E-Mail: viktoria.heimann@fau.de
Link to person

Digital images can be assumed to be regular two-dimensional grids of pixels. Transforming a digital image in any way, the pixel positions are transformed as well. Assuming an arbitrary image transform, pixels located on a regular integer grid before transform will be lying on arbitrary non-integer positions after transform. Pixel coordinates at arbitrary positions can neither be stored efficiently nor displayed on a digital screen. Therefore, image resampling is used to interpolate the scattered data onto a regular grid of pixel positions. We use Frequency-Selective Mesh-to-Grid Resampling (FSMR). As well as above-mentioned Frequency-Selective Reconstruction, FSMR takes advantage of spatially sparse matrices. Selecting and overlying suitable basis functions, an image can be resampled onto arbitrary positions and hence, can be displayed on a digital screen. FSMR can be used for affine and projective transforms, see the rotation below, for Super-Resolution, Frame-Rate Up-Conversion and many more.

Example for the rotation of an image by an arbitrary angle.

Image enhancement

Error Concealment of Image Data

Contact
Nils Genser, M.Sc.
E-Mail: nils.genser@fau.de
PD Dr.-Ing. habil. Jürgen Seiler
E-Mail: juergen.seiler@fau.de
Link to Person

If images or video sequences are transmitted over wireless channels or the internet, the risk of transmission errors is ubiquitous. This results in the problem that individual regions cannot be decoded and displayed correctly. But it is possible to estimate these lost areas from the other correctly received regions. To achieve this, we developed the Selective Extrapolation. This algorithm is able to reconstruct arbitrary image contents and can be applied to images as well as video sequences.

 

Examples for concealment of distorted image data

Original image
Distorted image
Concealed image

Image restauration by Selective Extrapolation

Besides concealment of distortions resulting from transmission errors, Selective Extrapolation can also be used for image restauration. In doing so, defects or disturbing objects can be removed from images. To achieve this, the regions to be extrapolated are marked manually in a first step. For this a binary mask with zero at the regions to replace is generated. Then, the image is divided into blocks and the 2D Selective Extrapolation is applied to all blocks that contain regions to be extrapolated.

Two examples for image restauration by Selective Extrapolation:

In order to evaluate the performance of the Selective Reconstruction yourself, we provide the algorithm here as a GitLab project.

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2000

1998

1997

1993