Fourier Neural Operator-based Hyperprior for Learned Image Compression

Description

This work investigates improved entropy modeling for learned image compression using a Fourier Neural Operator (FNO)-based hyperprior. Learned image codecs can be viewed as variational autoencoders (VAEs), which encode an image into a compact latent representation and reconstruct it at the decoder. For compression, this latent is quantized and entropy coded into a bitstream. Training optimizes a rate-distortion objective that balances bitrate and reconstruction quality. A hyperprior transmits compact side information about the main latent, enabling a more accurate conditional probability model.

The proposed approach replaces part of the conventional convolutional hyper synthesis transform with FNO blocks. Convolutional layers aggregate information from local neighborhoods and require depth to capture long-range dependencies; the FNO instead learns an operator that maps the hyperlatent field directly to the entropy model parameter fields $\big(\boldsymbol{\mu}(\cdot), \boldsymbol{\sigma}(\cdot)\big)$ of a conditional Gaussian model for the quantized main latent $(\hat{\boldsymbol{y}})$. The rate-distortion performance will be compared with existing learned image compression models using metrics such as PSNR, and MS-SSIM, as well as complexity metrics such as parameter count and runtime. The goal is to assess whether FNO-based global context improves entropy estimation and compression performance.

⛶

Prerequisites

• Course ‘Image and Video Compression’
• Fluent in Python programming
• Experience in training deep neural networks
• Good to have: Experience with learning-based image coding