Highlights

• Authors propose a new generative framework for medical image inpainting that bypasses the limitations of previous frameworks by enabling inpainting of arbitrarily shaped regions without a prior localization of the regions of interest.

Introduction

Distortions in medical images (e.g. motion artifacts, diffusion signal attenuation and dropouts, etc.) are tackled using a variety of methods. Inpainting methods have been proposed as an alternative to common, non-deep learning-based techniques.

In a previous work ¹ authors proposed an inpainting framework that had two major limitations:

• It is restricted to square-shaped inpainting regions
• It demands exact localization of the masked region of interest

The work proposed in this paper overcomes these limitations.

Methods

The proposed method, called ipA-MedGAN (for inpainting of arbitrarily regions in medical images using GANs), is constructed using a new cascaded generator network based on MultiRes-UNets, and two discriminator networks trained jointly as a conditional GAN (cGAN).

Their previous ip-MedGAN framework’s discriminator receives as input only the inpainted regions of interest instead of the full output and target images.

Authors argue that the need for localization is overcome using a combination of two discriminator networks with different receptive fields and different depth, one focusing on global properties, and the other on finer details. These contribute to the adversarial loss.

The non-adversarial loss is composed of three terms:

• Reconstruction loss (an \(L1\) error)
• A “perceptual loss” minimizing the discrepancy of the the resultant and the target images in the feature-space.
• A “style-reconstruction” loss inspired from neural style-transfer. This is achieved by using the feature-maps extracted from a pre-trained feature extractor network.

The contributions of each loss term were determined using Bayesian optimization.

Data

T2-weighted (FLAIR) brain MRI images from 44 subjects: 3028 scans from 33 subjects for training and 1072 scans from 11 subjects for testing.

Results

Square and random-shaped regions were cropped for the evaluation of the performance. Quantitative comparisons were carried out on the basis of:

• Universal Quality Index (UQI) ²
• Structural Similarity Index Measure (SSIM) ³
• Peak Signal to Noise Ratio (PSNR)
• Mean Squared Error (MSE)

Their baseline methods were

• Context Encoder (CE) ⁴
• ip-MedGAN ¹
• Gated-Conv ⁵
• pix2pix ⁶
• MedGAN ⁷

Results show an improvement in quantitative and qualitative performance.

Conclusions

Authors have proposed an inpainting framework capable of correcting distortions on medical images over arbitrarily shaped regions with no a priori knowledge on the pixel locations of the regions of interest.

Comments

The proposed framework works on 2D images, and authors have left using 3D images as future work.

The factor that allows to correct for arbitrarily-shaped regions is unclear.

Medical image distortions are not revealed as blank regions, so further experiments are needed to tell whether the approach provides a good accuracy when using typical medical image dropouts.

References