Multistain Immnumohistochemistry Tissue Segmentation using End-to-end Color Deconvolution Deep Neural networks

DATASET

APPROACHES

UNET multiple stainings

CD-UNET

UNET single staining

METHODS

COLOR DECONVOLUTION segment

UNET

Original UNET

Groundtruth: 1 per tile with: pathologists annotations (512x512x1 )

Class Imbalance: inherent imbalance of classes : use loss weghting strategy = median frequency baslancing

Each slide: split into
overlapping 512x512x3
at 10x (half or.resolution)

Data Augmentation: hue, brightness, scale jittering, add uniform noise to intensity channel, image flips, small random affine transformations

Annotations:

51 slides training (10% for validation) 26 slides testing

77 WSI colorectal carcinoma. H&E+8 IHC = 9 stains

Tissue: 16% (pixels)

Tumor: 19%

Necrosis: 7%

Exclude= irrelevance, artifacts,etc

Bacground:58%

Lower resolution needed for increasing contextual information when classifying pixel info

Optimization: SGD optimizer with momentum of 0.9

Multiple GPUs with synchronous SGD

Loss function: average of cross entropy loss. loss for ignored pixels=0 weight

GPUs: 15

Output: 512x512x4 with probabilities for each class for each pixel

Batch size:240 (16 per GPU)

Input: RGB 512x512 to range [0 1]

Default learning rate per GPU=0.001

Distributed module: PyTorch

UNET training

Dropout layers at the endo of the encoder and the decoder (less overfitting)

Batch normalization after every convolutional layer

Smaller network width: half the number of filters per laayer (higher speed and less overfitting)

Zero padding in all convulutional layers (preserve input size)

Increased generalization capabilities

Smooth convergence of F1 scores after 200 epochs

No smooth convergence.

Big difference for necrosis and background

Prefers to train a single model for all stains rather than multiple models

3 filters in second layer to mantain architecture's input size

Each layer followed by nonlinear function

6 filters in the first layers for each stain color

ReLu and Batch Normalization

Color deconvolution parameters learned as part of segmentation network

No predefinition of stain basis parameters

Faster learning

Better generalization

F1 scores converge faster and smoother. Higher Scores

RESULTS

Color Deconvolution Segment Visualization

Apply activation maximization on filters of first layer

CD segment output: visualize output with different stains

Activation max of the first layer filters: generate synthetic images that max activate response

Feature visualization

Insert noise and select target output

Synthetic images for tissue: regular cell structures in the center and far from it

Max tumor and necrosis score: look for patterns of condensed large distorted cell nuclei around target pixel

Max tissue score: tissue cell nuclei far from target pixel. patterns of multiple tissue cell nuclei around target pixel

Pixel Attribution

Process: use smooth grad + average score gradients → ReLu → threshold gradient image

Tumor: the gradients higlight tumor cell nuclei in target pixel surrounding and ignore other textures

Effective receptive field different between categories

Normal tissue : the gradients higlight healthy nuclei in target pixel surrounding and ignore other textures. Large effective receptive field