Feature Fusion for Medical Imaging: Lessons from DIA-VXNET

Diabetic eye disease detection from fundus images is a problem where two backbones see different things. VGG16 captures fine local texture; XceptionNet's depthwise separable convolutions capture broader, more abstract patterns. Fusing them should help — if you can get their features to align.

This is the story of DIA-VXNET, published in Biomedical Signal Processing and Control.

The alignment problem

Pre-trained backbones emit feature maps of different spatial and channel dimensions. You can't just concatenate them:

vgg_feat   = vgg16_base(x)        # (None, 7, 7, 512)
xcept_feat = xception_base(x)     # (None, 10, 10, 2048)
# concatenate([...]) -> shape mismatch!

The transition block

The fix is a small learnable block that projects both feature maps to a common shape before fusion:

from tensorflow.keras import layers, Model

def transition(t, filters, target_hw):
    t = layers.Conv2D(filters, 1, padding="same")(t)
    t = layers.BatchNormalization()(t)
    t = layers.Activation("relu")(t)
    return layers.Resizing(*target_hw)(t)

def fuse(vgg_feat, xcept_feat, target=(7, 7), filters=256):
    a = transition(vgg_feat,   filters, target)
    b = transition(xcept_feat, filters, target)
    return layers.Concatenate()([a, b])

Now the concatenation is well-defined, and a classifier head sits on top:

merged = fuse(vgg_feat, xcept_feat)
z = layers.GlobalAveragePooling2D()(merged)
z = layers.Dropout(0.4)(z)
out = layers.Dense(n_classes, activation="softmax")(z)
model = Model(inputs, out)

Did it help? Yes.

We compared the fused model against 21 single- and dual-backbone combinations:

Fusion isn't magic — it's complementarity. The gain comes from backbones that disagree in useful ways, not from stacking similar ones.

Practical notes

• Freeze the backbones first, train the head, then fine-tune end-to-end with a low learning rate.
• A 1×1 conv in the transition block controls channel count cheaply — don't skip the BatchNorm.
• Always validate against single-backbone baselines; fusion that doesn't beat its parts is just extra FLOPs.

Read the full paper from the Publications page.