Radiographic Image Analysis

Deep Hybrid Learning for Radiographic Image Analysis

Deep Learning algorithms are quite powerful and effective for doing image analysis. Complex Deep Neural Network architectures like DenseNet-201 has been extremely powerful for classifying complicated medical images like radiographic (X-ray) images. As discussed in my medium article, the main reason why Deep Learning approaches are successful, is because deep learning removes the need of manual feature engineering and can learn features on it’s own. Feature engineering for complicated medical images like X-ray images are always complicated, so classical machine learning fails to out-perform deep learning methods.

But can we combine the benefits of both deep learning and machine learning and create a fusion network that might perform better than either of them?

Actually we can! And we will call this fusion network as Deep Hybrid Network and this approach to creating a fusion network will be termed as Deep Hybrid Learning (DHL)!

For more details of DHL, please take a look at the medium article. But overall, in DHL, we will use a Deep Neural Network to generate features from Radiographic Images and then perform the classification task by using a classical Machine Learning classifier.

As we can see from the above figure, this is a two step process. In the first step we will use a Deep Neural Network to extract features from the raw image dataset, thereby removing the need for manual feature engineering and in the next step, we will use these extracted features to run a machine learning classification algorithm, thereby removing the fully connected classification layers from a typical Deep Learning model.

How to implement a DHN?

In this section, we will use Keras, Tensorflow and Scikit-Learn to implement the concept of Deep Hybrid Network. The following code snippet shows us how to load the data :

opath_len = len(orig_path)
path = '../data'
train_image = []
added_features = []
for image_path in data['Path'][:20000]:
    #print('../data' + image_path[opath_len:])
    img = image.load_img(path  + image_path[opath_len:],target_size= 250,250,3))
    img = image.img_to_array(img)
    img = img/255
    train_image.append(img)

Now, we will see how our simple DNN looks like:

num_classes = 2 
## CNN model - Deep Neural Network code
model_LN = Sequential()
model_LN.add(Conv2D(filters=16, kernel_size=(5, 5), activation="relu", input_shape=(250,250,3)))
model_LN.add(MaxPooling2D(pool_size=(2, 2)))
model_LN.add(Dropout(0.25))
model_LN.add(Conv2D(filters=32, kernel_size=(5, 5), activation='relu'))
model_LN.add(MaxPooling2D(pool_size=(2, 2)))
model_LN.add(Dropout(0.25))
model_LN.add(Conv2D(filters=64, kernel_size=(5, 5), activation="relu"))
model_LN.add(MaxPooling2D(pool_size=(2, 2)))
model_LN.add(Dropout(0.25))
model_LN.add(Conv2D(filters=64, kernel_size=(5, 5), activation='relu'))
model_LN.add(MaxPooling2D(pool_size=(2, 2)))
model_LN.add(Dropout(0.25))
model_LN.add(Flatten())
model_LN.add(Dense(128, activation='relu'))
model_LN.add(Dropout(0.5))
model_LN.add(Dense(64, activation='relu'))
model_LN.add(Dropout(0.5))
#model.add(Dense(num_classes, activation='sigmoid'))
model_LN.add(Dense(num_classes, activation='softmax'))

model_LN.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=8, verbose=1, min_delta=1e-4)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=4, verbose=1, min_delta=1e-4)
callbacks_list = [early_stop, reduce_lr]

But for deep hybrid learning as we have said, we will not have the fully collected layers of the DNN model, so this is how it will look like :

num_classes = 2 
## Deep Hybrid Network's DNN layer which will be used for feature extraction
conv_base= Sequential()
conv_base.add(Conv2D(filters=16, kernel_size=(5, 5), activation="relu", input_shape=(250,250,3)))
conv_base.add(MaxPooling2D(pool_size=(2, 2)))
conv_base.add(Dropout(0.25))
conv_base.add(Conv2D(filters=32, kernel_size=(5, 5), activation='relu'))
conv_base.add(MaxPooling2D(pool_size=(2, 2)))
conv_base.add(Dropout(0.25))
conv_base.add(Conv2D(filters=64, kernel_size=(5, 5), activation="relu"))
conv_base.add(MaxPooling2D(pool_size=(2, 2)))
conv_base.add(Dropout(0.25))
conv_base.add(Conv2D(filters=64, kernel_size=(5, 5), activation='relu'))
conv_base.add(MaxPooling2D(pool_size=(2, 2)))
conv_base.add(Dropout(0.25))
conv_base.add(Flatten())

Also, DHN supports the concept of using pre-trained networks, for something like transfer learning. The below code shows how we can use a DenseNet model for this purpose :

from keras.applications import DenseNet121
conv_base = DenseNet121(include_top=False, weights='imagenet',input_shape=(250, 250, 3))

Now. let’s see how to extract features using a DNN.

from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(rescale=1./255)

# Feature Extraction
def extract_features(X_train, y_train, sample_count):
    batch_size = 32
    features = np.zeros(shape=(sample_count, 7, 7, 1024))
    labels = np.zeros(shape=(sample_count))
    datagen.fit(X_train)
    generator = datagen.flow(X_train, y_train, batch_size=32)
    i = 0
    for inputs_batch, labels_batch in generator:
        features_batch = conv_base.predict(inputs_batch)
        #print(features_batch.shape)
        features[i * batch_size : (i + 1) * batch_size] = features_batch
        labels[i * batch_size : (i + 1) * batch_size] = labels_batch
        i += 1
        if i * batch_size >= sample_count:
            break
    return features, labels

train_features, train_labels = extract_features(xTrain, yTrain, xTrain.shape[0])

Finally, we will use these features and use a Machine Learning Classification algorithm to fit on the extracted features. So, we will see how to apply Random Forest, AdaBoost and XGBoost on the extracted features.

Random Forest :

from sklearn.ensemble import RandomForestClassifier
hybrid_model_RF = RandomForestClassifier()
hybrid_model_RF.fit(xTrain, yTrain)
predicted_test_labels = hybrid_model_RF.predict(xTest)

AdaBoost :

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
hybrid_model_AB = AdaBoostClassifier(
    DecisionTreeClassifier(max_depth=3),
    n_estimators=10
)
hybrid_model_AB.fit(xTrain, yTrain)
predicted_test_labels = hybrid_model_AB.predict(xTest)

XGBoost :

from xgboost import XGBClassifier
hybrid_model_XG = XGBClassifier()
hybrid_model_XG.fit(xTrain, yTrain)
predicted_test_labels = hybrid_model_XG.predict(xTest)

And, finally as discussed in my medium article, this is how the final model architecture looks like :

In the medium article, I have talked about the model evaluation part. Below is the code to perform model evaluation using the various metrics discussed in the artcile

import numpy as np
from sklearn import metrics
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelBinarizer
import seaborn as sns

def multiclass_roc_auc_score(y_test, y_pred, average="macro"):
        lb = LabelBinarizer()
        lb.fit(y_test)
        y_test = lb.transform(y_test)
        y_pred = lb.transform(y_pred)
        return metrics.roc_auc_score(y_test, y_pred, average=average)
        
def evaluate_model_performance_valid_data(y_orig, y_pred):
        accuracy = metrics.accuracy_score(y_orig, y_pred)
        print('Accuracy is:',accuracy)

        # For precision of each class individually use average= None
        precision = metrics.precision_score(y_orig, y_pred, average='macro')
        print('Precision is:',precision)

        # For recall of each class individually use average= None
        recall = metrics.recall_score(y_orig, y_pred, average='macro')
        print('Recall is:',recall)

        # For recall of each class individually use average= None
        f1_score_value = metrics.f1_score(y_orig, y_pred, average='macro')
        print('F1 Score is:',f1_score_value)
        
        # AUC Scores
        auc_score = multiclass_roc_auc_score(y_orig, y_pred)
        print('AUC score is:', auc_score)

        cm = metrics.confusion_matrix(y_orig, y_pred)
        print(cm)

        plt.figure(figsize=(5,5))
        sns.heatmap(cm, annot=True, fmt=".2f", linewidths=.5, square = True, cmap = 'Blues');
        plt.ylabel('Actual label');
        plt.xlabel('Predicted label');
        all_sample_title = 'AUC Score: {:.2f}'.format(auc_score)
        plt.title(all_sample_title, size = 12)

And this is how the result will look like:

I have seen remarkable results so far using this approach. I would encourage all the readers to try this out and let me know if they are getting in improvement in the performance both in terms of execution time and as well as in terms model performance.

With this we come to the end of this article. Please comment or connect with me, with your feedback or any suggestions!