Multi-layer Perceptrons or MLPs in Python

In this article, I provide code in the form of a notebook that can be used to understand how Multi-Layer Perceptrons can be implemented in Python. The code also visualizes the decision boundary learnt by the model from the data. You can also play with the actual code on colab.

You can definitely implement a neural network from scratch in Python but in this article we make use of the MLP classifier implemented in the scikit-learn Python package. You can get an idea of the default parameters used, like the number of layers, activation function etc. by going to the documentation in their website. In the code that follows, we use a single hidden layer with 100 neurons. The activation function used is `relu` which is the default. The model optimizes the log-loss function using stochastic gradient descent. For the purpose of this tutorial we also make use of a synthetic dataset provided by the scikit-learn team and generated by the function `make_moons`. The train_test_split function provided by the scikit-learn package helps us to split the dataset into train and test. The default is to split the data into a 75:25 (train:test) ratio. Additionally, the function can split the data in a stratified manner based on the labels provided. This is helpful in dealing with class-imbalances in the data; the splitting is done in a way that the number of examples for each label are comparable. The MLP classifier model is initialized by specifying number of layers and the size of the layer and also the maximum number of iterations to run. The actual model training happens in the mlp.fit method. 

Finally we try to visualize what the model has learnt using the test set. In this case, the model learns a decision boundary based on the values of the two features that helps it to predict a new sample into either of the two classes. This decision boundary can be visualized by sampling a grid of points in the input feature space and obtaining the predicted labels (either 0 or 1). The predicted labels are then passed to the filled contour function in `matplotlib` which finds the contour line and fills it using the colour map specified and overplots this on top of the actual data using a transparency value. 

In summary these are things that we do in this tutorial:

• First we visualize the synthetic data that has two features and a label
• Then we initialize and train an MLP classifier on the training set
• Then we predict it on the test set and visualize the decision boundary found by the network

In [2]:

import numpy as np
import sklearn.datasets as ds
import matplotlib.pyplot as plt
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split

In [11]:

data, labels = ds.make_moons(n_samples=150, 
                             shuffle=True, 
                             noise=0.05, 
                             random_state=None)

In [12]:

def plot_data(data, labels):
    fig, ax = plt.subplots()
    ax.scatter(data[labels==0, 0], data[labels==0, 1], 
                   c='orange', s=40, label='oranges')
    ax.scatter(data[labels==1, 0], data[labels==1, 1], 
                   c='blue', s=40, label='blues')
    ax.set(xlabel='X',
           ylabel='Y',
           title='Moons')

In [17]:

X_train, X_test, y_train, y_test = train_test_split(data, labels, stratify=labels,
                                                    random_state=1)

In [18]:

plot_data(X_train, y_train)

In [19]:

plot_data(X_test, y_test)

In [20]:

mlp = MLPClassifier(hidden_layer_sizes=(100,), random_state=1, max_iter=3000)
clf = mlp.fit(X_train, y_train)

In [23]:

plot_data(X_test, y_test)

# create a mesh to plot in
h = .02  # step size in the mesh

x_min, x_max = data[:, 0].min() - 1, data[:, 0].max() + 1
y_min, y_max = data[:, 1].min() - 1, data[:, 1].max() + 1

xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))


Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
plt.contourf(xx, yy, Z, cmap=plt.cm.coolwarm, alpha=0.3);

References

MLP implementation using scikit-learn
https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html

Synthetic dataset using scikit-learn
https://python-course.eu/machine-learning/artificial-datasets-with-scikit-learn.php

Visualizing decision boundary
https://stackoverflow.com/questions/32921268/multilayer-perceptron-visualizing-decision-boundaries-2d-in-python