The XOR (exclusive OR) problem is an example that highlights the limitations of a single-layer Perceptron. XOR is a logical operation that outputs true only when the inputs differ. Unlike the AND or OR functions, XOR cannot be separated by a straight line, making it a non-linearly separable problem.
A Perceptron fails to solve XOR because it can only create linear decision boundaries, and the problem is non-linear by nature.
Just add the following lines of code at the end of the last piece of code:
# XOR data points
X_xor = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y_xor = np.array([0, 1, 1, 0])# Attempted decision boundary
plt.scatter(X_xor[y_xor == 0][:, 0], X_xor[y_xor == 0][:, 1], color='red', label='Class 0')
plt.scatter(X_xor[y_xor == 1][:, 0], X_xor[y_xor == 1][:, 1], color='blue', label='Class 1')
plt.plot([0, 1], [0.5, 0.5], 'k--', label='Decision boundary (attempt)')
plt.xlabel('Input 1')
plt.ylabel('Input 2')
plt.legend()
plt.title('XOR Problem and Perceptron Limitation')
plt.show()
Hidden Layer Addition: The model now includes a hidden layer with two neurons and ReLU activation. An extra layer is the secret to allow the network to capture the non-linearity required to solve the XOR problem.
Activation Function: ReLU (relu) is used in the hidden layer, which helps in learning complex patterns, and the sigmoid activation function is used in the output layer to produce a probability output.
Optimizer and Loss: The optimizer is switched to adam for better performance, and the loss function remains binary_crossentropy, suitable for binary classification.
Training: The model is trained for 1000 epochs, which should be sufficient for convergence on the XOR problem.
Decision Boundary Plotting: The plot_decision_boundary function generates a grid of input values and predicts the output using the trained model. The predictions are then used to plot the decision boundary, showing how the model separates the XOR inputs.
By running this code, you’ll create a Multilayer Perceptron capable of solving the XOR problem, and you’ll be able to visualize the decision boundary that the model learns.
The decision boundary shows that our model is still far from ideal. There are many fine-tuning options, such as changing the learning rate, the number of epochs, and even the number of neurons in our neural network. I will leave this fun part for you to play around with!
This article explored the Perceptron, the foundation of neural networks! We started by implementing one from scratch and understanding its structure and limitations as well.
We implemented an extended model using TensorFlow to create a Multilayer Perceptron (MLP), demonstrating how additional layers and neurons can deal with non-linear classifications. Through this process, we emphasized the importance of model complexity, activation functions, and fine-tuning in order to generate better results.
Like what you see?
Be the first to comment