Artificial Neural Networks¶
The Origin of Neural Networks¶
How does a biological neuron work? Brainstorm what you know.
Neural networks are inspired from the way biological neurons work. Despite this common origin, Artificial Neural Networks work very differently. As an analogy, think how birds fly and how planes fly.
What is Deep Learning?¶
The terms Artificial Neural Network (ANN) and Deep Learning usually refer to the same thing.
The first neural networks were experimented with in the 1950s. During their history they have resurfaced several times, only to become uninteresting again. Over the last decade, ANNs have started a triumphant rise to the most powerful type of machine learning we know today.
Warning
Training neural networks requires large datasets. Training from scratch with fewer than 10,000 data points is a bad idea most of the time.
How Artificial Neurons work¶
In a single neuron, inputs and bias are multiplied with weights and then transformed by an activation function:
The maths behind a neuron can be described by a single formula:
Does that equation look familiar to you?
The function f is called activation function. What do you get if you use a linear or sigmoid activation function?
Exercise: Set the weights manually¶
Set the coefficients \(w_1\), \(w_2\) and the bias \(w_0\) of the neuron in such a way that it implements a logical AND:
| x1 | x2 | desired output |
|----|----|----------------|
| 0 | 0 | 0 |
| 1 | 0 | 0 |
| 0 | 1 | 0 |
| 1 | 1 | 1 |
Could you also implement a logical OR?
Feed-Forward Networks¶
In Feed-Forward Neural Networks (also called Multi-Layer Perceptrons or MLPs), there are multiple neurons and two or more layers. The first layers are called hidden layers. The last one is called output layer. The information moves strictly from left to right:
Hypothetically, a network with two layers is large enough to learn anything. In practice, deep networks with multiple layers and other architectures are much more efficient.
Neurons on top of neurons extend the equation above by a the hidden layer:
Exercise: XOR in Python¶
The logical XOR is more challenging:
| x1 | x2 | desired output |
|----|----|----------------|
| 0 | 0 | 0 |
| 1 | 0 | 1 |
| 0 | 1 | 1 |
| 1 | 1 | 0 |
XOR is a non-linear problem that puzzled researchers for years half a century ago. You need a hidden layer to solve it. Feel free to try finding the weights on your own.
The code below implements the feed-forward network from the diagram:
input1 = float(input("enter first input : "))
input2 = float(input("enter second input : "))
w1 = [____, ____]
bias1 = ____
dot1 = input1 * w1[0] + input2 * w1[1] + bias1
out1 = 1.0 if dot1 > 0.5 else 0.0
w2 = [____, ____]
bias2 = ____
dot2 = input1 * w2[0] + input2 * w2[1] + bias2
out2 = 1.0 if dot2 > 0.5 else 0.0
w3 = [___, ____]
bias3 = ____
dot3 = out1 * w3[0] + out2 * w3[1] + bias3
out3 = 1.0 if dot3 > 0.5 else 0.0
print(out3)
Note
There is a solution where all three biases are zero.