A configurable N layers neural network implemented in Python3. Trainer code uses back propagation.
The only dependency is NumPy, install it running:
pip3 install -r requirements.txt
Example to create a network of three layers with 784 neurons in the first layer, 20 in the second and 10 in the last one.
from neuralnetwork.network import NeuralNetwork
neural_network = NeuralNetwork([784, 20, 10])
from neuralnetwork.network import NeuralNetwork
from neuralnetwork.training import Trainer, TrainingDataSample
# training_data is a list of TrainingDataSample
# Each TrainingDataSample has the input value and its expected result
# See simple-example.py and digits-recognizer-traini.py
trainer = Trainer()
trainer.train(
network=neural_network,
training_data=training_data,
learning_step=1,
batch_iterations=20,
batch_size=50,
min_improvement_per_batch=0.000001,
max_batches_without_improvement=100,
cost_estimator_batch_size=5000
)
This will train the network using the stocastic gradient descent method
- Build a batch used for cost estimation, if
cost_estimator_batch_size
isNone
, use all the samples intraining_data
. - Estimate current cost for cost estimation batch
- Pick a batch of
batch_size
samples fromtraining_data
- Repeat
batch_iterations
:- Calculate cost function gradient for each sample
- Average the gradients
- Scale gradients by
learning_step
- Update networks weights and biases substracting the gradients
- Estimate updated cost for cost estimation batch
- The picked batch produced an "improvement" if
updated cost < min(previous costs) * (1-min_improvement_per_batch)
- Repeat until
max_batches_without_improvement
consecutive batches are processed without making improvement (as defined in 5).
result = neural_network.feedforward(data)
Since training can require a lot of time it's useful to be able to save and restore neural networks parameters.
- Saving network configuration:
from neuralnetwork.network import NeuralNetwork, serialize_neural_network
with open("network_setup.txt", "w") as f:
serialize_neural_network(neural_network, f)
- Loading network configuration:
from neuralnetwork.network import load_from_file
with open("network_setup.txt", "r") as f:
neural_network = load_from_file(f)
See example simple-example.py
. A points classifier in two teams depending on if x > y
.
This example trains and uses a neural network to recognize hand written digits.
I used the MNIST dataset to train and test the network. The dataset has 60000 digits for training and 10000 for testing. This example needs its inputs in CSV format, run the script in datasets/mnist/download_datasets.sh
to get them.
Since the inputs are 28x28 pixels = 784 pixels and the outputs are 10 possible digits the network must have 784 neurons in the first layer and 10 in the last one. I choose to have only one intermediate layer of 20 neurons.
-
digits-recognizer-train.py
trains the network and producesdigits_recognizer_setup.txt
with the trained network configuration. -
digits-recognizer-test.py
test each case in the testing dataset and outputs the success rate.
This example achieves a success rate of ~93%, that could be improved re training and trying a different number of hidden layers and neurons count in each layer.
I've used many sources of information and really recommend
-
Videos from 3Blue1Brown
-
Neural Networks and Deep Learning free online book, specially chapter 2. Book site