Aka artificial neural network..
A particularly useful way of representing nonlinear functions, for problems in Machine learning. It is a very good model for many problems, and learning algorithms produce very good results with them. In particular deep learning (which uses ANNs with many layers). It is a nonlinear classifier, and Regression analysis model.
But what *is* a Neural Network? -- Deep learning, Part 1
Hugo Larochelle class videos (website). Andrew Ng intro. NN. Learning parameters in a NN is generally a non-convex optimization problem, which makes it very hard to reach global optima. – book
nice visualization – very cool interactive visualization library in javascript –> https://tensorspace.org/
Definition
Neuron has:
1) inputs
2) weight vectors, that multiplies the input vector or activation vector of hidden layers.
3) bias, that is added to result
4) Activation function takes as argument the result of the above (called pre-activation or input activation)
5) The result (called activation) may be the input of other neurons in the next layer, in a multilayer feedforward neural network.
6) The activation of the last layer, is the output
Overall... we are multiplying by matrices and applying simple nonlinear function
See Neural network theory, Mathematical modelling of neural networks, for more on the theory
Learning by optimization
Learning by minimizing cost function (see Learning theory)
Training neural networks - optimization. There are several global optima, and plateaus. Uses Gradient descent, in particular SGD.
An efficient algorithm to compute the gradients of the loss function for (SGD) w.r.t. the ANN's parameters is Backpropagation.
see more at Learning theory
Parameter initialization for NNs
Neural networks [2.9] : Training neural networks - parameter initialization
Model selection for neural networks
Neural networks [2.10] : Training neural networks - model selection. How to set the hyperparameters. Can use Cross-validation.
Types of neural networks
- Feedforward neural network (basic)
- Convolutional neural network. Good for image recognition for e.g.
- Recurrent. Good for sequences in time
- Long-Short term memory NN.
- Spiking neural network
- Residual neural network
- Boltzmann machine
- Autoencoder
- Deep belief network
- Generative adversarial network
- Hopfield network
- Binarized neural network
- Extreme learning machine
Many models in Machine learning can be seen as neural networks
Early video that created about TTS using ANNs (NetTalk), see Speech synthesis
A Neural Network in 11 Lines of Python
More models, and generalizations
Backpropagation, temporal networks, etc..
Visualizing and Understanding Deep Neural Networks by Matt Zeiler
Two Minute Papers - Estimating Matrix Rank With Neural Networks
Physical implementations:
Chemical implementations of neural networks and Turing machines
More
Layerless neural networks? See Chico Calmagro's work with Ard Louis.
On the complex backpropagation algorithm
Neural networks for control systems—A survey
Genetic deep neural networks using different activation functions for financial data mining
Structure Discovery of Deep Neural Network Based on Evolutionary Algorithms
Genetic algorithms for evolving deep neural networks
Implementation of Evolutionary Algorithms for Deep Architectures
See ideas here: Idea for neural network for chemical synethesis and manufacturing etc. Facebook post: https://www.facebook.com/guillermovalleperez/posts/10153853693416223?
Statistical mechanics of neural networks
Neural networks and physical systems with emergent collective computational abilities
Spin-glass models of neural networks