Artificial neural network

Artificial neural network

• 1.
  ArtificialArtificial neural networkneural network Yük.Blig. Müh. Mustafa Aadel MashjalYük. Blig. Müh. Mustafa Aadel Mashjal 158229001009158229001009
• 2.
  Background - Neural Networkscan be : - BiologicalBiological models - ArtificialArtificial models - We wish to produce artificial systems capable of complex calculation similar to the human brain.
• 3.
  describe the transmissionofdescribe the transmission of information and some main ideasinformation and some main ideas • The brain is consist of a mass of interconnected neurons – each neuron is connected to many other neurons • Neurons transmit signals to each other • Whether a signal is sent, depends on the strength of the bond (synapse) between two neurons
• 4.
  How Does theBrain Work ? (1) NEURON - It is the cell that performs information processing in the brain. - Nervous tissue its consist of neurons, which receive and transmit impulses
• 5.
  Each consists of: SOMA, DENDRITES, AXON, and SYNAPSE. How Does the Brain Work ? (2)
• 6.
  Brain vs. DigitalComputers (1) - Computers require hundreds of cycles to simulate a firing of a neuron. - The brain can fire all the neurons in a single step. ParallelismParallelism - Serial computers require billions of cycles to perform some tasks but the brain takes less than a second. e.g. Face Recognition
• 7.
  Definition of NeuralNetwork A Neural Network is a system that consist of many simple processing elements operating in parallel which can achieve, store, and use experimental knowledge.
• 8.
  What isWhat is ArtificialArtificial NeuralNeural Network?Network?
• 9.
  Neurons vs. Units(1) - Each element of NN is a node called unit. - Units are connected by links. -Each link has a numeric weight.
• 10.
  Neurons vs units(2) • ANNs incorporate the two fundamental components of biological neural nets: 1. Neurones (nodes) 2. Synapses (weights)
• 11.
  1- A setof connecting links, each link characterized by a weight: W1, W2, …, Wm 2- An adder function (linear combiner) which computes the weighted sum of the inputs: 3- Activation function (squashing function) for limiting the amplitude of the output of the neuron. ∑= = m 1 jjxwnetj j Structure of a nodeStructure of a node
• 12.
  ActivationActivation Functions - Usedifferent functions to obtain different models. - 3 most common choices : 1) Step function 2) Sign function 3) Sigmoid function - An output of 1 represents firing of a neuron down the axon.
• 13.
  Step Function Perceptrons
• 14.
  3 Activation Functions
• 15.
  Feed-Forward Neural Network Architectures Thefeed-forward neural network was the first and most simple type of artificial neural network devised. In this network the information moves in only one direction—forward: From the input nodes data goes through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network. • Two different classes of network architectures – single-layer feed-forward neurons are organized – multi-layer feed-forward in acyclic layers • The architecture of a neural network is linked with the learning algorithm used to train
• 16.
  Single Layer Feed-forward Thesimplest kind of neural network is a single-layer perceptron network, which consists of a single layer of output nodes; the inputs are fed directly to the outputs via a series of weights. In this way it can be considered the simplest kind of feed- forward network. The sum of the products of the weights and the inputs is calculated in each node, and if the value is above some threshold the neuron fires and takes the activated value; otherwise it takes the deactivated value. Neurons with this kind of activation function are also called artificial neurons or linear threshold units.
• 18.
  This class ofnetworks consists of multiple layers of computational units, usually interconnected in a feed-forward way. Each neuron in one layer has directed connections to the neurons of the next layer. In many applications the units of these networks apply a sigmoid function as an activation function Multi Layer Feed-forward
• 19.
  Supervised Learning Algorithm •The learning algorithm would fall under this category if the desired output for the network is also provided with the input while training the network. By providing the neural network with both an input and output pair it is possible to calculate an error based on it's target output and actual output. It can then use that error to make corrections to the network by updating it's weights. • Single Layer neural network can be trained by a simple learning algorithm that is usually called the Delta rule. • Multi Layer neural network can be trained by a learning algorithm that is usually called the Bakpropgation.
• 20.
  Delta rule  Thedelta rule is specialized version of backpropagation's learning rule, for use with single layer neural networks.  It calculates the errors between calculated output and sample output data, and uses this to create a modification to the weights, thus implementing a form of gradient descent.  t (target) , o (output) Mu (learning rate) , beta (error) , x (input) collect old weight with the new modification weight and thus changing the networks weight
• 21.
  Unsupervised Learning Algorithm Inthis paradigm the neural network is only given a set of inputs and it's the neural network's responsibility to find some kind of pattern within the inputs provided without any external aid.
• 22.
  Hebb Rule Hebb methodis used in the learning of networks that use Unsupervised Learning , and the weight is modify by the equation: Mu (learning rate) , o (output) , x (input) Then collect the output with the initial weight :
• 23.
  Introduction toIntroduction to BackpropagationBackpropagation -In 1969 a method for learning in multi-layer network, Backpropagation, was invented by Bryson and Ho. - Backpropagation is generalization of the delta rule to multi-layered feedforward networks. - Backpropagation is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent (a first-order repeated optimization algorithm). It calculates the gradient of a loss function with respect to all the weights in the network, so that the gradient is fed to the optimization method which in turn uses it to update the weights, in an attempt to minimize the loss function.
• 24.
  Backpropagation LearningBackpropagation Learning DetailsDetails •Use gradient descent to minimize the error – propagate deltas to adjust for errors backward from outputs to hidden layers to inputs
• 25.
  Backpropagation Algorithm –Main Idea – error in hidden layers The ideas of the algorithm can be summarized as follows : 1. Computes the error term for the output units using the observed error. 2. From output layer, repeat - propagating the error term back to the previous layer and - updating the weights between the two layers until the earliest hidden layer is reached.
• 26.
  Forward Propagation ofActivityForward Propagation of Activity • Step 1: Initialise weights at random, choose a learning rate η • Until network is trained: • For each training example i.e. input pattern and target output(s): • Step 2: Do forward pass through net (with fixed weights) to produce output(s) – i.e., in Forward Direction, layer by layer: • Inputs applied • Multiplied by weights • Summed • ‘Squashed’ by sigmoid activation function • Output passed to each neuron in next layer – Repeat above until network output(s) produced
• 27.
  XOR- exampleXOR- example
• 28.
  Step 3. Back-Step3. Back- propagation ofpropagation of errorerror Backprop output layer
• 32.
  bias neuron ininput layer Bias Neurons inBias Neurons in BackpropBackpropagationagation LearningLearning
• 33.
  Least-Mean-Square (LMS)Least-Mean-Square (LMS) AlgorithmAlgorithm Leastmean squares (LMS) algorithms are a class of adaptive filter used to simulate a required filter by finding the difference between the desired and the actual signal. It was invented in 1960 by Stanford University professor Bernard Widrow .
• 37.
  Examples andExamples and ApplicationsApplications ofANNof ANN
• 38.
  Neural Network inPractice NNs are used for classification and function approximation or mapping problems which are: - Tolerant of some imprecision. - Have lots of training data available. - Hard and fast rules cannot easily be applied.
• 39.
  NETalk (1987) • Mappingcharacter strings into phonemes so they can be pronounced by a computer • Neural network trained how to pronounce each letter in a word in a sentence, given the three letters before and three letters after it in a window • Output was the correct phoneme • Results – 95% accuracy on the training data – 78% accuracy on the test set
• 40.
  Other Examples • Neurogammon(Tesauro & Sejnowski, 1989) – Backgammon learning program • Speech Recognition (Waibel, 1989) • Character Recognition (LeCun et al., 1989) • Face Recognition (Mitchell)
• 41.
  ALVINNALVINN • Steer avan down the road – 2-layer feedforward • using backpropagation for learning – Raw input is 480 x 512 pixel image 15x per sec – Color image preprocessed into 960 input units – 4 hidden units – 30 output units, each is a steering direction
• 42.
  Neural Network Approaches ALVINN- Autonomous Land Vehicle In a Neural Network
• 43.
  Learning on-the- fly • ALVINNlearned as the vehicle traveled – initially by observing a human driving – learns from its own driving by watching for future corrections – never saw bad driving • didn’t know what was dangerous, NOT correct • computes alternate views of the road (rotations, shifts, and fill-ins) to use as “bad” examples – keeps a buffer pool of 200 pretty old examples to avoid overfitting to only the most recent images
• 44.
  Feed-forward vs. Interactive Nets •Feed-forward – activation propagates in one direction – We usually focus on this • Interactive – activation propagates forward & backwards – propagation continues until equilibrium is reached in the network – We do not discuss these networks here, complex training. May be unstable.
• 45.
  Ways of learningwith an ANN • Add nodes & connections • Subtract nodes & connections • Modify connection weights – current focus – can simulate first two • I/O pairs: – given the inputs, what should the output be? [“typical” learning problem]
• 46.
  More Neural NetworkMoreNeural Network ApplicationsApplications - May provide a model for massive parallel computation. - More successful approach of “parallelizing” traditional serial algorithms. - Can compute any computable function. - Can do everything a normal digital computer can do. - Can do even more under some impractical assumptions.
• 47.
  Neural Network Approachesto driving - Developed in 1993. - Performs driving with Neural Networks. - An intelligent VLSI image sensor for road following. - Learns to filter out image details not relevant to driving. Hidden layer Output units Input units •Use special hardware •ASIC •FPGA •analog
• 48.
  Neural Network Approaches HiddenUnits Output unitsInput Array
• 49.
  Actual Products Available ex1.Enterprise Miner:ex1. Enterprise Miner: - Single multi-layered feed-forward neural networks. - Provides business solutions for data mining. ex2. Nestor:ex2. Nestor: - Uses Nestor Learning System (NLS). - Several multi-layered feed-forward neural networks. - Intel has made such a chip - NE1000 in VLSI technology.
• 50.
  Ex1. Software tool- Enterprise Miner - Based on SEMMA (Sample, Explore, Modify, Model, Access) methodology. - Statistical tools include : Clustering, decision trees, linear and logistic regression and neural networks. - Data preparation tools include : Outliner detection, variable transformation, random sampling, and partition of data sets (into training, testing and validation data sets).
• 51.
  Ex 2. HardwareTool - Nestor - With low connectivity within each layer. - Minimized connectivity within each layer results in rapid training and efficient memory utilization, ideal for VLSI. - Composed of multiple neural networks, each specializing in a subset of information about the input patterns. - Real time operation without the need of special computers or custom hardware DSP platforms •Software exists.
• 52.
  SummarySummary - Neural networkis a computational model that simulate some properties of the human brain. - The connections and nature of units determine the behavior of a neural network. - Perceptrons are feed-forward networks that can only represent linearly separable functions.
• 53.
  Summary - Given enoughunits, any function can be represented by Multi-layer feed-forward networks. - Backpropagation learning works on multi-layer feed-forward networks. - Neural Networks are widely used in developing artificial learning systems.
• 54.
  ReferencesReferences - Russel, S.and P. Norvig (1995). Artificial Intelligence - A Modern Approach. Upper Saddle River, NJ, Prentice Hall. - Sarle, W.S., ed. (1997), Neural Network FAQ, part 1 of 7: Introduction, periodic posting to the Usenet newsgroup comp.ai.neural-nets, URL: ftp://ftp.sas.com/pub/neural/FAQ.html
• 55.
  Eddy Li Eric Wong MartinHo Kitty Wong Sources