Perceptron & Neural Networks

1
Perceptron and Neural Networks
Shaik Nagur Shareef
Dept. of CSE
Vignan’s University
Output Values
Input Signals (External Stimuli)

Contents
Introduction
Neural Networks
Perceptron and Examples
Types of NN
Applications
January 20, 2019 2Shaik Nagur Shareef

Human Information processing system
 Highly complex, nonlinear, and parallel
computer.
 Has the capability to organize its structural
constituents, known as neurons, to perform
certain computations.
 Pattern recognition, Perception, and Motor
control.
 Many times faster than the fastest digital
computer in existence today
 Neurons are information-processing units in
the human brain.
January 20, 2019 3Shaik Nagur Shareef
Interconnected Neurons forms a neural (nerve) net

Human Nervous System
 Viewed as a Three-stage system.
 Central to the system is the brain, represented by the neural (nerve) net, which
continually receives information, perceives it, and makes appropriate decisions.
 The arrows pointing from pointing from left to right indicate the forward transmission.
 The arrows pointing from right to left signify the presence of feedback in the system.
 The receptors convert stimuli from the human body or the external environment into
electrical impulses that convey information to the neural net.
 The effectors convert electrical impulses generated by the neural net into discernible
responses as system outputs.
January 20, 2019 Shaik Nagur Shareef 4

Machine Information Processing System
 Neural Networks are made of Artificial Neurons,
 A Neural Network is a machine that is designed to model the way in which the brain
performs a particular task or function of interest.
 The network is usually implemented by using electronic components or is simulated in
software on a digital computer.
 Neural Networks perform useful computations through a process of learning.

Biological Neuron vs Machine Neuron

Neural Network
 A Neural Network is a massively parallel distributed processor made up of simple
processing units that has a natural propensity for storing experiential knowledge and
making it available for use.
 It resembles the brain in two respects:
1. Knowledge is acquired by the network from its environment through a learning process.
2. Interneuron connection strengths, known as synaptic weights, are used to store the
acquired knowledge.
 The procedure used to perform the learning process is called a learning algorithm, the
function of which is to modify the synaptic weights of the network in an orderly fashion
to attain a desired design objective.
 The modification of synaptic weights provides the traditional method for the design of
neural networks.

Neural Network

Artificial Neuron - [McPi43]
 In 1943, Warren McCulloch and Walter Pitts introduced one of the first artificial
neurons [McPi43].
 The main feature of their neuron model is that a weighted sum of input signals is
compared to a threshold to determine the neuron output.
 When the sum is greater than or equal to the threshold, the output is 1.
 When the sum is less than the threshold, the output is 0.
 Networks of these neurons in principle, compute any arithmetic or logical function.
 The parameters of [McPi43] networks had to be designed, as no training method was
available.

Perceptron- [Rose58]
 In the late 1950s, Frank Rosenblatt and several other researchers developed a class of
neural networks called perceptrons.
 The neurons in these networks were similar to those of McCulloch and Pitts.
 Rosenblatt's key contribution was the introduction of a learning rule for training
perceptron networks to solve pattern recognition problems [Rose58].
 He proved that his learning rule will always converge to the correct network weights, if
weights exist that solve the problem.
 Learning was simple and automatic.
 Examples of proper behavior were presented to the network, which learned from its
mistakes.
 The perceptron could even learn when initialized with random values for its weights
and biases.

Learning Rule
 It is a Procedure for modifying the weights and biases of a network.
 This procedure may also be referred to as a training algorithm.
 The purpose of the learning rule is to train the network to perform some task.
 There are many types of neural network learning rules.
 They fall into three broad categories:
1. Supervised learning
2. Unsupervised learning
3. Reinforcement (or graded) learning

Supervised Learning
 In supervised learning, the learning rule is provided with a set of examples (the training
set) of proper network behavior:
 Here pq is an input to the network and tq is the corresponding correct (target) output.
 As the inputs are applied to the network, the network outputs are compared to the
targets.
 The learning rule is then used to adjust the weights and biases of the network in order to
move the network outputs closer to the targets.
 The perceptron learning rule falls in this supervised learning category.

Unsupervised Learning
 In unsupervised learning, the weights and biases are modified in response to network
inputs only.
 There are no target outputs available.
 At first glance this might seem to be impractical.
 How can you train a network if you don't know what it is supposed to do?
 Most of these algorithms perform some kind of clustering operation.
 They learn to categorize the input patterns into a finite number of classes.
 This is especially useful in such applications as vector quantization.

Reinforcement Learning
 Reinforcement learning is similar to supervised learning, except that, instead of being
provided with the correct output for each network input, the algorithm is only given a
grade.
 The grade (or score) is a measure of the network performance over some sequence of
inputs.
 This type of learning is currently much less common than supervised learning.
 It appears to be most suited to control system applications

Perceptron Architecture

Procedure
 consider the network weight matrix



Activation Function(hardlim Function)
Step Function Sigmoid Function

Single-Neuron Perceptron-Example

 Assign Values to weights and bias



 Therefore, the network output will be 1 for the region above and to the
right of the decision boundary.

Logic Function: AND gate



Weight Update Rule

Training Algorithms
 Adjust neural network weights to map inputs to outputs.
 Use a set of sample patterns where the desired output (given the inputs presented) is
known.
 The purpose is to learn to generalize
Recognize features which are common to good and bad exemplars

Training: Key Terms
 Epoch: Presentation of the entire training set to the neural network.
 In the case of the AND function an epoch consists of four sets of inputs being presented
to the network (i.e. [0,0], [0,1], [1,0], [1,1])
 Error: The error value is the amount by which the value output by the network differs
from the target value.
 For example, if we required the network to output 0 and it output a 1, then Error = -1
 Online training: Update weights after each sample
 Offline (batch training): Compute error over all samples
Then update weights
 Training: Backpropagation procedure
Gradient descent strategy (usual problems)
 Prediction: Compute outputs based on input vector & weights

Gradient Descent Concept
 Error: Sum of squares error of inputs with current weights
 Compute rate of change of error w.r.t each weight
Which weights have greatest effect on error?
Effectively, partial derivatives of error w.r.t weights
In turn, depend on other weights => chain rule
 E = G(w)
Error as function of weights
 Find rate of change of error
Follow steepest rate of change
Change weights so that error is minimized

Gradient Descent Algorithm
Gradient-Descent(training_examples, )
Each training example is a pair of the form <(x1,…xn),t> where (x1,…,xn) is the vector
of input values, and t is the target output value,  is the learning rate (e.g. 0.1)
 Initialize each wi to some small random value
 Until the termination condition is met, Do
Initialize each wi to zero
For each <(x1,…xn),t> in training_examples Do
Input the instance (x1,…,xn) to the linear unit and compute the output o
For each linear unit weight wi Do
 wi= wi +  (t-o) xi
For each linear unit weight wi Do
wi=wi+wi

Gradient Descent Algorithm
(w1,w2)
(w1+w1,w2 +w2)

Back-Propagation
 A training procedure which allows multi-layer feedforward Neural Networks to be
trained;
 Can theoretically perform “any” input-output mapping;
 Can learn to solve linearly inseparable problems.
 For feed-forward networks:
A continuous function can be differentiated allowing gradient-descent.
Back-propagation is an example of a gradient-descent technique.
jiw  kjw 
iy jyi j k
j k

Back-Propagation Algorithm
 Initialize each wi to some small random value
 Until the termination condition is met, Do
For each training example <(x1,…xn),t> Do
Input the instance (x1,…,xn) to the network and compute the network outputs ok
For each output unit k
k=ok(1-ok)(tk-ok)
For each hidden unit h
h=oh(1-oh) k wh,k k
For each network weight w,j Do
wi,j=wi,j+wi,j where
wi,j=  j xi,j

Types of Neural Networks
Standard NN Recurrent NNConvolutional NN

Benefits of Neural Networks
 Nonlinearity: An artificial neuron can be linear or nonlinear.
 Input–Output Mapping: A popular paradigm of learning, called learning with a
teacher, or supervised learning, involves modification of the synaptic weights of a
neural network by applying a set of labelled training examples, or task examples.
 Adaptivity: Neural networks have a built-in capability to adapt their synaptic weights
to changes in the surrounding environment.
 Evidential Response: In the context of pattern classification, a neural network can be
designed to provide information not only about which particular pattern to select, but
also about the confidence in the decision made.
 Fault Tolerance: A neural network, implemented in hardware form, has the potential
to be inherently fault tolerant, or capable of robust computation, in the sense that its
performance degrades gracefully under adverse operating conditions.

Applications
Natural language Processing
Optical Character Recognition
Speech recognition
Neural Machine Translation
Video Classification
Emotion Recognition
Face Recognition
Object Detection
Image Classification

Perceptron & Neural Networks

More Related Content

What's hot

Similar to Perceptron & Neural Networks

More from NAGUR SHAREEF SHAIK

Recently uploaded

Perceptron & Neural Networks