Neural network final NWU 4.3 Graphics Course

Neural Network: A Brief Overview
Presented by
Md. Mahedi Hasan
Lecturer
Dept. of CSE

Outline
 Introduction
 Background
 How the human brain works
 A Neuron Model
 A Simple Neuron
 Pattern Recognition example
 A Complicated Perceptron

Outline Continued
 Different types of Neural Networks
 Network Layers and Structure
 Training a Neural Network
 Learning process
 Neural Networks in use
 Use of Neural Networks in C.A.I.S. project
 Conclusion

Introduction
 What are Neural Networks?
– Neural networks are a new method of programming
computers.
– They are exceptionally good at performing pattern
recognition and other tasks that are very difficult to program
using conventional techniques.
– Programs that employ neural nets are also capable of
learning on their own and adapting to changing conditions.

Background
 An Artificial Neural Network (ANN) is an information processing
paradigm(a typical example or pattern of something; a model.)
that is inspired by the biological nervous systems, such as the
human brain’s information processing mechanism.
 The key element of this paradigm is the novel structure of the
information processing system. It is composed of a large
number of highly interconnected processing elements
(neurons) working in unison to solve specific problems. NNs,
like people, learn by example.
 An NN is configured for a specific application, such as pattern
recognition or data classification, through a learning process.
Learning in biological systems involves adjustments to the
synaptic connections that exist between the neurons. This is
true of NNs as well.

How the Human Brain learns
 In the human brain, a typical neuron collects signals from others
through a host of fine structures called dendrites.
 The neuron sends out spikes of electrical activity through a long, thin
stand known as an axon, which splits into thousands of branches.
 At the end of each branch, a structure called a synapse converts the
activity from the axon into electrical effects that inhibit or excite
activity in the connected neurons.

A Neuron Model
 When a neuron receives excitatory input that is sufficiently large
compared with its inhibitory input, it sends a spike of electrical activity
down its axon. Learning occurs by changing the effectiveness of the
synapses so that the influence of one neuron on another changes.
 We conduct these neural networks by first trying to deduce the essential
features of neurons and their interconnections.
 We then typically program a computer to simulate these features.

A Simple Neuron
 An artificial neuron is a device with many inputs and
one output.
 The neuron has two modes of operation;
 the training mode and
 the using mode.

A Simple Neuron (Cont.)
 In the training mode, the neuron can be trained to fire (or not),
for particular input patterns.
 In the using mode, when a taught input pattern is detected at
the input, its associated output becomes the current output. If
the input pattern does not belong in the taught list of input
patterns, the firing rule is used to determine whether to fire or
not.
 The firing rule is an important concept in neural networks and
accounts for their high flexibility. A firing rule determines how
one calculates whether a neuron should fire for any input
pattern. It relates to all the input patterns, not only the ones on
which the node was trained on previously.

What is Pattern??
 “A pattern is the opposite of a chaos; it is an
entity vaguely defined, that could be given a
name.”
 “The assignment of a physical object or
event to one of several pre-specified
categories” -- Duda & Hart
 A pattern is an object, process or event

Pattern Recognition
 An important application of neural networks is pattern
recognition. Pattern recognition can be implemented by using a
feed-forward neural network that has been trained accordingly.
 During training, the network is trained to associate outputs with
input patterns.
 When the network is used, it identifies the input pattern and
tries to output the associated output pattern.
 The power of neural networks comes to life when a pattern that
has no output associated with it, is given as an input.
 In this case, the network gives the output that corresponds to a
taught input pattern that is least different from the given pattern.

Pattern Recognition (cont.)
 Suppose a network is trained to recognize
the patterns T and H. The associated
patterns are all black and all white
respectively as shown above.

Since the input pattern looks more like a ‘T’, when
the network classifies it, it sees the input closely
resembling ‘T’ and outputs the pattern that
represents a ‘T’.

The input pattern here closely resembles ‘H’
with a slight difference. The network in this
case classifies it as an ‘H’ and outputs the
pattern representing an ‘H’.

 Here the top row is 2 errors away from a ‘T’ and 3 errors away
from an H. So the top output is a black.
 The middle row is 1 error away from both T and H, so the
output is random.
 The bottom row is 1 error away from T and 2 away from H.
Therefore the output is black.
 Since the input resembles a ‘T’ more than an ‘H’ the output of
the network is in favor of a ‘T’.

A Complicated Perceptron
 A more sophisticated Neuron is know as the McCulloch and
Pitts model (MCP).
 The difference is that in the MCP model, the inputs are
weighted and the effect that each input has at decision making,
is dependent on the weight of the particular input.
 The weight of the input is a number which is multiplied with the
input to give the weighted input.

A Complicated Perceptron (cont.)
 The weighted inputs are then added together and if
they exceed a pre-set threshold value, the
perceptron / neuron fires.
 Otherwise it will not fire and the inputs tied to that
perceptron will not have any effect on the decision
making.
 In mathematical terms, the neuron fires if and only if;
X1W1 + X2W2 + X3W3 + ... > T

A Complicated Perceptron
 The MCP neuron has the ability to adapt to a
particular situation by changing its weights and/or
threshold.
 Various algorithms exist that cause the neuron to
'adapt'; the most used ones are the Delta rule and
the back error propagation.

Different types of Neural Networks
 Feed-forward networks
– Feed-forward NNs allow signals to travel one way
only; from input to output. There is no feedback
(loops) i.e. the output of any layer does not affect
that same layer.
– Feed-forward NNs tend to be straight forward
networks that associate inputs with outputs. They
are extensively used in pattern recognition.
– This type of organization is also referred to as
bottom-up or top-down.

Continued
 Feedback networks
– Feedback networks can have signals traveling in both
directions by introducing loops in the network.
– Feedback networks are dynamic; their 'state' is changing
continuously until they reach an equilibrium point.
– They remain at the equilibrium point until the input changes
and a new equilibrium needs to be found.
– Feedback architectures are also referred to as interactive or
recurrent, although the latter term is often used to denote
feedback connections in single-layer organizations.

Diagram of an NN
Fig: A simple Neural Network

Network Layers
 Input Layer - The activity of the input units
represents the raw information that is fed into the
network.
 Hidden Layer - The activity of each hidden unit is
determined by the activities of the input units and the
weights on the connections between the input and
the hidden units.
 Output Layer - The behavior of the output units
depends on the activity of the hidden units and the
weights between the hidden and output units.

Continued
 This simple type of network is interesting because
the hidden units are free to construct their own
representations of the input.
 The weights between the input and hidden units
determine when each hidden unit is active, and so by
modifying these weights, a hidden unit can choose
what it represents.

Network Parameters
 How are the weights initialized?
 How many hidden layers and how many
neurons?
 How many examples in the training set?

Weights
 In general, initial weights are randomly
chosen, with typical values between -1.0 and
1.0 or -0.5 and 0.5.
 There are two types of NNs. The first type is
known as
– Fixed Networks – where the weights are fixed
– Adaptive Networks – where the weights are
changed to reduce prediction error.

Size of Training Data
 Rule of thumb:
– the number of training examples should be at least five to
ten times the number of weights of the network.
 Other rule:
a)-(1
|W|
N >
|W|= number of weights
a = expected accuracy on
test set

Training Basics
 The most basic method of training a neural
network is trial and error.
 If the network isn't behaving the way it
should, change the weighting of a random
link by a random amount. If the accuracy of
the network declines, undo the change and
make a different one.
 It takes time, but the trial and error method
does produce results.

Training: Backprop algorithm
 The Backprop algorithm searches for weight values that
minimize the total error of the network over the set of training
examples (training set).
 Backprop consists of the repeated application of the following
two passes:
– Forward pass: in this step the network is activated on one
example and the error of (each neuron of) the output layer
is computed.
– Backward pass: in this step the network error is used for
updating the weights. Starting at the output layer, the error
is propagated backwards through the network, layer by
layer. This is done by recursively computing the local
gradient of each neuron.

Back Propagation
 Back-propagation training algorithm
 Backprop adjusts the weights of the NN in order to
minimize the network total mean squared error.
Network activation
Forward Step
Error
propagation
Backward Step

The Learning Process
 The memorization of patterns and the
subsequent response of the network can be
categorized into two general paradigms:
– Associative mapping
– Regularity detection

The Learning Process (cont.)
 Recall: Adaptive networks are NNs that allow
the change of weights in its connections.
 The learning methods can be classified in
two categories:
– Supervised Learning
– Unsupervised Learning

Supervised Learning
 Supervised learning which incorporates an external
teacher, so that each output unit is told what its
desired response to input signals ought to be.
 An important issue concerning supervised learning is
the problem of error convergence, ie the
minimization of error between the desired and
computed unit values.
 The aim is to determine a set of weights which
minimizes the error. One well-known method, which
is common to many learning paradigms is the least
mean square (LMS) convergence.

Supervised Learning
 In this sort of learning, the human teacher’s
experience is used to tell the NN which outputs are
correct and which are not.
 This does not mean that a human teacher needs to
be present at all times, only the correct
classifications gathered from the human teacher on
a domain needs to be present.
 The network then learns from its error, that is, it
changes its weight to reduce its prediction error.

Unsupervised Learning
 Unsupervised learning uses no external teacher
and is based upon only local information. It is also
referred to as self-organization, in the sense that it
self-organizes data presented to the network and
detects their emergent collective properties.
 The network is then used to construct clusters of
similar patterns.
 This is particularly useful is domains were a
instances are checked to match previous scenarios.
For example, detecting credit card fraud.

Neural Network in Use
Since neural networks are best at identifying patterns or trends
in data, they are well suited for prediction or forecasting
needs including:
– sales forecasting
– industrial process control
– customer research
– data validation
– risk management
ANN are also used in the following specific paradigms:
recognition of speakers in communications; diagnosis of
hepatitis; undersea mine detection; texture analysis; three-
dimensional object recognition; hand-written word recognition;
and facial recognition.

Neural networks in Medicine
 Artificial Neural Networks (ANN) are currently a 'hot'
research area in medicine and it is believed that they
will receive extensive application to biomedical
systems in the next few years.
 At the moment, the research is mostly on modeling
parts of the human body and recognizing diseases
from various scans (e.g. cardiograms, CAT scans,
ultrasonic scans, etc.).

Continued
 Neural networks are ideal in recognizing diseases
using scans since there is no need to provide a
specific algorithm on how to identify the disease.
 Neural networks learn by example so the details of
how to recognize the disease are not needed.
 What is needed is a set of examples that are
representative of all the variations of the disease.
 The quantity of examples is not as important as the
'quality'. The examples need to be selected very
carefully if the system is to perform reliably and
efficiently.

Use of Neural Networks in C.A.I.S.
Project
 In Project, a Neural Network implementing
supervised learning should be used.
 This would allows the system to first predict an
output and then compare its output with the gold
standard to calculate its error.
 By implementing a Neural Network utilizing the Back
propagation algorithm, we can assure that the
network will be able to reduce its prediction error
given the right training examples.

Continued
 However, to do this, the network needs to have
enough information/ variables to make connection to
the diagnosis of different variations of heart murmur.
 Simply the signal processed sound may give us an
output, but it has the possibility to be error prone.
 Hence the network will need to have a supply of
contextual data, gathered from experts while
training, to make strong and correct predictions.

Disadvantage of Neural Network
 The individual relations between the input
variables and the output variables are not
developed by engineering judgment so that
the model tends to be a black box or
input/output table without analytical basis.
 The sample size has to be large.
 Requires lot of trial and error so training can
be time consuming.

Conclusion
 A Neural Network can be successfully used
in the C.A.I.S. project, given that we have
access to enough training data.
 Since we need it to be highly accurate, we
need to carefully gather contextual variables
that closely relate to diagnosing heart
murmur and its various grades.

Neural network final NWU 4.3 Graphics Course

In this document