Posts with «feed forward» label

Neural Network (Part 6) : Back Propagation, a worked example

A worked example of a Back-propagation training cycle.




In this example we will create a 2 layer network (as seen above), to accept 2 readings, and produce 2 outputs. The readings are (0,1) and the expectedOutputs in this example are (1,0).

Step 1: Create the network

NeuralNetwork NN = new NeuralNetwork();   
NN.addLayer(2,2);
NN.addLayer(2,2);
float[] readings = {0,1};
float[] expectedOutputs = {1,0};
NN.trainNetwork(readings,expectedOutputs);

This neural network will have randomised weights and biases when created.
Let us assume that the network generates the following random variables:

LAYER1.Neuron1
Layer1.Neuron1.Connection1.weight = cW111 = 0.3
Layer1.Neuron1.Connection2.weight = cW112 = 0.8
Layer1.Neuron1.Bias = bW11 = 0.5

LAYER1.Neuron2
Layer1.Neuron2.Connection1.weight = cW121 =  0.1
Layer1.Neuron2.Connection2.weight = cW122 =  0.1
Layer1.Neuron2.Bias = bW12 = 0.2

LAYER2.Neuron1
Layer2.Neuron1.Connection1.weight = cW211 = 0.6
Layer2.Neuron1.Connection2.weight = cW212 = 0.4
Layer2.Neuron1.Bias = bW21 = 0.4

LAYER2.Neuron2
Layer2.Neuron2.Connection1.weight = cW221 = 0.9
Layer2.Neuron2.Connection2.weight = cW222 = 0.9
Layer2.Neuron2.Bias = bW22 = 0.5




Step 2: Process the Readings through the Neural Network

a) Provide the Readings to the first layer, and calculate the neuron outputs

The readings provided to the neural network is (0,1), which go straight through to the first layer (layer1).
Starting with Layer 1:
Layer1.INPUT1 = 0
Layer1.INPUT2 =1

   Calculate Layer1.Neuron1.NeuronOutput
   ConnExit (cEx111) = ConnEntry (cEn111)  x Weight (cW111) = 0 x 0.3 = 0;
   ConnExit (cEx112) = ConnEntry (cEn112)  x Weight (cW112) = 1 x 0.8 = 0.8;
   Bias (bEx11) = ConnEntry (1) x Weight (bW11) = 1 x 0.4 = 0.4
   NeuronInputValue11 = 0 + 0.8 + 0.4 = 1.2
   NeuronOutputValue11 = 1/(1+EXP(-1 x 1.2)) = 0.768525
  
  Calculate Layer1.Neuron2.NeuronOutput
   ConnExit (cEx121) = ConnEntry (cEn121)  x Weight (cW121) = 0 x 0.1 = 0;
   ConnExit (cEx122) = ConnEntry (cEn122)  x Weight (cW122) = 1 x 0.1 = 0.1;
   Bias (bEx12) = ConnEntry (1) x Weight (bW12) = 1 x 0.2 = 0.2
   NeuronInputValue12 = 0 + 0.1 + 0.2 = 0.3
   NeuronOutputValue12 = 1/(1+EXP(-1 x 0.3)) = 0.574443


b) Provide LAYER2 with Layer 1 Outputs.

Now lets move to  Layer 2:
Layer2.INPUT1 = NeuronOutputValue11 = 0.768525
Layer2.INPUT2 = NeuronOutputValue12 = 0.574443

   Calculate Layer2.Neuron1.NeuronOutput
   ConnExit (cEx211) = (cEn211)  x Weight (cW211) = 0.768525 x 0.6 = 0.461115;
   ConnExit (cEx212) = (cEn212)  x Weight (cW212) = 0.574443 x 0.4 = 0.229777;
   Bias (bEx21) = ConnEntry (1) x Weight (bW21) = 1 x 0.4 = 0.4
   NeuronInputValue21 = 0.461115 + 0.229777 + 0.4 = 1.090892
   NeuronOutputValue21 = 1/(1+EXP(-1 x 1.090892)) = 0.74855
  
  Calculate Layer2.Neuron2.NeuronOutput
   ConnExit (cEx221) = (cEn221)  x Weight (cW221) = 0.768525  x 0.1 = 0.076853;
   ConnExit (cEx222) = (cEn222)  x Weight (cW222) = 0.574443  x 0.1 = 0.057444;
   Bias(bEx22) = ConnEntry (1) x Weight (bW22) = 1 x 0.5 = 0.5
   NeuronInputValue22 = 0.076853 + 0.057444 + 0.5 = 0.634297  
   NeuronOutputValue22 = 1/(1+EXP(-1 x 0.634297)) = 0.653463



Step 3) Calculate the delta error for neurons in layer 2
     -Because layer 2 is the last layer in this neural network -
      we will use the expected output data (1,0) to calculate the delta error.
   
LAYER2.Neuron1:
Let Layer2.ExpectedOutput1 = eO21 = 1    
      Layer2.ActualOutput1= aO21 = NeuronOutputValue21= 0.74855         
      Layer2.Neuron1.deltaError1 = dE21

dE21 =     aO21       x      (1 - aO21)     x  (eO21 - aO21)
       =  (0.74855)  x  (1 - 0.74855)  x  (1 - 0.74855)
        = (0.74855)  x     (0.25145)     x    (0.25145)
        = 0.047329



LAYER2.Neuron2:
Let Layer2.ExpectedOutput2 = eO22 = 0         
      Layer2.ActualOutput2     = aO22 = NeuronOutputValue22 = 0.653463      
      Layer2.Neuron2.deltaError = dE22

dE22  =      aO22       x      (1 - aO22)       x  (eO22 - aO22)
        = (0.653463)  x  (1 - 0.653463)  x  (0 - 0.653463)
        = (0.653463)  x     (0.346537)     x    (-0.653463)
        = -0.14797




Step 4) Calculate the delta error for neurons in layer 1

LAYER1.Neuron1 delta Error calculation

Let              Layer1.Neuron1.deltaError  = dE11 
                            Layer1.actualOutput1  = aO11 = NeuronOutputValue11 =  0.768525
      Layer2.Neuron1.Connection1.weight = cW211   =  0.6
                     Layer2.Neuron1.deltaError = dE21 =  0.047329
      Layer2.Neuron2.Connection1.weight = cW221   =  0.9
                     Layer2.Neuron2.deltaError = dE22 = -0.14797

dE11 = (aO11)          x  (1 -   aO11)         x ( [cW211   x   dE21]      +   [cW221  x    dE22] )
           = (0.768525) x   (1 - 0.768525)     x   ([0.6        x  0.047329]  +   [  0.9      x  -0.14797]  )
           = -0.01864

LAYER1.Neuron2 delta Error calculation

Let              Layer1.Neuron2.deltaError  = dE12 
                            Layer1.actualOutput2  = aO12    = NeuronOutputValue12 =  0.574443
      Layer2.Neuron1.Connection2.weight = cW212   =  0.4
                     Layer2.Neuron1.deltaError = dE21 =  0.047329
      Layer2.Neuron2.Connection2.weight = cW222   =  0.9
                     Layer2.Neuron2.deltaError = dE22 = -0.14797

dE12  = (aO12)          x  (1 -   aO12)         x ( [cW212  x     dE21]      +   [cW222  x    dE22] )
           = (0.574443) x   (1 - 0.574443)  x     ([0.4      x  0.047329]  +      [  0.9      x  -0.14797]  )
           = -0.02793





Step 5) Update Layer_2 neuron connection weights and bias (with a learning rate (LR) = 0.1)


Layer 2, Neuron 1 calculations:

Let
Layer2.Neuron1.Connection1.New_weight = New_cW211
Layer2.Neuron1.Connection1.Old_weight   =   Old_cW211   = 0.6
Layer2.Neuron1.Connection1.connEntry =                 cEn211 = 0.768525
Layer2.Neuron1.deltaError =                                       dE21 = 0.047329

New_cW211 = Old_cW211 + (LR x cEn211 x dE21)
                     =    0.6            + (0.1 x 0.768525 x 0.047329)
                     =    0.6            + ( 0.003627)
                     =    0.603627



Layer2.Neuron1.Connection2.New_weight = New_cW212
Layer2.Neuron1.Connection2.Old_weight   =   Old_cW212 = 0.4
Layer2.Neuron1.Connection2.connEntry =                cEn212 = 0.574443
Layer2.Neuron1.deltaError =                                      dE21 = 0.047329

New_cW212 = Old_cW212 + (LR x cEn212 x dE21)
                     =    0.4            + (0.1 x 0.574443 x 0.047329)
                     =    0.4            + (0.002719)
                     =    0.402719



Layer2.Neuron1.New_Bias = New_Bias21
Layer2.Neuron1.Old_Bias =    Old_Bias21 = 0.4
Layer2.Neuron1.deltaError =             dE21 = 0.047329

New_Bias21 = Old_Bias21 + (LR x  1  x  de21)
                     =  0.4              + (0.1 x 1  x 0.047329)
                     =  0.4              + (0.0047329)
                     =  0.4047329


--------------------------------------------------------------------

Layer 2, Neuron 2 calculations:

Layer2.Neuron2.Connection1.New_weight = New_cW221
Layer2.Neuron2.Connection1.Old_weight =    Old_cW221 = 0.9
Layer2.Neuron2.Connection1.connEntry =               cEn221 = 0.768525
Layer2.Neuron2.deltaError =                                     dE22 = -0.14797

New_cW221 = Old_cW221 + (LR x cEn221 x dE22)
                     =    0.9            + (0.1 x 0.768525 x -0.14797)
                     =    0.9            + ( -0.01137)
                     =    0.88863


Layer2.Neuron2.Connection2.New_weight = New_cW222
Layer2.Neuron2.Connection2.Old_weight =    Old_cW222 = 0.9
Layer2.Neuron2.Connection2.connEntry =              cEn222 = 0.574443
Layer2.Neuron2.deltaError =                                    dE22 = -0.14797

New_cW222 = Old_cW222 + (LR x cEn222 x dE22)
                     =    0.9            + (0.1 x 0.574443 x -0.14797)
                     =    0.9            + (-0.0085)
                     =    0.8915


Layer2.Neuron2.New_Bias = New_Bias22
Layer2.Neuron2.Old_Bias =    Old_Bias22 =  0.5
Layer2.Neuron2.deltaError =             dE22 = -0.14797

New_Bias22 = Old_Bias22 + (LR x  1  x  de22)
                     =  0.5              + (0.1 x  1  x  -0.14797)
                     =  0.5            +   (-0.014797)
                     =  0.485203



--------------------------------------------------------------------------


Step 6) Update Layer_1 neuron connection weights and bias.

Layer 1, Neuron 1 calculations:

Let
Layer1.Neuron1.Connection1.New_weight = New_cW111
Layer1.Neuron1.Connection1.Old_weight   =   Old_cW111   =  0.3
Layer1.Neuron1.Connection1.connEntry =                 cEn111 = 0
Layer1.Neuron1.deltaError =                                       dE11 = -0.01864

New_cW111 = Old_cW111 + (LR   x  cEn111   x   dE11)
                     =  0.3              +   (0.1   x     0      x    -0.01864)
                     =  0.3              +   ( 0 )
                     =  0.3    


Layer1.Neuron1.Connection2.New_weight = New_cW112
Layer1.Neuron1.Connection2.Old_weight   =   Old_cW112 = 0.8
Layer1.Neuron1.Connection2.connEntry =               cEn112 = 1
Layer1.Neuron1.deltaError =                                      dE11 = -0.01864

New_cW112 = Old_cW112 + (LR   x  cEn112   x   dE11)
                     =  0.8    +            (0.1     x    1     x     -0.01864)
                     =  0.8    +            (-0.001864)
                     =  0.798136   


Layer1.Neuron1.New_Bias = New_Bias11
Layer1.Neuron1.Old_Bias =    Old_Bias11 = 0.5
Layer1.Neuron1.deltaError =             dE11 = -0.01864

New_Bias11 = Old_Bias11 + (LR   x  1   x  dE11)
                     =  0.5              + (0.1   x 1   x -0.01864 )
                     =  0.5              + (-0.001864)
                     =  0.498136

--------------------------------------------------------------------

Layer 1, Neuron 2 calculations:

Layer1.Neuron2.Connection1.New_weight = New_cW121
Layer1.Neuron2.Connection1.Old_weight =    Old_cW121 = 0.1
Layer1.Neuron2.Connection1.connEntry =               cEn121 = 0
Layer1.Neuron2.deltaError =                                     dE12 =   -0.02793

New_cW121 = Old_cW121 + (LR  x  cEn121 x dE12)
                     =  0.1               + (0.1  x     0     x  -0.02793 )
                     =  0.1   +   (0)
                     =  0.1




Layer1.Neuron2.Connection2.New_weight = New_cW122
Layer1.Neuron2.Connection2.Old_weight =    Old_cW122 = 0.1
Layer1.Neuron2.Connection2.connEntry =              cEn122 = 1
Layer1.Neuron2.deltaError =                                    dE12 =  -0.02793

New_cW122 = Old_cW122 + (LR  x  cEn122  x   dE12)
                     =  0.1                + (0.1   x    1      x  -0.02793)
                     =  0.1    +  (-0.002793)
                     =  0.097207



Layer1.Neuron2.New_Bias = New_Bias12
Layer1.Neuron2.Old_Bias =    Old_Bias12 =  0.2
Layer1.Neuron2.deltaError =             dE12 =  -0.02793

New_Bias12 = Old_Bias12 + (LR    x  1  x  de12)
                     =  0.2             +   (0.1  x  1  x  -0.02793)
                     =  0.2             +  (-0.002793)
                     =  0.197207


----------------------------------------------------------------------

All done. That was just one training cycle. Thank goodness we have computers !
A computer can process these calculations really quickly, and depending on how complicated your neural network is (ie. number of layers, and number of neurons per layer), you may find that the training procedure may take some time. But believe me, if you have designed it right, it is well worth the wait.
Because once you have the desired weights and bias values set up, you are good to go, and as you receive data, the computer can do a single forward pass in a fraction of a second, and you will get your desired output, hopefully :)

Here is a complete Processing.org script that demonstrates the use of my neural network.
Neural Network (Part 7): Cut and Paste Code (click here).

If you liked my tutorial - please let me know in the comments. It is sometimes hard to know if anyone is actually reading this stuff. If you use my code in your own project, I am also happy for you to leave a link to a YouTube video etc in the comments also.

To go back to the table of contents click here
ScottC 14 Aug 10:11
arduinobasics  back propagation  best arduino blog  feed forward  neural network  processing.org  project  training  tutorial  

Neural Network (Part 1) - The Connection

Introduction

In this tutorial, I am going to walk you through my interpretation of a neural network. I will use terminology that makes sense to me, hoping that Neural Network enthusiasts don't get offended by my novice approach.

This is what a feed-forward Neural network normally looks like:


 The input layer receives input from the outside world, and passes this value to the hidden layer.

The value that reaches the hidden layer depends on the connection between the layers.

Each connection has a weight. This weight multiplier can either increase or decrease the value coming from the input layer. Like water coming out of a tap, you can make it come out faster or slower (or not at all). This weight can even be negative, which would mean that the water is being sucked up, rather than pouring out.

The hidden layer then processes the values, and pass them onto the output layer. The connections between the hidden layer and the output layer also have weights. The values in the output layer are processed and produce a final set of results. The final results can be used to make yes/no decisions, or can be used to make certain classifications etc etc. In my case, I would like to receive sensor data from the arduino, pass this info to the neural network, and get it to classify the data into 4 different classes (Red, Yellow, Green or Ambient), but we will get to see this in another tutorial. For now, we are just going to design a neural network that can be applied to any of your arduino projects... so lets start from the ground up.

I have programmed this neural network in the Processing language, and have decided to break the neural network into smaller bits. Each structural component of the neural network is a class (as you will soon discover).


The connection






ConnEntry (x) : is the value being fed into the connection.
Weight (m) : Is the value that either amplifies or weakens the ConnEntry value.
ConnExit (y): Is the output value of the connection.

The relationship between y and x is linear, and can be described by the following formula.
  • y = mx
In other words, you multiply the ConnEntry value by the Weight to get the ConnExit value.

Here is the code for the Connection class:



processing code Connection Class

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
/* -------------------------------------------------------------------------

A connection determines how much of a signal is passed through to the neuron. 

------------------------------------------------------------------------*/



class Connection{

  float connEntry;

  float weight;

  float connExit;

  

  /*This is the default constructor for a Connection  */

  Connection(){

    randomiseWeight();

  }

  

  /*A custom weight for this Connection constructor  */

  Connection(float tempWeight){

    setWeight(tempWeight);

  }

  

  /*Function to set the weight of this connection  */

  void setWeight(float tempWeight){

    weight=tempWeight;

  }

  

  /*Function to randomise the weight of this connection  */

  void randomiseWeight(){

    setWeight(random(2)-1);

  }

  

 /*Function to calculate and store the output of this Connection  */

  float calcConnExit(float tempInput){

    connEntry = tempInput;

    connExit = connEntry * weight;

    return connExit;

  }

}


When a connection class is constructed, it automatically randomises the weight for you.
Here are some of the other functions of this class:
  • setWeight() : allows you to specifically set the weight of this connection.
  • randomiseWeight() : can be used to wipe the "memory" of the connection if required.
  • calcConnExit() function is the main function of this class. It is the one that multiplies the connEntry value with the Weight to produce a connExit value as described above.



Up next: Neural Network (Part 2): The Neuron

-------------------------------------------------------------------------


To go back to the table of contents click here





ScottC 11 Aug 16:16
arduinobasics  best arduino blog  code  connection  feed forward  neural network  processing.org  project  tutorial  weight