Neural network XOR learning

Question

Im struggling to test if my neural network is working properly and I have tried with XOR because that is the simple test case.

Is the XOR a good test case or should I use something else?

I have set up my neural network code in Ada and I have modeled after this tutorial. My neural network with (2-inputs, 3-hidden, 1-output) can not learn XOR. What is the problem in following code?

with Ada.Text_IO;use Ada.Text_IO;
with Ada.Float_Text_IO;use Ada.Float_Text_IO;
with Ada.Numerics.Generic_Elementary_Functions;


procedure Main is

   Learning_Rate : Float := 0.5;

   function Sigmoid(X : Float) return Float is
      package Math is new Ada.Numerics.Generic_Elementary_Functions(Float); use Math;
      e : constant Float := 2.7;
   begin
      return 1.0 / (1.0 + e**(-X));
   end;

   function Sigmoid_Derivative (X : Float) return Float is
   begin
      return Sigmoid(X) * (1.0 - Sigmoid(X));
   end;


   type Float_Array is array (Positive range <>) of Float;
   type Node;
   type Node is record
      S : Float := 0.0; --Summation
      Y : Float := 0.0; --Output
      W : Float_Array(1..10) := (others => 0.0); --Weigths
      D : Float := 0.0; --Delta error
   end record;
   type Layer is array (Positive range <>) of Node;



   --Forward calculations

   procedure Calculate_Summation (N : in out Node; L : in Layer) is
   begin
      N.S := 0.0;
      for Index in L'Range loop
         N.S := N.S + L(Index).Y * N.W(Index);
      end loop;
   end;

   procedure Calculate_Summation (Destination : in out Layer; Source : in Layer) is
   begin
      for N of Destination loop
         Calculate_Summation(N, Source);
      end loop;
   end;

   procedure Calculate_Output (L : in out Layer) is
   begin
      for N of L loop
         N.Y := Sigmoid(N.S);
      end loop;
   end;


   --Backpropogation

   procedure Calculate_Delta (L : in out Layer; N : in Node ) is
   begin
      for Index in L'Range loop
         L(Index).D := L(Index).D + N.D * N.W(Index);
      end loop;
   end;

   procedure Calculate_Delta (Destination : in out Layer; Source : in Layer) is
   begin
      for N of Source loop
         Calculate_Delta(Destination, N);
      end loop;
   end Calculate_Delta;


   function Calculate_Delta_Weight(D, S, X : Float) return Float is
   begin
      return Learning_Rate * D * Sigmoid_Derivative(S) * X;
   end;


   --Weight adjustment

   procedure Calculate_Weight(N : in out Node; L : in Layer) is
   begin
      for Index in L'Range loop
         N.W(Index) := N.W(Index) + Calculate_Delta_Weight(N.D, N.S, L(Index).Y);
      end loop;
      N.D := 0.0;
   end;

   procedure Calculate_Weight(Destination : in out Layer; Source : in Layer) is
   begin
      for N of Destination loop
         Calculate_Weight(N, Source);
      end loop;
   end;

   LI : Layer(1..2);
   LH : Layer(1..3);
   LO : Layer(1..1);

   procedure Learn (A, B, Target : Float) is
   begin

      LI(1).Y := A;
      LI(2).Y := B;

      Calculate_Summation( Source => LI, Destination => LH );
      Calculate_Output(LH);

      Calculate_Summation( Source => LH, Destination => LO );
      Calculate_Output(LO);

      LO(1).D := Target - LO(1).Y;

      Put("A,B,T ="); Put(A, 3,3,0);Put(B, 3,3,0);Put(Target, 3,3,0);
      Put("     Y ="); Put(LO(1).Y, 3,3,0);
      Put("     D ="); Put(LO(1).D, 3,3,0);

      Calculate_Delta(Source => LO, Destination => LH);

      Calculate_Weight(Source => LH, Destination => LO);
      Calculate_Weight(Source => LI, Destination => LH);

   end;


begin

   for I in 1..1000 loop
      Learn(1.0, 1.0, 0.0);New_Line;
      Learn(1.0, 0.0, 1.0);New_Line;
      Learn(0.0, 1.0, 1.0);New_Line;
      Learn(0.0, 0.0, 0.0);New_Line;
      New_Line;
   end loop;
end Main;

The last output is:

A,B,T =  1.000  1.000  0.000     Y =  0.497     D = -0.497
A,B,T =  1.000  0.000  1.000     Y =  0.495     D =  0.505
A,B,T =  0.000  1.000  1.000     Y =  0.494     D =  0.506
A,B,T =  0.000  0.000  0.000     Y =  0.505     D = -0.505

Where Y is output from the neural network and T is wanted target.

Answer 1

First, your weights should be initialized, preferably to random values. For example, Ada.Numerics.Float_Random.Random( foo );
Second, your network should learn better if you add a bias input (a constant input, for example 1.0)

With this, D should start converging towards 0.0.

You can help yourself in a situation like this by printing all the inputs/outputs and weights in your network, including the hidden layer. In your case, a pattern will emerge that should not be there (a result of all weights being 0.0)