Search code examples
c#neural-networknormalizationnormalizeencog

How do I normalize Non-CSV data in Encog


So I'm using Jeff Heaton's Neural Network library.

When trying to solve the Iris plant classification problem I have an issue with data normalization.

I am able to Normalize a CSV file using the following method:

public void NormalizeFile(FileInfo SourceDataFile, FileInfo NormalizedDataFile, FileInfo NormalizationConfigFile)
    {

        var wizard = new AnalystWizard(_analyst);
        wizard.Wizard(SourceDataFile, _useHeaders, AnalystFileFormat.DecpntComma);
        var norm = new AnalystNormalizeCSV();
        norm.Analyze(SourceDataFile, _useHeaders, CSVFormat.English, _analyst);
        norm.ProduceOutputHeaders = _useHeaders;
        norm.Normalize(NormalizedDataFile);

        // save normalization configuration, which can be used later to denormalize to get the raw output.
        _analyst.Save(NormalizationConfigFile);

    }

So far so good... The program works with a high degree of accuracy.

The problem occurs when I want to enter the values into my console application.

I have some input data

  • sepal width
  • sepal length
  • petal width
  • petal length

Each of these values has a different high/low I would like to normalize these values so that I can feed them into my network without writing a CSV file to disk.


Solution

  • I later realised that what I really needed as an analyser that would all me to automatically normalise a mixture of qualitative (nominal) and quantitive data (just like the CSV implementation).

    The problem was that the existing code was tightly coupled to CSV files. To combat this I wrote my own encog extension method library.

    it can be found here: https://github.com/KiransHub/encog-dotnet-core

    Here's an example of it in action:

        public void NormalizeDataExample()
        {
            List<LoadedMarketData> AppleMarketData = GetMarketData("AAPL");
            List<LoadedMarketData> MicrosoftMarketData = GetMarketData("MSFT");
            List<LoadedMarketData> YahootMarketData = GetMarketData("YHOO");
    
            List<LoadedMarketData> MarketData = new List<LoadedMarketData>();
            MarketData.AddRange(AppleMarketData);
            MarketData.AddRange(MicrosoftMarketData);
            MarketData.AddRange(YahootMarketData);
    
            DataSet dataSet = new DataSet().Convert(MarketData, "Market DataSet");
            var analyst = new EncogAnalyst();
            var wizard = new AnalystWizard(analyst);
            wizard.Wizard(dataSet);
    
            var normalizer = new AnalystNormalizeDataSet(analyst);
            var normalizedData = normalizer.Normalize(dataSet);
    
    
        }