Search code examples
c#wpfeventsfilesystemwatcherdynamic-data-display

D3 plotter and file watcher, interference?


I am using file watcher monitoring the file output, and then dynamically display the results using d3 (dynamic data display). Here is more details:

  1. Data are series of tiff file generated one by one quickly (20-25 miliseconds per file generation);
  2. Once there is a file coming in, event is fired, and the file is processed (some stats calculation);
  3. The results will be sent to a d3 plotter, which will dynamically display the results on a charter.

When I close the d3 charter window, the file watcher catcher pretty much all the events, but here is my problem:

  • However, when I leave the d3 charter window open, even when there is not any stats calculation, the file watcher drop many events (some big gap occurs).

Here is what we have tried:

  • Increase the watcher buffer, but, it still drop events as long as the d3 charter window is open.
  • Using C++ dll to calculate stats, but it looks even more slow.

So I am wondering:

  • Do d3 plotter and file watcher conflict/interfere with each other?
  • Can I possibly use them together to to real time plot?
  • Is there anyway to get around my problem?

Any pointers are appreciated.

Here is the code for d3 plotter, including constructor, plotter initiator, and plotter update:

#region Constructor
public SignalStatsDisplay()
{
    InitializeComponent();

    //   timeDomainPlotter.Legend.Remove();

    _initialChildrenCount = timeDomainPlotter.Children.Count;

    int count = timeDomainPlotter.Children.Count;

    //do not remove the initial children
    if (count > _initialChildrenCount)
    {
        for (int i = count - 1; i >= _initialChildrenCount; i--)
        {
            timeDomainPlotter.Children.RemoveAt(i);
        }
    }

    _nMaxStatsOneChannel = Enum.GetNames(typeof(Window1.ROISignalList)).Length;
    _curveBrush = new Brush[_nMaxStatsOneChannel];
    _statsEnable = new int[_nMaxStatsOneChannel];

    for (int i = 0; i < _nMaxStatsOneChannel; i++)
    {
        _curveBrush[i] = new SolidColorBrush((Color)ColorConverter.ConvertFromString(_colorList[i]));
        _statsEnable[i] = 0;
    }

    _nActiveStatsOneChannel = 0;
}
#endregion Constructor

public void InitiateSignalAnalysisPlot()
{
    _statsName = Enum.GetNames(typeof(Window1.ROISignalList));

    int count = 0;
    _statsEnableIndex = new int[_nActiveStatsOneChannel];
    for (int i = 0; i < _nMaxStatsOneChannel; i++)  // assign color
    {
        if (_statsEnable[i] == 1)
        {
            _statsEnableIndex[count] = i;
            count++;
        }
    }



    if (_nActiveChannel > 0)    // timeDomainPlotter init
    {
        _dataX = new List<double[]>();
        _dataY = new List<double[]>();

        double[] dataXOneCh = new double[_signalLength];
        double[] dataYOneCh = new double[_signalLength];

        dataXOneCh[0] = 0;
        dataYOneCh[0] = 0;

        for (int i = 0; i < _nActiveChannel; i++)
        {
            for (int j = 0; j < _nActiveStatsOneChannel; j++)
            {
                _dataX.Add(dataXOneCh);    // data x-y mapping init
                _dataY.Add(dataYOneCh);

                EnumerableDataSource<double> xOneCh = new EnumerableDataSource<double>(dataXOneCh);
                EnumerableDataSource<double> yOneCh = new EnumerableDataSource<double>(dataYOneCh);

                xOneCh.SetXMapping(xVal => xVal);
                yOneCh.SetXMapping(yVal => yVal);

                CompositeDataSource dsOneCh = new CompositeDataSource(xOneCh, yOneCh);

                LineAndMarker<MarkerPointsGraph> lam = timeDomainPlotter.AddLineGraph(dsOneCh,
                    new Pen(_curveBrush[_statsEnableIndex[j]], 2),
                    new CirclePointMarker { Size = 5, Fill = _curveBrush[_statsEnableIndex[j]] },
                    new PenDescription(_statsName[_statsEnableIndex[j]]));

            }
        }

        Action FitToView = delegate()
        {
            timeDomainPlotter.FitToView();
        };
        this.Dispatcher.Invoke(System.Windows.Threading.DispatcherPriority.Normal, FitToView);
    }
    else
    {
        return;
    }
}

public void RedrawSignalAnalysisPlot()
{
    int startIndex = _initialChildrenCount;

    if ((_nActiveStatsOneChannel > 0) && (_dataX != null) && (_dataY != null))
    {
        CompositeDataSource[] dsCh = new CompositeDataSource[_nActiveStatsOneChannel];
        int m, n;
        int index;

        for (int i = 0; i < _nActiveChannel; i++)
        {
            for (int j = 0; j < _nActiveStatsOneChannel; j++)
            {
                index = i * _nActiveStatsOneChannel + j;
                if (_dataX[index].Length == _dataY[index].Length)
                {
                    EnumerableDataSource<double> xOneCh = new EnumerableDataSource<double>(_dataX[index]);
                    xOneCh.SetXMapping(xVal => xVal);
                    EnumerableDataSource<double> yOneCh = new EnumerableDataSource<double>(_dataY[index]);
                    yOneCh.SetYMapping(yVal => yVal);
                    CompositeDataSource ds = new CompositeDataSource(xOneCh, yOneCh);

                    Action UpdateData = delegate()
                    {
                        m = i * 2;
                        n = j * 2;

                        // ((LineGraph)timeDomainPlotter.Children.ElementAt(startIndex + n + m * _nActiveStatsOneChannel)).DataSource = ds;
                        // ((LineGraph)timeDomainPlotter.Children.ElementAt(startIndex + n + m * _nActiveStatsOneChannel)).LinePen
                        //    = new Pen(new SolidColorBrush(_curveBrush[j]), 1);

                        ((MarkerPointsGraph)timeDomainPlotter.Children.ElementAt(startIndex + n + 1 + m * _nActiveStatsOneChannel)).DataSource = ds;
                        //  ((MarkerPointsGraph)timeDomainPlotter.Children.ElementAt(startIndex + n + 1 + m * _nActiveStatsOneChannel)).Marker
                        //      = new CirclePointMarker { Size = 5, Fill = Brushes.Green };
                    };

                    this.Dispatcher.Invoke(System.Windows.Threading.DispatcherPriority.Normal, UpdateData);
                }

            }


        }

        /* Action PlotFitToView = delegate()
          {
              timeDomainPlotter.FitToView();
          };

         this.Dispatcher.Invoke(System.Windows.Threading.DispatcherPriority.Normal, PlotFitToView);*/
    }
}

Here is how files are monitored: Where events (file written) are filed

void tiffWatcher_EventChanged(object sender, WatcherExEventArgs e)
{
    string fileName = ((FileSystemEventArgs)(e.Arguments)).FullPath;
    string fileExt = StringExtension.GetLast(fileName, 4);

    if (!IsFileLocked(fileName))
    {

        Action EventFinished = delegate()
        {
            CreateListViewItem(fileName, "Finished", DateTime.Now.ToString("HH:mm:ss.fff"));
        };

        listView1.Dispatcher.Invoke(System.Windows.Threading.DispatcherPriority.Normal, EventFinished);

        _tiffName.Add( fileName);
        _tiffFilledCount++;    
    }
}

tiff data processing, and data communication to d3 plotter:

void tiffTimer_Tick(object sender, EventArgs e)
{
    //throw new NotImplementedException();
    byte[] image = new byte[_signalManager.ImgWidth * _signalManager.ImgHeight];

    if (_tiffFilledCount - _tiffProcessedCount >= 1)
    {


        string fileName = _tiffName[_tiffProcessedCount++];
        char filePre = fileName[49];
        int indexBeigin = fileName.LastIndexOf("_");
        int indexEnd = fileName.LastIndexOf(".");

        _signalIndex = Convert.ToInt32(fileName.Substring(indexBeigin + 1, indexEnd - indexBeigin - 1)) - 1; // 0 based//
        _deltaT = ExtractTiffDeltaT(fileName, "DeltaT=", 1);
        _channelIndex = (int)Enum.Parse(typeof(ChannelList), Convert.ToString(filePre));
        TIFFIImageIO.LoadTIFF(fileName, ref image);
        _signalManager.Image = image;
        for (int i = 0; i < _nActiveStatsOneChannel; i++)
        {
            _signalManager.GetSignal( _signalDisplay.StatsEnableIndex[i], ref _signal);
            UpdateSignal(_channelIndex, i, _tiffProcessedCount-1, _deltaT, _signal);
        }

      // if( _tiffProcessedCount % 5 == 0)
        _signalDisplay.SetData(_XList, _YList, true);
    }
}

I use a Timer.Tick to process the file every 50-100 miliseconds, still testing.


Solution

  • The way I handled it was to setup a Timer, cyclically handling calculation, display. So far, it works fine. Of course there will be many events are missing, but we handle large amount of points (thousands of them), so it is OK to hundreds to do the plot. Here is the code:

    private Timer _tiffTimer;
    void Window1_Loaded(object sender, RoutedEventArgs e)
            {
                //throw new NotImplementedException();
                _tiffTimer = new Timer();
                _tiffTimer.Interval = 50;  // change interval to change performance
                _tiffTimer.Tick += new EventHandler(tiffTimer_Tick);
                _tiffTimer.Start();
    
            }
    
    void tiffTimer_Tick(object sender, EventArgs e)
            {
                //do your stuff here
    
            }