Search code examples
c#data-collection

Data Collection from multiple workstations

I'm currently working on developing a windows service which will be deployed to all the workstation within the company. The purpose of the service is to collect various statistics and record them in a central location.

for the first iteration, the service is doing direct call to SQL to record the every time there is data to record. Testing it on about 120 workstation told me that it is not a good idea and won't scale.

Can someone recommend an alternative strategy for collecting the data in question which won't overload the database server? This will be deployed to about 2000 machines.

I'm working with C#.

Thanks.

Solution

  • Just answering to wrap up the question. Here is how I ended up implementing it.

    Data Collection

    • A C# service runs on the workstations. At different intervals (not predetermined, based on certain events on the system) it fires off data collection.
    • Data is written to a file on the local workstation.
    • Data file is then copied to a network location.

    Data Parsing

    • An "importer" program runs on a schedule. It goes through all the text files created since it last ran, parses them and imports them into SQL.

    The reason I implemented it this way is to:

    • Avoid resource issues with my SQL server that I mentioned in the original question.
    • Files created are really small, 1-2KB
    • Importer runs a couple of minutes in the past to avoid colliding with any copies in progress.
    • Importer deletes the files which have been processed and skips the files which are not within it's "lastscantime" and "currentTime-Xminute" window.