Search code examples
text-fileslinqdatasource

Read data from txt files - using Linq-Entities?


in my current ASP.net MVC 3.0 project i am stuck with a situation.

  1. I have four .txt files each has approximatly 100k rows of records
  2. These files will be replaced with new files on weekly bases.

I need to Query data from these four text files, I am not able to choose the best and efficient way to do this.

3 ways I could think

  1. Convert these text files to XML on a weekly basis and query it with Linq-XML
  2. Run a batch import weekly from txt to SQL Server and query using Linq-Entities
  3. avoid all conversions and query directly from text files.

Can any one suggest a best way to deal with this situation.

update:


Url of the Text File URL to access the txt File with credentials
I should connect to this file with credentials.

once i connect successfully, I will have the text file as below with Pipeline as Deliminator
This is the text file


Data in Text File is as this Now i have to look up for the field highlighted in yellow and get the data in that row.

Note: First two lines of the text file are headers of the File.


Solution

  • Well As i Found a way my self. Hope this will be useful for any who are interested to get this done.

    string url = "https://myurl.com/folder/file.txt";
    WebClient request = new WebClient();
                    request.Credentials = new NetworkCredential(ConfigurationManager.AppSettings["UserName"], ConfigurationManager.AppSettings["Password"]);
                    Stream s = request.OpenRead(url);
                    using (StreamReader strReader = new StreamReader(s))
                    {
                        for (int i = 0; i <= 1; i++)
                            strReader.ReadLine();
                        while (!strReader.EndOfStream)
                        {
                            var CurrentLine = strReader.ReadLine();
                            var count = CurrentLine.Split('|').Count();
                            if (count > 3 && CurrentLine.Split('|')[3].Equals("SearchString"))
                            {
                                #region Bind Data to Model
                                //var Line = CurrentLine.Split('|');
                                //CID.RecordType = Line[0];
                                //CID.ChangeIdentifier = Line[1];
                                //CID.CoverageID = Convert.ToInt32(Line[2]);
                                //CID.NationalDrugCode = Line[3];
                                //CID.DrugQualifier = Convert.ToInt32(Line[4]); 
                                #endregion
                                break;
                            }
                        }
                        s.Close();
                    }
                    request.Dispose();