开发者

C# linq Files.ReadAllLines() fails for large 650MB CSV file

The following code works when I work with CSV files under 1MB but fails when I try to read 600MB file. Any reason why? Or any fixes?

What I am trying to do is read a large raw CSV file in Visual C# 2010 and manipulate the contents, could be line by line or to memory at one go and export 5 files with certain selections using LINQ. These 5 files are to be used in various processes so need them to be split into 5 different files with very different content.

When the file is small the codes work perfect but when it's too big it gives me the messagebox from Exception handling "Cannot write to source destination开发者_如何学编程". I have tried both ReadAllLines() and ReadLines() Please could you advise me. Thanks.

public void button2_Click(object sender, EventArgs e)
    {

        string file_name = textBox1.Text.ToString();
        // Get the directories to split the file in.
        string directoryPath = Path.GetDirectoryName(textBox1.Text.ToString());
        if (File.Exists(file_name) == true)
        {
            try
            {
                StreamReader readerfile = new StreamReader(file_name);

                var BillSummaryQuery1 =
                    (from line in File.ReadAllLines(file_name)
                     let linerecord = line.Split(',').ToList()
                     select line).ToList();

                #region Start Writing BillSummary.CSV


                //lines removed

                #endregion End writing BillSummary.CSV

                #region Start writing Notes.CSV

                //lines removed


                #endregion Notes.CSV



                string message =
                        "Bill Translated Successfully! \r\nFiles located in: " + directoryPath;
                MessageBox.Show(message, "Success", MessageBoxButtons.OK, MessageBoxIcon.Information);

            }

            catch (Exception)
            {
                string message2 = "Cannot write to source destination";
                MessageBox.Show(message2, "Error");



            }


        }
        else
        {
            MessageBox.Show("No such file exists","Error",MessageBoxButtons.OK,MessageBoxIcon.Error);
        }


If you are using a StreamReader, why don't use it ?

public void button2_Click(object sender, EventArgs e)
{
    string file_name = textBox1.Text.ToString();
    // Get the directories to split the file in.
    string directoryPath = Path.GetDirectoryName(textBox1.Text.ToString());
    if (File.Exists(file_name) == true)
    {
        try
        {
            using (StreamReader reader= new StreamReader(file_name))
            {
                string line = null;
                while ((line = reader.ReadLine()) != null)
                {
                    // Do your stuff
                }
            }
        }

        catch (Exception ex)
        {
            Trace.TraceError(ex.Message);
            string message2 = "Cannot write to source destination";
            MessageBox.Show(message2, "Error");
        }
    }
    else
    {
        MessageBox.Show("No such file exists", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
    }
}


Rolling your own CSV reader is a waste of time unless the files that you're reading are guaranteed to be very simple. Use a pre-existing, tried-and-tested implementation instead.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜