Read a large file into a byte array with chunks in C#

Today in this article we shall see the simple and easy approach on how to Read a Large File in Chunks in C# using the chunking approach.

The discussed approach is very useful while dealing with large-size of files (.TXT or .CSV or .XLSX) like in GB or TB.

However, while you create chunks please make sure the chunks are evenly created and you do not miss the data from one chunks to another chunks.

Please visit this article for more information.

Today in this article, we will cover below aspects,

For demonstration purposes, I have used a very simple example here.

Let’s say I have a Text file size of 10 MB and I would like to read the file in a chunk of size 1 MB.

So for the above 10 MB file, we will be creating ~ 10 files with 1 MB each.

Below is the overall logic which reads the file as a byte array and writes it to the file.

 private static void ReadFile(string filePath)
        {
            const int MAX_BUFFER = 1048576; //1MB 
            byte[] buffer = new byte[MAX_BUFFER];
            int bytesRead;
            int noOfFiles = 0;
            using (FileStream fs = File.Open(filePath, FileMode.Open, FileAccess.Read))
            using (BufferedStream bs = new BufferedStream(fs))
            {
                while ((bytesRead = bs.Read(buffer, 0, MAX_BUFFER)) != 0) //reading 1mb chunks at a time
                {
                    noOfFiles++;
                    //Let's create a small size file using the data. Or Pass this data for any further processing.

                    File.WriteAllBytes($"Test{noOfFiles}.txt", buffer);
                }
            }
        }

In the above code, we are reading the file using the FileStream Open method which lets you open FileStream on the specified path, with the specified mode.

Create a file using WriteAllBytes

Please note that the WriteAllBytes method creates a new file, writes the specified byte array to the file, and then closes the file also. If the target file already exists, it is overwritten.

Once after successful reading, You shall see a total of 10 files of 1MB size go generated in the selected output folder.

The issue with file data truncated between chunks

It’s possible above chunk logic will miss/truncate some data. Kindly use the below logic for a more full-proof approach.

If you need to resolve the issue with truncated data or partial data issues, please refer article on simplified approach Resolving Issues with file data truncated between chunks

References

• Resolving Issue with large file data truncated between chunks

Do you have any comments or ideas or any better suggestions to share?

Please sound off your comments below.

Happy Coding !!

Please bookmark this page and share it with your friends. Please Subscribe to the blog to receive notifications on freshly published(2024) best practices and guidelines for software design and development.