GoLang: Reader-Writer implemented via bufio library to copy file

How to copy a file in small portions into a target file?

Let’s talk about the why?

When you copy large files, you want to ensure that you are copying it piece by piece to avoid out-of-memory errors. As a good practice, it’s best to consider optimizations during development to avoid edge cases that might creep up in production.

Imagine a scenario where your copy file has been working flawlessly until one fine day when a 5GB file is thrown at it to be copied from one location to another. If memory is a problem, then it will throw a out-of-memory and crash.

Now the complete example

package main

import (
	"bufio"
	"io"
	"os"
)

func main() {
	// Objective: Copy contents of FileA to FileB
	// Perform this step in chuncks instead of using the
	// built-in copy functionality.
	// Why?
	// Illustrates how to use bufio library with a reader and writer

	// Source
	inFile, err := os.Open("./source-file.txt")
	if err != nil {
		panic(err)
	}
	defer inFile.Close()

	// Target
	outFile, err := os.Create("./target-file.txt")
	if err != nil {
		panic(err)
	}
	defer outFile.Close()

	// Read a bit then write a bit -- repeat until EOF
	reader := bufio.NewReader(inFile)
	writer := bufio.NewWriter(outFile)
	for {
		if slice, err := reader.ReadSlice('\n'); err == nil || err == io.EOF {
			writer.Write(slice)
			// IMPORTANT: Invoke flush()
			writer.Flush()
			if err == io.EOF {
				break
			}
		}
	}

}

Deeper analysis

• In lines 17-20, we Open() the source and panic if we have a file error.
• In line 21, we defer closing the file when we exit that function. You got to love Go for this one capability alone! :-)
• In lines 24-28, we repeat the same process for the outgoing file.
• In lines 31,32, we create the NewReader and NewWriter.
• In lines 33-42, we create loop until we reach the End-Of-File marker and in line 35 we write the output out to the file.
• In lines 35, It is important to always call writer.Flush() to commit the writes to file. Failure to do this will end up with an empty file.

Ready to try this on your local system?

Feel free to check out the project located here:

GoLang-CopyFile

All source code and steps to compile have been included at the repository above. Ping me if you run into any issues.

Cheers!

Today’s Quote

When you talk, you are only repeating what you already know.
But if you listen, you may learn something new.
- The Dalai Lama, XIV