Generate or Create a Large text file in Linux

Linux offers several commands for creating, manipulating and managing text files. We can categorize these commands into regular text-manipulating commands and system text-manipulating commands.

Regular text manipulating commands

We use these commands to create and maintain regular text files. These commands are optimized to manipulate the text easily. These commands include several features and functions to make text editing more manageable. These commands are also known as the text editors. Some popular text editors are the vi, vim, emacs, nano, gedit, and pico.

System text manipulating commands

We use these commands to create or generate the dummy or sample files. These commands are optimized to create text files of any size. These commands do not include any feature or function to manipulate the text. Some popular system text manipulating commands are dd, yes, fallocate, truncate, and touch.

Selecting a command

Which command you should use depends on your requirements. If you want to store meaningful data in the text file, use the regular text manipulating command. If you want to create a file for testing and debugging purposes, use the system text manipulating command.

Let's take an example. You want to store product-related information in a text file. In this case, you should use a regular text editor. But, if you need a 1GB text file for testing, you should never use a regular text editor. Creating a large text file for testing from the regular text editor is only a waste of time. For example, you need a 1GB text file to test an archive utility. If you create this file from the regular text editor, you may have to spend more than one hour adding, creating, and copying the useless text in this file. But, if you use a system text manipulating command, you can make this file in a few seconds. In the below section, we will understand how to use the system text manipulating commands to create files of any size for testing and debugging purposes.

Using the dd command

The dd command takes four arguments: source, destination, block size, and counter. It copies contents from the source to the destination. It uses the block size and the counter to control the copy operation. Bock size and counter allow us to specify the size of the destination file.

It uses the following syntax.

#dd if=[source] of=[destination] bs=[block-size] count=[counter]

The source defines the source file path. We can use the /dev/zero file as the source. It is a special file. It contains null characters. It returns them to the command that reads it. Only a command or utility can read this file in a particular block size. For example, if a command reads in 1kb block size, it returns 1kb null characters.

The destination specifies the file path. The dd command reads null characters from the /dev/zero file and saves them in this file.

A block size defines how much data we want the dd command to read simultaneously. For example, if we set this value to 1M, it reads 1 Mib data.

A counter specifies the number of times the dd command reads the /dev/zero file in the given block size. For example, if we set this value to 10, it reads the file ten times.

The dd command examples

Make a test directory and create three test files of different sizes.

#mkdir /test

The following command creates a 10Kib file.

#dd if=/dev/zero of=/test/file1 bs=1K count=10

The following command creates a 20Mib file.

#dd if=/dev/zero of=/test/file2 bs=1M count=20

The following command creates a 3Gib file.

#dd if=/dev/zero of=/test/file3 bs=1G count=3

Use the ls -lh command to verify files.

#ls -lh /test/

After the practice, remove the test directory.

#rm -rf /test

Using the yes command to generate a test file

You can use the yes command if you want a file that contains some custom characters and lines instead of null characters. The yes command continuously prints the supplied string on the console. We can store the output in a file. We can use the head command to control the file size and break the loop.

It uses the following syntax.

#yes [text or string] | head -c [size of file] > [name of file]

The yes command examples

Make a test directory and create three files of different sizes containing a custom text string.

#mkdir /test
#cd /test

The following command creates a 50Kib file containing the custom test string.

#yes This is a test file | head -c 50K > file1

The following command creates a 50Mib file containing the custom test string.

#yes This is a test file | head -c 50M > file2

The following command creates a 1Gib file containing the custom test string.

#yes This is a test file | head -c 1G > file3

Use the ls -lh command to verify files.

#ls -lh /test/

Remove the test directory to free up the space.

#cd
#rm -rf /test

Using the fallocate and truncate commands

If you do not care about the file's contents, you can use the fallocate and the truncate commands. Instead of writing any data in the file, these commands only manipulate the allocated disk space of the file. Since these commands do not put any character in the file, you can generate a file of any size in a few seconds.

The differences between both commands are the following.

The fallocate command allocates all of the space to the file without writing a single byte of data in the file. If you use the fallocate command to create a 20GB file, you will get a file that consumes 20GB of actual disk space but contains no data.

The truncate command creates a sparse file instead of the actual file. The difference between a sparse file and an actual file is that a sparse file doesn't consume the allocated space. It consumes the space only when we add data to it.

For example, you created two 50GB files, one from the fallocate command and another from the truncate command. The first file immediately consumes all the allocated 50GB space, while the second consumes no data. Since the truncate command does not put any data in the file, the consumed disk space remains unchanged.

Let's take one more example. Suppose you have 5 GB disk space and want to create a file of 10GB for testing. Since the fallocate command allocates all assigned space immediately, you can't make a file of 10GB if you only have 5GB of disk space. In this case, you can use the truncate command. Since the truncate command creates the sparse file, and a sparse file does not consume any disk space until it contains user data, you can easily create a file of 10GB for testing, even if you only have 5GB of disk space.

To generate a file from the fallocate command, use the following syntax.

#fallocate -l [size of file] [name of file]

The following are examples of the fallocate commands.

#fallocate -l 1G file1
#fallocate -l 2G file2

The first command creates a 1Gib file, while the second generates a 2Gib file. Use the ls -lh and du -h commands to verify the file size.

#ls -lh
#du -h file*

The truncate command uses the following syntax.

#truncate -s [file-size] [name of the file]

The following are examples of the truncate commands.

#truncate -s 1G file1
#truncate -s  2G file2

The first command creates a 1Gib file, while the second generates a 2Gib file. Use the ls -lh and du -h commands to verify the file size.

#ls -lh
#du -h file*

Generating an empty file

The touch command creates an empty or zero-size file. It uses the following syntax.

#touch [file name]

The following command generates an empty file.

#touch file

Use the du -h command to verify the file size.

#du -h file

Conclusion

Creating and managing large files for testing and debugging purposes can be efficiently accomplished using system text-manipulating commands. Unlike regular text editors, system text-manipulating commands such as dd, yes, fallocate, and truncate can generate large files quickly and easily. These commands save significant time and effort when generating test files, ensuring an efficient workflow in various testing and debugging scenarios.