dd

The dd command without any options simply reads stdin and writes stdout; it's a copier like cat ("dd" stands for datadumper). If that were the end of it, there would be no reason for it to exist.

However, it can do much more, such as converting ASCII to EBCDIC and vice versa. The need for that has almost (but not quite) disappeared, yet dd still has use. It can block and unblock records (converting vast sequences of bytes to line oriented data and vice versa), programatically skip or seek past input or output (useful for reading or writing X bytes into a file), and more.

Given a file "t" containng "123456790",

 dd if=t bs=1 skip=3
 

produces "4567890", and

echo "abc" | dd of=t seek=3 count=2 bs=1 conv=notrunc
 

leaves "t" as "123ab67890".

If "t" is instead:

123467890
123467890
123467890
123467890
 

Then "dd if=t conv=block cbs=10 " will produce exactly 40 bytes of "1234567890123456789012345678901234567890". Given those 40 bytes, "dd if=u conv=unblock cbs=5" will give you back 8 lines:

12345
67890
12345
67890
12345
67890
12345
67890
 

At the end of any transfer, dd always reports the number of full and partial blocks it read and wrote. Given our same file "t"

echo "1234567890" > t
dd if=t ibs=3 obs=4 
1234567890
3+1 records in
2+1 records out
dd if=t ibs=3 obs=1
1234567890
3+1 records in
11+0 records out
 

Using unblock and block with dd to add linefeeds and white space padding

Some databases work with fixed length records, and sometimes you need to convert those to something you can work with as ordinary text. The "dd" program is the answer for that. Another common need is to pad ascii text with white space to make it the lines a constant length (this might be needed to import into some other program or database). Perl or awk can handle that easily.

In this case, the request was to unblock every 756 bytes and create a constant length output:

First, to convert to lf every 756 characters:

dd if=yourfile of=newfile conv=unblock cbs=756

The next step is to add whitespace. Good old awk can always do that if it's text:

cat newfile | awk '{ printf "%-956s\n",$0 }'

Combine these together with

dd if=yourfile conv=unblock cbs=756 | awk '{ printf "%-956s\n",$0 }' >

Test with small files first to avoid typos.

dd can get a small file from your big file:

dd if=yourfile of=smallfile bs=756 count=3

"dd" can do all sorts of conversion for you - ebcdic to ascii, and more.

You can't use wild cards with "dd". However you can do something like this:

for i in VOL.*
do
echo "next disk"
read ak
dd if=$i of=/dev/fd0135ds18 bs=18k
done
 

See xxd - a little known tool worth knowing

Watch out for blocking

A customer used "dd" to store a file on tape and wondered why the file was different when he recovered it using dd.

The problem is that a tape is a block device - that is, it will always return a full block of data, a block being whatever its block size is. So unless the file written was an even multiple of the tapes block size, there will be nulls tacked on when you bring the file back from tape with dd.

Moral: use tar or cpio or something else that restores files, not blocks.

That's not to say dd is useless with tape. In fact a very common use is this:

tar cvf /tmp/mytar.tar
dd if=/tmp/mytar.tar of=/dev/tape
 

This might be because you want to make multiple tapes of the same tar data; do the tar once, dd it as many times as you like. The resultant tapes can be extracted with tar.

Another very common use is across networks

tar cvf - . | rcmd otherbox dd of=/dev/tape
 

I couldn't tell you how many times I've done that.

You might also do this for performance - a different block size could write the tape more quickly. See Help with solving an error message on a cpio backup.

This thread has a an interesting deep dive into 'dd' and tape drive usage..

Got something to add? Send me email.

1 comment

See also volcopy for a way to "dd" UNIX filesystems.

--BigDumbDinosaur





Mon Sep 8 17:06:13 2014: 12525   anonymous




This is fantastically useful even in 2014!
For smaller files, the GNU fold command can accomplish what dd and awk do. For example, "fold -b -w 150 file " will add a newline after every 150th byte

------------------------

Have you tried Searching this site?

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us