Transferring Large Files With a Spotty Internet Connection

Writer’s Note: It’s been brought up to my attention that rsync is a much smoother alternative to this method. While that is true, many distros do not come installed with rsync by default.

More often than not, I’ve encountered times where I’ve wanted to transfer a files that are several gigabytes large, sometimes 10+, from one machine on one network to another machine on another network. I go through the whole rigmarole of making sure I have enough space, and sending them off only to have the transfer stall after 45% of the transfer has finished. I have to start all over again and it’s a real piss-off.

A while ago, however, I had been sent a rar far that had several parts to it and I had to have all parts to extract the contents. I figured why can’t I do that here on Linux? Well, you can. Here’s how.

There’s a nifty tool called split. It’s very simple and easy to use. I’ll give an example:

split -b 1G LTSP-DHCP-DNS.ova "LTSP-DHCP-DNS.ova.part"

Looking at this command, it’s not hard to figure out. split is the command. -b 1G is the special flag, along with the specification that tells split how large you want each part to be. LTSP-DHCP-DNS.ova is the file I want to split, which here it’s an exported virtual machine. Lastly, "LTSP-DHCP-DNS.ova.part" is the nomenclature that I want split to use when creating the new split files. Split will add a two letter identifier to tell you which part of the puzzle you have. For example LTSP-DHCP-DNS.ova.partah is the 8th part in the a set.

Result

Once the file’s been split like pictured above (the original, then its parts), go ahead and use your favourite transfer protocol to bring the parts from point A to B. As you see here, I’ve used scp.

Luckily, this time it didn’t stall. But if it had stalled or completely lost connection, I would simply restart the transfer but use wildcards. Example:

scp LTSP-DHCP-DNS.ova.parta[c-m] volta:

This would start the transfer at LTSP-DHCP-DNS.ova.partc and move all the way to LTSP-DHCP-DNS.ova.partm, which in this case happens to be the last one.

Once you’ve transferred all the files, ssh into the destination machine and you will be using cat to reassemble the files. In my case the command would be the following:

cat LTSP-DHCP-DNS.ova.part* > LTSP-DHCP-DNS.ova

Of course, firstly we have the cat command. Then we put the wildcard-equipped filename that signifies we want all files that match the beginning of that filename. The > redirects the cat output directly into the last part of the command here which we see is the LTSP-DHCP-DNS.ova, thus reassembling my exported virtual machine.

Note: There will be no output signifying success or not for the cat command. Also, make sure that your locale will enable your parts to be put together back into order.

Title Image Credit: Lifehacker

2 Trackbacks & Pingbacks

  1. Links 8/5/2017: Debian GNU/Linux 8.8, Chromebook Shipments Up 38% | Techrights
  2. Transfer large files over a spotty connection | 0ddn1x: tricks with *nix

Leave a Reply

Your email address will not be published.


*