Building upon our previous adventure with SSH file transfers, where we discovered a clever method to bypass SCP for single files, today we're transferring multiple files and even directories.
Last time, I shared a neat trick for moving individual files using gzip and base64 within SSH sessions. Now, if you have multiple files, it's quite a bit of work to do this one by one. Today, we are improving on the technique to efficiently handle multiple files:
1. Tar, Compress, Convert, Copy
In comparison to the single-file method, not much changes:
# Note: macOS does not support "-w 0" and it's not needed there. Leave it off.
tar -cf - *.conf| gzip -c | base64 -w 0; echo
# or if you are on a modern system, use xz
tar -cf - *.conf| xz | base64 -w 0; echo
The above one-liner would compress all files ending with .conf . You could also list the files explicitly and even add directories.
This works mostly thanks to the tar command, as it takes care of packing everything into an uncompressed archive. Compression is then done by gzip (or xz).
A note on macOS: macOS ships with a base64 utility that does not support the -w 0 flag. Simply leave it off, it's not needed.
Copy the resulting string into your clipboard and head on to step 2.
2. Paste, Decode, Decompress, Untar
Over at your destination, the reversal is also quite short.
Important: The command below overwrites existing files without asking. Make sure you are in the correct directory, or adjust the tar command at the end.
echo "PASTED_BASE64_STRING" | base64 --decode | gunzip | tar -xf -
# or if you've used xz
echo "PASTED_BASE64_STRING" | base64 --decode | unxz | tar -xf -
Voila! The files are decoded, decompressed, and extracted right where you need them.
Pro Tip: Verifying Integrity of Multiple Files
Don’t forget to check if your files made the journey intact! A quick way is to compare checksums between source and destination. You can find how to do just that in the other article.
Wrapping Up
Gone are the days of cumbersome file transfers. At least for small files and directories. While this might not be necessary when using tools like Ansible, Chef, Puppet or even Kubernetes, it is still a good tool to keep in the devops/sysadmin toolbelt. Especially when working directly on servers.
