boto-rsync is a great tool for interacting with object storage systems like S3, but it’s not without limitation. We all know about the 5GB limit for a single PUT, which isn’t a problem for clients that can handle multipart upload. Sadly, boto-rsync doesn’t handle that, and until someone patches it, we need a way to break up large objects. This can crudely be done with split:
|
1 |
perl -e 'my @files = split("\n", `find . -size +5G`); foreach ( @files ) { print "splitting $_...\n"; `split --verbose -b 4G \"$_\" \"$_\"`; print "split done, removing $_...\n"; `rm \"$_\"`; }' |
This disadvantage to this is that retrievals need to manually be catted together, which obviously isn’t always a good solution.
boto-rsync’s other weakness is in handling UTF8 filenames. Improperly-encoded filenames will throw a 400 Bad Request and cause the script to choke and die, rather than gracefully skipping the failing file and moving on. Re-encoding files with proper UTF8 fixes this:
|
1 2 3 |
for type in d f; do find -type $type | perl -l -ne '/^([\000-\177]|[\300-\337][\200-\277]|[\340-\357][\200-\277]{2}|[\360-\367][\200-\277]{3}|[\370-\373][\200-\277]{4}|[\374-\375][\200-\277]{5})*$/ or print' | rename 'BEGIN {binmode STDIN, ":encoding(latin1)"; use Encode;} $_=encode("utf8", $_)' done |
Not pretty, but it works. Note that directories need to be checked and renamed first before handling files specifically.
UPDATE – These issues have both been addressed in https://github.com/dreamhost/boto_rsync