{"id":3560,"date":"2023-11-04T12:53:04","date_gmt":"2023-11-04T11:53:04","guid":{"rendered":"https:\/\/kudzia.eu\/b\/?p=3560"},"modified":"2025-03-02T10:56:35","modified_gmt":"2025-03-02T09:56:35","slug":"parallel-rsync","status":"publish","type":"post","link":"https:\/\/kudzia.eu\/b\/2023\/11\/parallel-rsync\/","title":{"rendered":"parallel rsync"},"content":{"rendered":"\n<p>rsync remains my main tool for transferring backups or just moving data between servers. but it has some pain points &#8211; e.g. rsync&#8217;s checksum calculation or ssh over which data is piped can easily saturate single CPU core before i run out of storage I\/O or network bandwidth. <\/p>\n\n\n\n<p>how to parallelize it &#8211; based on <a href=\"https:\/\/getbetter.ro\/tech\/rsync-tricks\/\">this<\/a> blog post:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#!\/bin\/bash -e\r\n\r\nif &#91; $# -ne 2 ] ; then\r\n        echo \"syntax: parallel-rsync.sh src dst\"\r\n        exit 2\r\nfi\r\ntmpdir=$(mktemp -d \/tmp\/parallel-rsync.XXXXXXXXXXX)\r\necho \"using $tmpdir\"\r\nrsync --archive --verbose --partial --progress --dry-run --itemize-changes \"$1\" \"$2\"|grep -E '^&lt;' |cut -d\" \" -f2 | split - -n r\/8 \"$tmpdir\/list\"\r\nls \"$tmpdir\/list\"* | parallel --lb -t -j 8 rsync --archive --verbose --partial --progress --files-from {}  \"$1\" \"$2\"\r\nrsync --archive --verbose --progress \"$1\" \"$2\"\r\nrm -rf \"$tmpdir\"\r\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>rsync remains my main tool for transferring backups or just moving data between servers. but it has some pain points &#8211; e.g. rsync&#8217;s checksum calculation or ssh over which data is piped can easily saturate single CPU core before i run out of storage I\/O or network bandwidth. how to parallelize it &#8211; based on [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[51],"tags":[56],"class_list":["post-3560","post","type-post","status-publish","format-standard","hentry","category-unimportant","tag-rsync"],"_links":{"self":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3560","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/comments?post=3560"}],"version-history":[{"count":1,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3560\/revisions"}],"predecessor-version":[{"id":3561,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3560\/revisions\/3561"}],"wp:attachment":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/media?parent=3560"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/categories?post=3560"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/tags?post=3560"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}