{"id":2729,"date":"2017-04-29T17:51:23","date_gmt":"2017-04-29T16:51:23","guid":{"rendered":"https:\/\/kudzia.eu\/b\/?p=2729"},"modified":"2017-04-29T17:51:23","modified_gmt":"2017-04-29T16:51:23","slug":"ghettovcb-failing-randomly-on-larger-vms","status":"publish","type":"post","link":"https:\/\/kudzia.eu\/b\/2017\/04\/ghettovcb-failing-randomly-on-larger-vms\/","title":{"rendered":"ghettoVCB failing randomly on larger VMs"},"content":{"rendered":"<p>at work we&#8217;re using happily <a href=\"https:\/\/github.com\/lamw\/ghettoVCB\">ghettoVCB.sh<\/a> to back up and restore VMWare ESXi VMs. since a few weeks we&#8217;ve started to experience occasional failures of backups, only for one &#8211; larger VMs.<\/p>\n<p>in the logs produced in \/tmp\/ghettoVCB-2017-04-xxx.log we got:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nOption --adaptertype is deprecated and hence will be ignored Destination disk format: sparse with 2GB maximum extent size Cloning disk '\/vmfs\/volumes\/datastore1\/serverName\/server.vmdk'...\r\nClone: 18% done.Failed to clone disk: Failed to lock the file (16392).\r\n<\/pre>\n<p>or<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nOption --adaptertype is deprecated and hence will be ignored Destination disk format: sparse with 2GB maximum extent size Cloning disk '\/vmfs\/volumes\/datastore1\/serverName\/server.vmdk'...\r\nClone: 79% done.Failed to clone disk: Connection timed out (7208969).\r\n<\/pre>\n<p>after some head scratching, watching at <i>iostat -x 1<\/i> and <i>ifstat -b 1 -i eth0<\/i> on the nefs server that was receiving the file transfer done by the ghettoVCB i&#8217;ve realized this:<br \/>\n the receiving linux box would first accept all of the data sent over the network without flushing it early to the disk, then &#8211; once the RAM was filled up &#8211; it&#8217;d slow down and eventually stop accepting more NFS writes while writing down the data from RAM to disk. in the meanwhile vmware&#8217;s disk cloneing operation would time out.<\/p>\n<p>stupid me&#8230; i was earlier playing with \/proc\/sys\/vm\/dirty_expire_centisecs and left it at a very high level of 300000. i&#8217;ve also &#8216;tuned&#8217; dirty_ratio and dirty_background_ratio in a naive hope of limiting disk IO caused by some backups that would involve receiving the data over NFS, compressing it and then running rdiff-backup on the bzip2&#8217;ed files.<\/p>\n<p>after reverting values for those three parameters my problems are gone.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>at work we&#8217;re using happily ghettoVCB.sh to back up and restore VMWare ESXi VMs. since a few weeks we&#8217;ve started to experience occasional failures of backups, only for one &#8211; larger VMs. in the logs produced in \/tmp\/ghettoVCB-2017-04-xxx.log we got: Option &#8211;adaptertype is deprecated and hence will be ignored Destination disk format: sparse with 2GB [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[77],"class_list":["post-2729","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-backups"],"_links":{"self":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/2729","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/comments?post=2729"}],"version-history":[{"count":3,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/2729\/revisions"}],"predecessor-version":[{"id":2732,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/2729\/revisions\/2732"}],"wp:attachment":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/media?parent=2729"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/categories?post=2729"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/tags?post=2729"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}