{"id":3012,"date":"2019-09-26T18:41:27","date_gmt":"2019-09-26T17:41:27","guid":{"rendered":"https:\/\/kudzia.eu\/b\/?p=3012"},"modified":"2019-09-26T08:42:26","modified_gmt":"2019-09-26T07:42:26","slug":"iostat-x-1-reporting-100-utilization-of-nearly-idle-nvme-drives","status":"publish","type":"post","link":"https:\/\/kudzia.eu\/b\/2019\/09\/iostat-x-1-reporting-100-utilization-of-nearly-idle-nvme-drives\/","title":{"rendered":"iostat -x 1 reporting 100% utilization of nearly-idle NVMe drives"},"content":{"rendered":"<p>after an upgrade to debian buster i&#8217;ve noticed that both <i>iostat -x 1<\/i> and munin&#8217;s diskstats_utilization report that NVMe drives are busy most of the time. some empirical tests showed that disks are actually idle, performance did not drop.<\/p>\n<p>upgrade to 5.2 kernel resolved the miss-reporting issue.<\/p>\n<p><!--more--><\/p>\n<p>some related bug reports:<br \/>\nhttps:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=927184<br \/>\nhttps:\/\/bugs.centos.org\/view.php?id=15723<br \/>\nhttps:\/\/github.com\/sysstat\/sysstat\/issues\/187<br \/>\nhttps:\/\/github.com\/munin-monitoring\/munin\/issues\/1119<br \/>\nhttps:\/\/unix.stackexchange.com\/questions\/517667\/nvme-disk-shows-80-io-utilization-partitions-show-0-io-utilization?noredirect=1&#038;lq=1<br \/>\nhttps:\/\/github.com\/netdata\/netdata\/issues\/5744#issuecomment-513873791<\/p>\n<p>so it&#8217;s a known, kernel-related issue.<\/p>\n<p>i&#8217;ve done a simple test &#8211; run single KVM guest with disk on top of NVMe storage. in the guest i&#8217;ve executed: <\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nwhile &#x5B; true ] ; do touch a ; sync ; sleep 0.5 ; done\r\n<\/pre>\n<p>and observed iostat -x on the host. with buster&#8217;s stock 4.19.0-6-amd64 kernel <i>iostat -x 1<\/i> reported constantly high utilisation:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n           0.02    0.00    0.02    0.00    0.00   99.96\r\n\r\nDevice            r\/s     w\/s     rkB\/s     wkB\/s   rrqm\/s   wrqm\/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util\r\nnvme0n1          0.00   14.00      0.00     50.00     0.00     0.00   0.00   0.00    0.00    0.07   0.86     0.00     3.57  61.14  85.60\r\nnvme1n1          0.00   14.00      0.00     50.00     0.00     0.00   0.00   0.00    0.00    0.07   0.86     0.00     3.57  61.14  85.60\r\nmd0              0.00   10.00      0.00     32.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     3.20   0.00   0.00\r\nsda              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\nsdb              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\nsdc              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\nscd0             0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\ndm-0             0.00   10.00      0.00     32.00     0.00     0.00   0.00   0.00    0.00    3.60   0.04     0.00     3.20   3.60   3.60\r\ndm-1             0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n           0.04    0.00    0.02    0.00    0.00   99.94\r\n\r\nDevice            r\/s     w\/s     rkB\/s     wkB\/s   rrqm\/s   wrqm\/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util\r\nnvme0n1          0.00   14.00      0.00     50.00     0.00     0.00   0.00   0.00    0.00    0.00   0.91     0.00     3.57  64.86  90.80\r\nnvme1n1          0.00   14.00      0.00     50.00     0.00     0.00   0.00   0.00    0.00    0.00   0.91     0.00     3.57  64.86  90.80\r\nmd0              0.00   10.00      0.00     32.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     3.20   0.00   0.00\r\nsda              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\nsdb              0.00    1.00      0.00     16.00     0.00     0.00   0.00   0.00    0.00    1.00   0.00     0.00    16.00   0.00   0.00\r\nsdc              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\nscd0             0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\ndm-0             0.00   10.00      0.00     32.00     0.00     0.00   0.00   0.00    0.00    3.60   0.04     0.00     3.20   3.60   3.60\r\ndm-1             0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\n<\/pre>\n<p>i&#8217;ve repeated the same test after upgrading the host to kernel from backports &#8211; 5.2.0-0.bpo.2-amd64 &#8211; problem was gone, %util looked reasonably:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n           0.04    0.00    0.02    0.00    0.00   99.94\r\n\r\nDevice            r\/s     w\/s     rkB\/s     wkB\/s   rrqm\/s   wrqm\/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util\r\nnvme0n1          0.00   14.00      0.00     50.00     0.00     0.00   0.00   0.00    0.00    0.07   0.00     0.00     3.57   1.14   1.60\r\nnvme1n1          0.00   14.00      0.00     50.00     0.00     0.00   0.00   0.00    0.00    0.07   0.00     0.00     3.57   1.14   1.60\r\nsda              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\nsdb              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\nmd0              0.00   10.00      0.00     32.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     3.20   0.00   0.00\r\nsdc              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\nscd0             0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\ndm-0             0.00   10.00      0.00     32.00     0.00     0.00   0.00   0.00    0.00    4.00   0.04     0.00     3.20   1.60   1.60\r\ndm-1             0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n           0.08    0.00    0.02    0.00    0.00   99.90\r\n\r\nDevice            r\/s     w\/s     rkB\/s     wkB\/s   rrqm\/s   wrqm\/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util\r\nnvme0n1          0.00   12.00      0.00     45.50     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     3.79   1.00   1.20\r\nnvme1n1          0.00   12.00      0.00     45.50     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     3.79   1.00   1.20\r\nsda              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\nsdb              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\nmd0              0.00   10.00      0.00     32.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     3.20   0.00   0.00\r\nsdc              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\nscd0             0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\ndm-0             0.00   10.00      0.00     32.00     0.00     0.00   0.00   0.00    0.00    3.20   0.03     0.00     3.20   1.60   1.60\r\ndm-1             0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>after an upgrade to debian buster i&#8217;ve noticed that both iostat -x 1 and munin&#8217;s diskstats_utilization report that NVMe drives are busy most of the time. some empirical tests showed that disks are actually idle, performance did not drop. upgrade to 5.2 kernel resolved the miss-reporting issue.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,51],"tags":[],"class_list":["post-3012","post","type-post","status-publish","format-standard","hentry","category-uncategorized","category-unimportant"],"_links":{"self":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3012","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/comments?post=3012"}],"version-history":[{"count":2,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3012\/revisions"}],"predecessor-version":[{"id":3014,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3012\/revisions\/3014"}],"wp:attachment":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/media?parent=3012"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/categories?post=3012"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/tags?post=3012"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}