{"id":3642,"date":"2024-08-10T17:22:00","date_gmt":"2024-08-10T16:22:00","guid":{"rendered":"https:\/\/kudzia.eu\/b\/?p=3642"},"modified":"2024-12-10T08:24:39","modified_gmt":"2024-12-10T07:24:39","slug":"new-supermicro-server-does-not-see-some-of-nvme-drives","status":"publish","type":"post","link":"https:\/\/kudzia.eu\/b\/2024\/08\/new-supermicro-server-does-not-see-some-of-nvme-drives\/","title":{"rendered":"New supermicro server does not &#8216;see&#8217; some of NVMe drives"},"content":{"rendered":"\n<p>i got 3 identical new servers from supermicro &#8211; <a href=\"https:\/\/www.supermicro.com\/en\/products\/system\/clouddc\/1u\/sys-121c-tn10r\">SYS-121C-TN10R<\/a>. all three have the same disk setup &#8211; 2+4 NVMe drives, but Debian 12 just see 5 of them instead of 6.<\/p>\n\n\n\n<p>when i run dmesg on newly installed system i can see those errors:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;    6.662992] pci 10000:80:01.0: BAR 13: no space for &#91;io  size 0x4000]\n&#91;    6.662994] pci 10000:80:01.0: BAR 13: failed to assign &#91;io  size 0x4000]\n&#91;    6.662995] pci 10000:80:03.0: BAR 13: no space for &#91;io  size 0x4000]\n&#91;    6.662996] pci 10000:80:03.0: BAR 13: failed to assign &#91;io  size 0x4000]\n&#91;    6.662999] pci 10000:80:01.0: BAR 13: no space for &#91;io  size 0x4000]\n&#91;    6.663000] pci 10000:80:01.0: BAR 13: failed to assign &#91;io  size 0x4000]\n&#91;    6.663001] pci 10000:80:03.0: BAR 13: no space for &#91;io  size 0x4000]\n&#91;    6.663002] pci 10000:80:03.0: BAR 13: failed to assign &#91;io  size 0x4000]\n\n<\/code><\/pre>\n\n\n\n<p><\/p>\n\n\n\n<p>to solve it i had to change PCIe bifurcation in BIOS: <\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>go to <em>Advanced<\/em> &gt; <em>Chipset configuration<\/em> &gt; <em>North bridge <\/em>&gt; <em>IIO Configuration<\/em>:\n<ul class=\"wp-block-list\">\n<li><em>CPU1 Configuration<\/em> &gt; <em>IIOU3<\/em> and change it from <em>Auto <\/em>to <em>x4x4x4x4<\/em><\/li>\n\n\n\n<li><em>CPU2 Configuration<\/em> &gt; <em>IIOU2<\/em> and change it from <em>Auto <\/em>to <em>x4x4x4x4<\/em><\/li>\n\n\n\n<li>since i was not using Intel&#8217;s not-exactly-hardware-RAID i&#8217;ve also went to <em>Intel VMD technology <\/em>&gt; <em>NVMe Mode Switch<\/em>: manual + disable for both CPUs<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>advanced &gt; chipset configuration &gt; north bridge &gt; IIO Configuration &gt; CPU1 Configuration &gt; IIOU3: x4x4x4x4, CPU2 Configuration &gt; IIOU2: x4x4x4x4, Intel VMD technology &gt; NVMe Mode Switch: manual + disable for both CPUs<\/li>\n<\/ul>\n\n\n\n<p>i&#8217;ll admit &#8211; it&#8217;s far from obvious and i got help from the datacenter to figure it out. in the <a href=\"https:\/\/www.supermicro.com\/manuals\/superserver\/1U\/MNL-2522.pdf\">user manual<\/a> for that motherboard i&#8217;ve found &#8216;BIOS IIO Mapping to Motherboard&#8217;s NVMe slots \/ PCIe lanes are connected:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"456\" src=\"https:\/\/kudzia.eu\/b\/wp-content\/uploads\/2024\/08\/image-1024x456.png\" alt=\"\" class=\"wp-image-3648\" srcset=\"https:\/\/kudzia.eu\/b\/wp-content\/uploads\/2024\/08\/image-1024x456.png 1024w, https:\/\/kudzia.eu\/b\/wp-content\/uploads\/2024\/08\/image-300x133.png 300w, https:\/\/kudzia.eu\/b\/wp-content\/uploads\/2024\/08\/image-768x342.png 768w, https:\/\/kudzia.eu\/b\/wp-content\/uploads\/2024\/08\/image.png 1216w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>perhaps it explains why those specific IOUs had to be reconfigured, but it&#8217;s far from obvious for me till now.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p>oh well &#8211; this time Dell wins &#8211; there, regardless to which slot i&#8217;ve plugged NVMes &#8211; they just have been visible and working without additional fiddling.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>i got 3 identical new servers from supermicro &#8211; SYS-121C-TN10R. all three have the same disk setup &#8211; 2+4 NVMe drives, but Debian 12 just see 5 of them instead of 6. when i run dmesg on newly installed system i can see those errors: to solve it i had to change PCIe bifurcation in [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-3642","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3642","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/comments?post=3642"}],"version-history":[{"count":4,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3642\/revisions"}],"predecessor-version":[{"id":3703,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3642\/revisions\/3703"}],"wp:attachment":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/media?parent=3642"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/categories?post=3642"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/tags?post=3642"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}