{"id":3472,"date":"2023-04-12T18:00:43","date_gmt":"2023-04-12T17:00:43","guid":{"rendered":"https:\/\/kudzia.eu\/b\/?p=3472"},"modified":"2023-04-22T17:06:46","modified_gmt":"2023-04-22T16:06:46","slug":"playing-with-min-io-cluster","status":"publish","type":"post","link":"https:\/\/kudzia.eu\/b\/2023\/04\/playing-with-min-io-cluster\/","title":{"rendered":"playing with min.io cluster"},"content":{"rendered":"\n<p><em>min.io<\/em> is open source [ although AGPL-3 licensed ] object storage software providing s3-compatible interface and handling clustering for HA &amp; data distribution. below &#8211; notes from setting it up on a tiny scale, without using Kubernetes.<\/p>\n\n\n<a class=\"wp-block-read-more\" href=\"https:\/\/kudzia.eu\/b\/2023\/04\/playing-with-min-io-cluster\/\" target=\"_self\">Read more<span class=\"screen-reader-text\">: playing with min.io cluster<\/span><\/a>\n\n\n<p>notes are based on RELEASE.2023-03-24T21-41-23Z<\/p>\n\n\n\n<p>i&#8217;ve fetched two binaries for the project:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>main server &#8211; <em>minio<\/em> &#8211; from https:\/\/github.com\/minio\/minio\/<\/li>\n\n\n\n<li>command line management tool &#8211; <em>mc<\/em> &#8211; https:\/\/github.com\/minio\/mc ; naming is unfortunate and overlaps with midnight commander<\/li>\n<\/ul>\n\n\n\n<p>my cluster has 4 nodes, looks like it&#8217;s the <a href=\"https:\/\/min.io\/docs\/minio\/linux\/operations\/checklists\/hardware.html\">recommended minimum<\/a> [ although i did ignore other suggestions e.g. for RAM or number of storage disks ].<\/p>\n\n\n\n<p>each node got its own DNS entry: mi0.kudzia.eu, mi1.kudzia.eu, mi2.kudzia.eu, mi3.kudzia.eu. i&#8217;ve also made firewall exception allowing unfiltered bidirectional communication between the nodes on TCP ports 9000-9001.<\/p>\n\n\n\n<p>min.io refuses to start if the folder for the data storage is part of the \/ mount, so i&#8217;m doing a dirty workaround:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dd if=\/dev\/zero of=\/d0.img bs=1GB count=5\nmkfs.ext4 \/d0.img\nmount \/d0.img \/mnt\/<\/code><\/pre>\n\n\n\n<p>on each of the nodes i&#8217;ve run the following command to start the server:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>export MINIO_ROOT_USER=admin\nexport MINIO_ROOT_PASSWORD=supersecretpassword\nminio server http:\/\/mi{0..3}.kudzia.eu\/mnt\/ --console-address :9001<\/code><\/pre>\n\n\n\n<p>this started a clean server. i&#8217;ve connected to it via the web management console &#8211; e.g. http:\/\/mi0.kudzia.eu:9001\/ ; there i:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>create a new bucket for data storage:\n<ul class=\"wp-block-list\">\n<li>select <em>Buckets <\/em>from the left menu,<\/li>\n\n\n\n<li>then <em>Create buckets<\/em>,<\/li>\n\n\n\n<li>give it name <em>bucket0<\/em> and leave options for <em>versioning<\/em>, <em>object locking<\/em> and <em>quota<\/em> disabled,<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>create a new policy giving users read\/write access to the bucket0:\n<ul class=\"wp-block-list\">\n<li>select <em>Policies<\/em> from the left menu,<\/li>\n\n\n\n<li><em>Create policy<\/em>,<\/li>\n\n\n\n<li>call it <em>bucket0_rw<\/em><\/li>\n\n\n\n<li>add content:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": &#91;\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": &#91;\n                \"s3:DeleteObject\",\n                \"s3:ListBucket\",\n                \"s3:ListBucketMultipartUploads\",\n                \"s3:ListMultipartUploadParts\",\n                \"s3:PutObject\",\n                \"s3:AbortMultipartUpload\",\n                \"s3:GetObject\",\n                \"s3:GetObjectLegalHold\",\n                \"s3:GetObjectRetention\"\n            ],\n            \"Resource\": &#91;\n                \"arn:aws:s3:::bucket0\",\n                \"arn:aws:s3:::bucket0\/*\"\n            ]\n        }\n    ]\n}<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>create a new user and API keys for it:\n<ul class=\"wp-block-list\">\n<li>select <em>Identity &gt; Users<\/em> from the left menu,<\/li>\n\n\n\n<li><em>Create user<\/em>,<\/li>\n\n\n\n<li>call it <em>user0<\/em>,<\/li>\n\n\n\n<li>assign policy <em>bucket0_rw<\/em><\/li>\n\n\n\n<li>save it without assigning any policies,<\/li>\n\n\n\n<li>click on the newly created user and then <em>Service accounts<\/em> and create a new API key<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>time to test it from command line ; i&#8217;ll start with <a href=\"https:\/\/rclone.org\/\">rclone<\/a> from linux command line.<\/p>\n\n\n\n<p>put this in .config\/rclone\/rclone.conf:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;minio]\ntype = s3\nprovider = Other\nenv_auth = false\naccess_key_id = ... access key id created above ...\nsecret_access_key = ... secret created in the same step ...\nendpoint = https:\/\/mi0.kudzia.eu:443\/\nacl = private\n<\/code><\/pre>\n\n\n\n<p>let&#8217;s try it:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@mi0:~# rclone ls minio:bucket0\/<\/code><\/pre>\n\n\n\n<p>there&#8217;s nothing &#8211; that&#8217;s expected; let&#8217;s try to upload a file:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@mi0:~# rclone --s3-no-check-bucket  copyto \/etc\/motd minio:bucket0\/motd.txt<\/code><\/pre>\n\n\n\n<p>and now list the bucket again:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@mi0:~# rclone ls minio:bucket0\/\n      286 motd.txt\n<\/code><\/pre>\n\n\n\n<p>why the <em>&#8211;s3-no-check-bucket <\/em>option? apparently &#8211; we don&#8217;t want the call to attempt creating a bucket if it does not exist yet, policy does not provide our user to do it. without it you&#8217;ll get:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@mi0:~# rclone copyto \/etc\/motd minio:bucket0\/motd.txt \n2023\/04\/12 16:40:53 ERROR : motd: Failed to copy: AccessDenied: Access Denied. \n        status code: 403, request id: 17553D7F5756899A, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 \n2023\/04\/12 16:40:53 ERROR : Attempt 1\/3 failed with 1 errors and: AccessDenied: Access Denied. \n        status code: 403, request id: 17553D7F5756899A, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 \n2023\/04\/12 16:40:53 ERROR : motd: Failed to copy: AccessDenied: Access Denied. \n        status code: 403, request id: 17553D7F57957FAA, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 \n2023\/04\/12 16:40:53 ERROR : Attempt 2\/3 failed with 1 errors and: AccessDenied: Access Denied. \n        status code: 403, request id: 17553D7F57957FAA, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 \n2023\/04\/12 16:40:53 ERROR : motd: Failed to copy: AccessDenied: Access Denied. \n        status code: 403, request id: 17553D7F57CFA46B, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 \n2023\/04\/12 16:40:53 ERROR : Attempt 3\/3 failed with 1 errors and: AccessDenied: Access Denied. \n        status code: 403, request id: 17553D7F57CFA46B, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 \n2023\/04\/12 16:40:53 Failed to copyto: AccessDenied: Access Denied. \n        status code: 403, request id: 17553D7F57CFA46B, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 <\/code><\/pre>\n\n\n\n<p>how about winscp? i had to change in <em>advanced <\/em>&gt; <em>environment <\/em>&gt; <em>s3<\/em> &gt;<em> url style <\/em>: <em>path<\/em><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p>exporting current configuration [ e.g. policies, users etc ]:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>mc admin cluster iam export myminio<\/code><\/pre>\n\n\n\n<p>useful links:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/min.io\/docs\/minio\/linux\/index.html\">official documentation<\/a>,<\/li>\n\n\n\n<li><a href=\"https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-set-up-an-object-storage-server-using-minio-on-ubuntu-18-04\">tutorial on digital ocean&#8217;s community docs<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/news.ycombinator.com\/item?id=32149340\">hackernews discussion mentioning min.io alternatives<\/a> <\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>min.io is open source [ although AGPL-3 licensed ] object storage software providing s3-compatible interface and handling clustering for HA &amp; data distribution. below &#8211; notes from setting it up on a tiny scale, without using Kubernetes. notes are based on RELEASE.2023-03-24T21-41-23Z i&#8217;ve fetched two binaries for the project: my cluster has 4 nodes, looks [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-3472","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3472","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/comments?post=3472"}],"version-history":[{"count":16,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3472\/revisions"}],"predecessor-version":[{"id":3497,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3472\/revisions\/3497"}],"wp:attachment":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/media?parent=3472"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/categories?post=3472"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/tags?post=3472"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}