playing with min.io cluster

min.io is open source [ although AGPL-3 licensed ] object storage software providing s3-compatible interface and handling clustering for HA & data distribution. below – notes from setting it up on a tiny scale, without using Kubernetes.

notes are based on RELEASE.2023-03-24T21-41-23Z

i’ve fetched two binaries for the project:

main server – minio – from https://github.com/minio/minio/
command line management tool – mc – https://github.com/minio/mc ; naming is unfortunate and overlaps with midnight commander

my cluster has 4 nodes, looks like it’s the recommended minimum [ although i did ignore other suggestions e.g. for RAM or number of storage disks ].

each node got its own DNS entry: mi0.kudzia.eu, mi1.kudzia.eu, mi2.kudzia.eu, mi3.kudzia.eu. i’ve also made firewall exception allowing unfiltered bidirectional communication between the nodes on TCP ports 9000-9001.

min.io refuses to start if the folder for the data storage is part of the / mount, so i’m doing a dirty workaround:

dd if=/dev/zero of=/d0.img bs=1GB count=5
mkfs.ext4 /d0.img
mount /d0.img /mnt/

on each of the nodes i’ve run the following command to start the server:

export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=supersecretpassword
minio server http://mi{0..3}.kudzia.eu/mnt/ --console-address :9001

this started a clean server. i’ve connected to it via the web management console – e.g. http://mi0.kudzia.eu:9001/ ; there i:

create a new bucket for data storage:
- select Buckets from the left menu,
- then Create buckets,
- give it name bucket0 and leave options for versioning, object locking and quota disabled,
create a new policy giving users read/write access to the bucket0:
- select Policies from the left menu,
- Create policy,
- call it bucket0_rw
- add content:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:GetObjectLegalHold",
                "s3:GetObjectRetention"
            ],
            "Resource": [
                "arn:aws:s3:::bucket0",
                "arn:aws:s3:::bucket0/*"
            ]
        }
    ]
}

create a new user and API keys for it:
- select Identity > Users from the left menu,
- Create user,
- call it user0,
- assign policy bucket0_rw
- save it without assigning any policies,
- click on the newly created user and then Service accounts and create a new API key

time to test it from command line ; i’ll start with rclone from linux command line.

put this in .config/rclone/rclone.conf:

[minio]
type = s3
provider = Other
env_auth = false
access_key_id = ... access key id created above ...
secret_access_key = ... secret created in the same step ...
endpoint = https://mi0.kudzia.eu:443/
acl = private

let’s try it:

root@mi0:~# rclone ls minio:bucket0/

there’s nothing – that’s expected; let’s try to upload a file:

root@mi0:~# rclone --s3-no-check-bucket  copyto /etc/motd minio:bucket0/motd.txt

and now list the bucket again:

root@mi0:~# rclone ls minio:bucket0/
      286 motd.txt

why the –s3-no-check-bucket option? apparently – we don’t want the call to attempt creating a bucket if it does not exist yet, policy does not provide our user to do it. without it you’ll get:

root@mi0:~# rclone copyto /etc/motd minio:bucket0/motd.txt 
2023/04/12 16:40:53 ERROR : motd: Failed to copy: AccessDenied: Access Denied. 
        status code: 403, request id: 17553D7F5756899A, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 
2023/04/12 16:40:53 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: Access Denied. 
        status code: 403, request id: 17553D7F5756899A, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 
2023/04/12 16:40:53 ERROR : motd: Failed to copy: AccessDenied: Access Denied. 
        status code: 403, request id: 17553D7F57957FAA, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 
2023/04/12 16:40:53 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: Access Denied. 
        status code: 403, request id: 17553D7F57957FAA, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 
2023/04/12 16:40:53 ERROR : motd: Failed to copy: AccessDenied: Access Denied. 
        status code: 403, request id: 17553D7F57CFA46B, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 
2023/04/12 16:40:53 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: Access Denied. 
        status code: 403, request id: 17553D7F57CFA46B, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 
2023/04/12 16:40:53 Failed to copyto: AccessDenied: Access Denied. 
        status code: 403, request id: 17553D7F57CFA46B, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

how about winscp? i had to change in advanced > environment > s3 > url style : path

exporting current configuration [ e.g. policies, users etc ]:

mc admin cluster iam export myminio

useful links:

Leave a Reply Cancel reply