Configure distributed Minio in multiple jail
Configure distributed Minio in multiple jail
In the last period I’ve done a lot of experiments, on my FreeBSD powered nuc, like configuring the DNS server, deploy Homeassistant (I will also write about it), installed nginx-plus (I will also write an article on how-to obtain a development/home license), and last but not least deploy a Minio cluster.
Ok, a Minio cluster on a single machine probably does not have any sense but I want to try how to configure it in order to understand it works.
Regarding MinIO is an implementation of the s3 API, which are a standard de-facto for object storage APIs, highly scalable and resilient. It allows also to host a static website or a dynamic one if the framework allows it (for instance vuejs).
Configure the environment
Minio for distributed deployment requires at least 4 drives up and running distributed among any number of replicas, in my deployment I’ve one drive per replica.
After the deployment of four different jails (see the Define Jail section in this article), you have also to allocate space for the jails using zfs dataset or whatever you prefer; for instance I’ve settled up a different dataset from each zpool and mounted in each jail on the /data path, to achieve this a dataset in a pool must be created running zfs create -o mountpoint=/mnt/minio-{1,2,3,4} <zpool>/<dataset> and then mapped to the relative jail fstab:
/usr/ports /jails/<jail>/usr/ports nullfs rw 0 0
/mnt/minio-{1,2,3,4} /jails/<jail>/data nullfs rw 0 0
After you have to install MinIO in each of them, for instance (assuming that each jail is called minio-x, where x is a number):
# Assuming that you are using tcsh
foreach jail `jls -v name | grep minio*`
jexec $jail pkg bootstrap; pkg upgrade -y; pkg install -y minio;
end
Then you have either to put each jail ip in the /etc/hosts with a symbolic name or register each jail ip in the dns (look also to the guide on How to deploy a jailed DNS server).
Now we can configure and run the MinIO service on each jail, I will explain the procedure for just one but they must be repeated for each jail (or even node if you are running a true distributed setup).
First of all we have to enable the minio services in /etc/rc.conf plus other variables to configure the running services:
minio_enable="YES"
minio_disks="http://<dns-or-ip>:<port>/<path-to-minio-folder> http://<dns-or-ip>:<port>/<path-to-minio-folder> http://<dns-or-ip>:<port>/<path-to-minio-folder> http://<dns-or-ip>:<port>/<path-to-minio-folder>"
minio_env="MINIO_ACCESS_KEY=<access-key> MINIO_SECRET_KEY=<secret-key> MINIO_REGION_NAME=<region>"
In the minio_disks section you have to specify each jail/node with the absolute path to the controlled disk (you have to specify multiple times a node if it has different drives under its control).
The minio_env section instead allows to specify multiple environment variables to configure properly the instances; I’ve specified the essential variables but there are more in the documentation.
Now we have to create log file and give it appropriate ownership, the log file must be located in /var/log/minio.log and must belong to the user and group minio which will be created during the package installation; also be sure that the selected path for the minio files belongs to the minio user and group.
After these configurations we can spin up the service and configure the other jails/nodes through service minio start, it will wait for other services to spun up and then will be available.
Accessing the cluster
To access the cluster there are 2 possibilities:
- Direct connection to one node/jail ip address:
http://<dns-or-ip>:<port> - Setup a loadbalancer/reverse proxy (with SSL termination eventually).
The first option is pretty simple, the minio specified port must be forwarded or opened to the external world (these option could be also the default if you are working with multiple network-exposed nodes), but in my case I’ve four jail inside the same hosts without any minio port exposed so I’ve opted in for the second solution.
I had previously settled up a nginx-plus jail (I will write an article on how to obtain a home/development plus license), and also settled up a jail with a CA (another future article) so the cluster is internally exposed with a TLS termination.
Regarding the nginx config file this is the result the one that I’m using (with parametrization):
upstream minio {
zone minio 128k;
server <minio-1>.<internal-domain>.<internal-gtld>:<port>
server <minio-2>.<internal-domain>.<internal-gtld>:<port>;
server <minio-3>.<internal-domain>.<internal-gtld>:<port>;
server <minio-4>.<internal-domain>.<internal-gtld>:<port>
}
server {
listen 80;
server_name <ext-name>.<internal-domain>.<internal-gtld>;
return 301 https://$host$request_uri;
}
server{
listen 443 ssl http2;
server_name <ext-name>.<internal-domain>.<internal-gtld>;
ssl_certificate <ca-obtained-cert-path>;
ssl_certificate_key <ca-obtained-key-path>;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_dhparam <path-to-nginx-dhparam>;
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
server_tokens off;
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
client_max_body_size 1000M; # Expanded body size
location / {
proxy_set_header Host $http_host;
proxy_pass http://minio;
health_check uri=/minio/health/live interval=10 fails=3 passes=2; # This keyword is available only with nginx-plus
}
}
Conclusion and future experiments
The cluster is pretty much stable, I’m currently using it for internal backups of non-critical stuff (for the critical one I’ve my Synology NAS). For further experiments on my cluster I’m curious about 3 different things:
- Multitenancy (doc)
- Event publishing in the distributed mode (if is possible)
- Complex website hosting (I’m not a frontend developer XD)