Will configure one single EC2 instance as a Victoria Metrics server to be used as Promethues storage.
The access to VM(victoria metrics) is done via port 8247 and is protected by http basic auth. All traffic is
encrypted with a self sign certificate.
Installation
Will install manually by downloading the releases from github and configure the local system.
Download binaries
# create a group and user for vm
$ sudo groupadd -r victoriametrics
$ sudo useradd -g victoriametrics victoriametrics
# download
$ curl -L https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.70.0/victoria-metrics-amd64-v1.70.0.tar.gz --output victoria-metrics-amd64-v1.70.0.tar.gz
# unpack and install it
$ sudo tar xvf victoria-metrics-amd64-v1.70.0.tar.gz -C /usr/local/bin/
$ chown root:root /usr/local/bin/victoria-metrics-prod
# create data directory
$ sudo mkdir /var/lib/victoria-metrics-data
$ chown -v victoriametrics:victoriametrics /var/lib/victoria-metrics-data
Configure the service
cat >> /etc/systemd/system/victoriametrics.service <<EOF
[Unit]
Description=High-performance, cost-effective and scalable time series database, long-term remote storage for Prometheus
After=network.target
[Service]
Type=simple
User=victoriametrics
Group=victoriametrics
StartLimitBurst=5
StartLimitInterval=0
Restart=on-failure
RestartSec=1
ExecStart=/usr/local/bin/victoria-metrics-prod \
-storageDataPath=/var/lib/victoria-metrics-data \
-httpListenAddr=127.0.0.1:8428 \
-retentionPeriod=1
ExecStop=/bin/kill -s SIGTERM $MAINPID
LimitNOFILE=65536
LimitNPROC=32000
[Install]
WantedBy=multi-user.target
EOF
At this point your can start the service systemctl enable victoriametrics.service --now, however the port 8428 is not
protected in any way nor is encrypted so will add basic authentication and tls encryption with a self sign certificate,
any valid certificate will work however. Note that listens only on localhost.
Vmauth
To protect the service will use vmauth which is part of a tool set released by victoria metrics.
# download and install the vm utils
$ curl -L https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.70.0/vmutils-amd64-v1.70.0.tar.gz --output vmutils-amd64-v1.70.0.tar.gz
$ sudo tar xvf vmutils-amd64-v1.70.0.tar.gz -C /usr/local/bin/
$ chown -v root:root /usr/local/bin/vm*-prod
Configure vmauth
Create a config file (config.yml) to enable basic authentication.
The format of the file is simple, you need a username and a password.
$ sudo mkdir -p /etc/victoriametrics/ssl/
$ sudo chown -vR victoriametrics:victoriametrics /etc/victoriametrics
$ sudo touch /etc/victoriametrics/config.yml
$ sudo chown -v victoriametrics:victoriametrics /etc/victoriametrics/config.yml
# generate a password for our user
$ python3 -c 'import secrets; print(secrets.token_urlsafe())'
KGKK_NoiciEMn6KdBk6CkcLHZt6TpB-Cgt12UFqnutU
# wite the config
$ sudo cat >> /etc/victoriametrics/config.yml <<EOF
> users:
> - username: "user1"
> password: "KGKK_NoiciEMn6KdBk6CkcLHZt6TpB-Cgt12UFqnutU"
> url_prefix: "http://127.0.0.1:8428"
> # end config
> EOF
Install a self sign certificate
$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout /etc/victoriametrics/ssl/victoriametrics.key -out /etc/victoriametrics/ssl/victoriametrics.crt
$ sudo chown -Rv victoriametrics:victoriametrics /etc/victoriametrics/ssl/
Enable vmauth service
cat >> /etc/systemd/system/vmauth.service <<EOF
[Unit]
Description=Simple auth proxy, router and load balancer for VictoriaMetrics
After=network.target
[Service]
Type=simple
User=victoriametrics
Group=victoriametrics
StartLimitBurst=5
StartLimitInterval=0
Restart=on-failure
RestartSec=1
ExecStart=/usr/local/bin/vmauth-prod \
--tls=true \
--auth.config=/etc/victoriametrics/config.yml \
--httpListenAddr=0.0.0.0:8247 \
--tlsCertFile=/etc/victoriametrics/ssl/victoriametrics.crt \
--tlsKeyFile=/etc/victoriametrics/ssl/victoriametrics.key \
ExecStop=/bin/kill -s SIGTERM $MAINPID
LimitNOFILE=65536
LimitNPROC=32000
[Install]
WantedBy=multi-user.target
EOF
Start and enable systemctl enable vmauth.service --now .
To test you will need first to construct a base64 string from the username and password you have written into the config.ymlfile.
For example user vmuser and password secret
$ echo -n 'vmuser:secret' | base64
$ dm11c2VyOnNlY3JldA==
# to test vmauth
$ curl -H 'Authorization: Basic dm11c2VyOnNlY3JldA==' --insecure https://localhost:8247/api/v1/query -d 'query={job=~".*"}'
Operations
Snaphots
List what’s available
curl 'https://localhost:8247/snapshot/list'
{"status":"ok","snapshots":["20211227145126-16C1DDB61673BA11"
Create a new snapshot
curl 'https://localhost:8247/snapshot/create'
{"status":"ok","snapshot":"20211227145526-16C1DDB61673BA12"}
List again the snapshots
curl -s 'https://localhost:8247/snapshot/list' | jq .
{
"status": "ok",
"snapshots": [
"20211227145126-16C1DDB61673BA11",
"20211227145526-16C1DDB61673BA12"
]
}
Backups
The snapshots are located on local disk under data path (parameter -storageDataPath=) on my instance
it resolves to storageDataPath=/var/lib/victoria-metrics-data/.
The data into snapshots is compressed with Zstandard.
To push the backups to s3 you can use vmbackup.
$ sudo vmbackup-prod -storageDataPath=/var/lib/victoria-metrics-data -snapshotName=20211227145526-16C1DDB61673BA12 -dst=s3://BUCKET-NAME/`date +%s`
...
2021-12-29T16:07:20.571Z info VictoriaMetrics/app/vmbackup/main.go:105 gracefully shutting down http server for metrics at ":8420"
2021-12-29T16:07:20.572Z info VictoriaMetrics/app/vmbackup/main.go:109 successfully shut down http server for metrics in 0.001 seconds
For more info you can see vmbackup.
