Victoria metrics on Aws EC2 instance

Will configure one single EC2 instance as a Victoria Metrics server to be used as Promethues storage.

The access to VM(victoria metrics) is done via port 8247 and is protected by http basic auth. All traffic is encrypted with a self sign certificate.

Installation

Will install manually by downloading the releases from github and configure the local system.

Download binaries

# create a group and user for vm
$ sudo groupadd -r victoriametrics
$ sudo useradd -g victoriametrics victoriametrics
 
# download
$ curl -L https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.70.0/victoria-metrics-amd64-v1.70.0.tar.gz --output victoria-metrics-amd64-v1.70.0.tar.gz

# unpack and install it
$ sudo tar xvf victoria-metrics-amd64-v1.70.0.tar.gz -C /usr/local/bin/
$ chown root:root /usr/local/bin/victoria-metrics-prod

# create data directory
$ sudo mkdir /var/lib/victoria-metrics-data
$ chown -v victoriametrics:victoriametrics /var/lib/victoria-metrics-data

Configure the service

cat >> /etc/systemd/system/victoriametrics.service <<EOF
[Unit]
Description=High-performance, cost-effective and scalable time series database, long-term remote storage for Prometheus
After=network.target

[Service]
Type=simple
User=victoriametrics
Group=victoriametrics
StartLimitBurst=5
StartLimitInterval=0
Restart=on-failure
RestartSec=1
ExecStart=/usr/local/bin/victoria-metrics-prod \
        -storageDataPath=/var/lib/victoria-metrics-data \
        -httpListenAddr=127.0.0.1:8428 \
        -retentionPeriod=1
ExecStop=/bin/kill -s SIGTERM $MAINPID
LimitNOFILE=65536
LimitNPROC=32000

[Install]
WantedBy=multi-user.target

EOF

At this point your can start the service systemctl enable victoriametrics.service --now, however the port 8428 is not protected in any way nor is encrypted so will add basic authentication and tls encryption with a self sign certificate, any valid certificate will work however. Note that listens only on localhost.

Vmauth

To protect the service will use vmauth which is part of a tool set released by victoria metrics.

# download and install the vm utils

$ curl -L https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.70.0/vmutils-amd64-v1.70.0.tar.gz --output vmutils-amd64-v1.70.0.tar.gz
$ sudo tar xvf vmutils-amd64-v1.70.0.tar.gz -C /usr/local/bin/
$ chown -v root:root /usr/local/bin/vm*-prod

Configure vmauth

Create a config file (config.yml) to enable basic authentication.

The format of the file is simple, you need a username and a password.

$ sudo mkdir -p /etc/victoriametrics/ssl/
$ sudo chown -vR victoriametrics:victoriametrics /etc/victoriametrics
$ sudo touch /etc/victoriametrics/config.yml
$ sudo chown -v victoriametrics:victoriametrics /etc/victoriametrics/config.yml

# generate a password for our user
$ python3  -c 'import secrets; print(secrets.token_urlsafe())'
KGKK_NoiciEMn6KdBk6CkcLHZt6TpB-Cgt12UFqnutU

# wite the config
$ sudo cat >> /etc/victoriametrics/config.yml <<EOF
> users:
>   - username: "user1"
>     password: "KGKK_NoiciEMn6KdBk6CkcLHZt6TpB-Cgt12UFqnutU"
>     url_prefix: "http://127.0.0.1:8428"
> # end config
> EOF

Install a self sign certificate

$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout /etc/victoriametrics/ssl/victoriametrics.key -out /etc/victoriametrics/ssl/victoriametrics.crt

$ sudo chown -Rv victoriametrics:victoriametrics /etc/victoriametrics/ssl/

Enable vmauth service

cat >> /etc/systemd/system/vmauth.service <<EOF
[Unit]
Description=Simple auth proxy, router and load balancer for VictoriaMetrics
After=network.target

[Service]
Type=simple
User=victoriametrics
Group=victoriametrics
StartLimitBurst=5
StartLimitInterval=0
Restart=on-failure
RestartSec=1
ExecStart=/usr/local/bin/vmauth-prod \
        --tls=true \
        --auth.config=/etc/victoriametrics/config.yml \
        --httpListenAddr=0.0.0.0:8247 \
        --tlsCertFile=/etc/victoriametrics/ssl/victoriametrics.crt \
        --tlsKeyFile=/etc/victoriametrics/ssl/victoriametrics.key \
ExecStop=/bin/kill -s SIGTERM $MAINPID
LimitNOFILE=65536
LimitNPROC=32000

[Install]
WantedBy=multi-user.target


EOF

Start and enable systemctl enable vmauth.service --now .

To test you will need first to construct a base64 string from the username and password you have written into the config.ymlfile.

For example user vmuser and password secret

$ echo -n 'vmuser:secret' | base64
$ dm11c2VyOnNlY3JldA==

# to test vmauth
$ curl -H 'Authorization: Basic dm11c2VyOnNlY3JldA==' --insecure https://localhost:8247/api/v1/query -d 'query={job=~".*"}'

Operations

Snaphots

List what’s available

curl 'https://localhost:8247/snapshot/list'

{"status":"ok","snapshots":["20211227145126-16C1DDB61673BA11"

Create a new snapshot

curl 'https://localhost:8247/snapshot/create'

{"status":"ok","snapshot":"20211227145526-16C1DDB61673BA12"}

List again the snapshots

curl -s 'https://localhost:8247/snapshot/list' | jq .
{
  "status": "ok",
  "snapshots": [
    "20211227145126-16C1DDB61673BA11",
    "20211227145526-16C1DDB61673BA12"
  ]
}

Backups

The snapshots are located on local disk under data path (parameter -storageDataPath=) on my instance it resolves to storageDataPath=/var/lib/victoria-metrics-data/.

The data into snapshots is compressed with Zstandard.

To push the backups to s3 you can use vmbackup.

$ sudo vmbackup-prod -storageDataPath=/var/lib/victoria-metrics-data  -snapshotName=20211227145526-16C1DDB61673BA12 -dst=s3://BUCKET-NAME/`date +%s`

...

2021-12-29T16:07:20.571Z        info    VictoriaMetrics/app/vmbackup/main.go:105        gracefully shutting down http server for metrics at ":8420"
2021-12-29T16:07:20.572Z        info    VictoriaMetrics/app/vmbackup/main.go:109        successfully shut down http server for metrics in 0.001 seconds

For more info you can see vmbackup.

Postgresql locks

Locks in postgres

Find locks

select pid, state, usename, query, query_start 
from pg_stat_activity 
where pid in (
  select pid from pg_locks l 
  join pg_class t on l.relation = t.oid 
  and t.relkind = 'r' 
  where t.relname = 'search_hit'
);

Killing locks

SELECT pg_cancel_backend(PID);

Haproxy socket stats

Enable stats

Reporting is provided if you enable stats into its config.

The setting is described at https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-stats%20enable

In this post I describe how to use the socket type.

Enable the stats socket

I enable it into the global section as so

global

  stats socket /var/lib/haproxy/stats group haproxy mode 664

What this does is:

• enable the stats socket under /var/lib/haproxy/stats
• the group owner is haproxy (running haproxy as user haproxy)
• permissions are rw (user), rw(group), r(others)

Note there is an option admin that will allow to control haproxy but I don’t use it.

Reading stats from socket (netcat)

You need to have installed netcat (nc).

$ echo 'show stat' | nc -U /var/lib/haproxy/stats
# pxname,svname,qcur,qmax,scur,smax,slim,
....
http_frontend,
....

Reading stats from socket (socat)

You need to install socat since is not frequently installed.

To use it

$ echo 'show stat' | socat stdio /var/lib/haproxy/stats
# pxname,svname,qcur,qmax,scur,smax,slim,
....
http_frontend,
....