Databricks AWS private link with conditional DNS forwarder

Databricks allows workspace to be accessible via a private ip so not publicly available.
This is useful in some cases where you want to restrict users to used it only if are connected to a VPN or equivalent.
These are the steps to achive this goal

1. Create a custom VPC in AWS
2. Connect the VPC to your infrastructure (VPN/Direct Connect)
3. Create VPC Endpoint for frontend and backend
4. Register private link in the databricks account and associate it to your workspace
5. Change access mode to Private from Public in the Private Access Settings
6. Create a private zone in Route53
7. Create an inbound resolver in Route53
8. Add an A entry in the zone that points to the address of the VPC Endpoint (step3)
9. Add a forwarder in your Private DNS to point to the inbound resolver (step 7) ip address(es)

For more details Databricks documentation explains in more details docs.databricks.com (search for private link). Once all is in place the flow is

+---------+                               +-------------+                                            +-----------+ +-------------------------+                                         +-------------+
| Client  |                               | PrivateDns  |                                            | PublicDns | | Route53InboundResolver  |                                         | Route53Zone |
+---------+                               +-------------+                                            +-----------+ +-------------------------+                                         +-------------+
     |                                           |                                                         |                    |                                                             |
     | my-workspace.cloud.databricks.com         |                                                         |                    |                                                             |
     |------------------------------------------>|                                                         |                    |                                                             |
     |                                           |                                                         |                    |                                                             |
     |                                           | my-workspace.cloud.databricks.com                       |                    |                                                             |
     |                                           |-------------------------------------------------------->|                    |                                                             |
     |                                           |                                                         |                    |                                                             |
     |                                           |       CNAME nvirginia.privatelink.cloud.databricks.com. |                    |                                                             |
     |                                           |<--------------------------------------------------------|                    |                                                             |
     |                                           |                                                         |                    |                                                             |
     |                                           | nvirginia.privatelink.cloud.databricks.com ?            |                    |                                                             |
     |                                           |----------------------------------------------------------------------------->|                                                             |
     |                                           |                                                         |                    |                                                             |
     |                                           |                                                         |                    | nvirginia.privatelink.cloud.databricks.com ?                |
     |                                           |                                                         |                    |------------------------------------------------------------>|
     |                                           |                                                         |                    |                                                             |
     |                                           |                                                         |                    |                                      Address is 172.16.0.10 |
     |                                           |                                                         |                    |<------------------------------------------------------------|
     |                                           |                                                         |                    |-----------------------------------------------------------\ |
     |                                           |                                                         |                    || nvirginia.privatelink.cloud.databricks.com A 172.16.0.10 |-|
     |                                           |                                                         |                    ||----------------------------------------------------------| |
     |                                           |                                                         |                    |                                                             |
     |                                           |                                                Answer address is 172.16.0.10 |                                                             |
     |                                           |<-----------------------------------------------------------------------------|                                                             |
     |                                           |                                                         |                    |                                                             |
     |                    Connect to 172.16.0.10 |                                                         |                    |                                                             |
     |<------------------------------------------|                                                         |                    |                                                             |
     |                                           |                                                         |                    |                                                             |

Remove Windows 10 drivers from command line

Normally you would use System Settings -> Apps & features ... but that didn't work for you so this will explain how to uninstall a driver from command line.
First if you searched and tried to remove the files from a location as C:\windows\system32\driverstore\FileRepository\ you might notice that is not possible since you need SYSTEM access.
To know what driver you want to remove you will need to list them.
Open a command prompt or powershell as Administrator than

dism /online /get-drivers /format:table > c:\drivers.txt

Open the file c:\drivers.txt and note the Published Name as per

Version: 10.0.19041.844

Image Version: 10.0.19045.3448

Obtaining list of 3rd party drivers from the driver store...

Driver packages listing:


-------------- | ----------------------------- | ----- | -------------------- | ---------------------------- | ---------- | ----------------
Published Name | Original File Name            | Inbox | Class Name           | Provider Name                | Date       | Version         
-------------- | ----------------------------- | ----- | -------------------- | ---------------------------- | ---------- | ----------------
oem77.inf      | nxdrv.inf                     | No    | Net                  | SonicWall                    | 10/18/2017 | 2.0.6.1         

Now to remove the driver take note of its Published Name as above list.

pnputil.exe /d oem77.inf

That's it.
Other tools that can help are https://learn.microsoft.com/en-us/sysinternals/downloads/autoruns though I found that would work most of the time but in some cases will not be able to remove the driver files but just registry cleanup.

Tmux cheetsheet

Attach and detach

$ tmux Start new tmux session
$ tmux attach Attach to tmux session running in the background
Ctrl+B d Detach from tmux session, leaving it running in the background
Ctrl+B & Exit and quit tmux
Ctrl+B ? List all key bindings (press Q to exit help screen)

Window management

Ctrl+B C Create new window If you are running more than one tmux session (more than one
PID), you can switch between the two clients.
Ctrl+B N Move to next window
Ctrl+B P Move to previous window
Ctrl+B L Move to last window
Ctrl+B 0-9 Move to window by index number
Ctrl+B ) Move to next session
Ctrl+B ( Move to previous session
Ctrl+B Ctrl+Z Suspend session	

Split window into panes

Ctrl+B % Vertical split (panes side by side)
Ctrl+B " Horizontal split (one pane below the other)
Ctrl+B CTRL+O Interchange pane position
Ctrl+B O Move to other pane
Ctrl+B ! Remove all panes but the current one from the window
Ctrl+B Q Display window index numbers
Ctrl+B Ctrl-Up/Down Resize current pane (due north/south)
Ctrl+B Ctrl-Left/Right Resize current pane (due west/east)

Pane related

join-pane -s 1 -t 0 -p 20  "Join pane source 1 into pane target 0 with 20% usage"
break pane  "remove all other panes like CTRL+B !"

# best to create some key bindings into tmux.conf
# pane movement vertical split
bind-key j command-prompt -p "join pane from:"  "join-pane -h -s '%%'"
bind-key s command-prompt -p "send pane to:"  "join-pane -h -t '%%'"

# pane movement
bind-key J command-prompt -p "join pane from:"  "join-pane -s '%%'"
bind-key S command-prompt -p "send pane to:"  "join-pane -t '%%'"

# move panes around
Ctrl+B <space>

Copy/Paste

CTRL+B [ enter copy mode (user arrows or CTRL+F CTRL+B CTRL+B to move)
SHIFT+v to start select
Movement keys to select
ENTER to copy

CTRL+B ] to paste

q to exit from copy mode

Misc

CTRL+B ? List all bindings
For more details - https://github.com/tmux/tmux/blob/master/key-bindings.c#L345

For all commands see into https://github.com/tmux/tmux/blob/master files that begin with cmd-

Online openssl private certificate and key with alternative DNS

Openssl added a nice alternative to the config file or extention to create requests with alternative DNS. This will create a key and certificate (not certificate request) with two additional DNS alt1.example.net and alt2.example.net

sudo openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -keyout mykey.key -out mycer.crt  -subj '/CN=main.example.net' -addext 'subjectAltName=DNS:alt1.example.net,DNS:alt2.example.net'

Victoria metrics on Aws EC2 instance

Will configure one single EC2 instance as a Victoria Metrics server to be used as Promethues storage.

The access to VM(victoria metrics) is done via port 8247 and is protected by http basic auth. All traffic is encrypted with a self sign certificate.

Installation

Will install manually by downloading the releases from github and configure the local system.

Download binaries

# create a group and user for vm
$ sudo groupadd -r victoriametrics
$ sudo useradd -g victoriametrics victoriametrics
 
# download
$ curl -L https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.70.0/victoria-metrics-amd64-v1.70.0.tar.gz --output victoria-metrics-amd64-v1.70.0.tar.gz

# unpack and install it
$ sudo tar xvf victoria-metrics-amd64-v1.70.0.tar.gz -C /usr/local/bin/
$ chown root:root /usr/local/bin/victoria-metrics-prod

# create data directory
$ sudo mkdir /var/lib/victoria-metrics-data
$ chown -v victoriametrics:victoriametrics /var/lib/victoria-metrics-data

Configure the service

cat >> /etc/systemd/system/victoriametrics.service <<EOF
[Unit]
Description=High-performance, cost-effective and scalable time series database, long-term remote storage for Prometheus
After=network.target

[Service]
Type=simple
User=victoriametrics
Group=victoriametrics
StartLimitBurst=5
StartLimitInterval=0
Restart=on-failure
RestartSec=1
ExecStart=/usr/local/bin/victoria-metrics-prod \
        -storageDataPath=/var/lib/victoria-metrics-data \
        -httpListenAddr=127.0.0.1:8428 \
        -retentionPeriod=1
ExecStop=/bin/kill -s SIGTERM $MAINPID
LimitNOFILE=65536
LimitNPROC=32000

[Install]
WantedBy=multi-user.target

EOF

At this point your can start the service systemctl enable victoriametrics.service --now, however the port 8428 is not protected in any way nor is encrypted so will add basic authentication and tls encryption with a self sign certificate, any valid certificate will work however. Note that listens only on localhost.

Vmauth

To protect the service will use vmauth which is part of a tool set released by victoria metrics.

# download and install the vm utils

$ curl -L https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.70.0/vmutils-amd64-v1.70.0.tar.gz --output vmutils-amd64-v1.70.0.tar.gz
$ sudo tar xvf vmutils-amd64-v1.70.0.tar.gz -C /usr/local/bin/
$ chown -v root:root /usr/local/bin/vm*-prod

Configure vmauth

Create a config file (config.yml) to enable basic authentication.

The format of the file is simple, you need a username and a password.

$ sudo mkdir -p /etc/victoriametrics/ssl/
$ sudo chown -vR victoriametrics:victoriametrics /etc/victoriametrics
$ sudo touch /etc/victoriametrics/config.yml
$ sudo chown -v victoriametrics:victoriametrics /etc/victoriametrics/config.yml

# generate a password for our user
$ python3  -c 'import secrets; print(secrets.token_urlsafe())'
KGKK_NoiciEMn6KdBk6CkcLHZt6TpB-Cgt12UFqnutU

# wite the config
$ sudo cat >> /etc/victoriametrics/config.yml <<EOF
> users:
>   - username: "user1"
>     password: "KGKK_NoiciEMn6KdBk6CkcLHZt6TpB-Cgt12UFqnutU"
>     url_prefix: "http://127.0.0.1:8428"
> # end config
> EOF

Install a self sign certificate

$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout /etc/victoriametrics/ssl/victoriametrics.key -out /etc/victoriametrics/ssl/victoriametrics.crt

$ sudo chown -Rv victoriametrics:victoriametrics /etc/victoriametrics/ssl/

Enable vmauth service

cat >> /etc/systemd/system/vmauth.service <<EOF
[Unit]
Description=Simple auth proxy, router and load balancer for VictoriaMetrics
After=network.target

[Service]
Type=simple
User=victoriametrics
Group=victoriametrics
StartLimitBurst=5
StartLimitInterval=0
Restart=on-failure
RestartSec=1
ExecStart=/usr/local/bin/vmauth-prod \
        --tls=true \
        --auth.config=/etc/victoriametrics/config.yml \
        --httpListenAddr=0.0.0.0:8247 \
        --tlsCertFile=/etc/victoriametrics/ssl/victoriametrics.crt \
        --tlsKeyFile=/etc/victoriametrics/ssl/victoriametrics.key \
ExecStop=/bin/kill -s SIGTERM $MAINPID
LimitNOFILE=65536
LimitNPROC=32000

[Install]
WantedBy=multi-user.target


EOF

Start and enable systemctl enable vmauth.service --now .

To test you will need first to construct a base64 string from the username and password you have written into the config.ymlfile.

For example user vmuser and password secret

$ echo -n 'vmuser:secret' | base64
$ dm11c2VyOnNlY3JldA==

# to test vmauth
$ curl -H 'Authorization: Basic dm11c2VyOnNlY3JldA==' --insecure https://localhost:8247/api/v1/query -d 'query={job=~".*"}'

Operations

Snaphots

List what’s available

curl 'https://localhost:8247/snapshot/list'

{"status":"ok","snapshots":["20211227145126-16C1DDB61673BA11"

Create a new snapshot

curl 'https://localhost:8247/snapshot/create'

{"status":"ok","snapshot":"20211227145526-16C1DDB61673BA12"}

List again the snapshots

curl -s 'https://localhost:8247/snapshot/list' | jq .
{
  "status": "ok",
  "snapshots": [
    "20211227145126-16C1DDB61673BA11",
    "20211227145526-16C1DDB61673BA12"
  ]
}

Backups

The snapshots are located on local disk under data path (parameter -storageDataPath=) on my instance it resolves to storageDataPath=/var/lib/victoria-metrics-data/.

The data into snapshots is compressed with Zstandard.

To push the backups to s3 you can use vmbackup.

$ sudo vmbackup-prod -storageDataPath=/var/lib/victoria-metrics-data  -snapshotName=20211227145526-16C1DDB61673BA12 -dst=s3://BUCKET-NAME/`date +%s`

...

2021-12-29T16:07:20.571Z        info    VictoriaMetrics/app/vmbackup/main.go:105        gracefully shutting down http server for metrics at ":8420"
2021-12-29T16:07:20.572Z        info    VictoriaMetrics/app/vmbackup/main.go:109        successfully shut down http server for metrics in 0.001 seconds

For more info you can see vmbackup.

Postgresql locks

Locks in postgres

Find locks

select pid, state, usename, query, query_start 
from pg_stat_activity 
where pid in (
  select pid from pg_locks l 
  join pg_class t on l.relation = t.oid 
  and t.relkind = 'r' 
  where t.relname = 'search_hit'
);

Killing locks

SELECT pg_cancel_backend(PID);

Haproxy socket stats

Enable stats

Reporting is provided if you enable stats into its config.

The setting is described at https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-stats%20enable

In this post I describe how to use the socket type.

Enable the stats socket

I enable it into the global section as so

global

  stats socket /var/lib/haproxy/stats group haproxy mode 664

What this does is:

• enable the stats socket under /var/lib/haproxy/stats
• the group owner is haproxy (running haproxy as user haproxy)
• permissions are rw (user), rw(group), r(others)

Note there is an option admin that will allow to control haproxy but I don’t use it.

Reading stats from socket (netcat)

You need to have installed netcat (nc).

$ echo 'show stat' | nc -U /var/lib/haproxy/stats
# pxname,svname,qcur,qmax,scur,smax,slim,
....
http_frontend,
....

Reading stats from socket (socat)

You need to install socat since is not frequently installed.

To use it

$ echo 'show stat' | socat stdio /var/lib/haproxy/stats
# pxname,svname,qcur,qmax,scur,smax,slim,
....
http_frontend,
....

AWS cli filter for security groups

There are times when I want to see the security groups on an AWS region. Nothing special really you can always use the aws cli :)

But wait ... there is so much output especially if you have many groups and many rules.

So this is a simple way to filter on the following values(you can add more values but is mostly what I use)

• VPC Id
• Group Name
• Group Id

Tools that I use

• aws cli (you need to install it)
• jq (available on many linux distros)
• awk (comes with any linux distro)

This is how you put all together

      
      	$ export GROUP='My SG'
        $ aws ec2 describe-security-groups --filters Name=group-name,Values="$GROUP" --output json| jq '.SecurityGroups[]| .VpcId, .GroupName, .GroupId'|  awk '{printf (NR%3==0) ? $0 "\n" : $0}'| sed -e 's/""/ - /g'
        # this will print
        "vpc-xxxxxx - My SG - sg-yyyy"
        # bonus - you can use a regex for GROUP
        $ export GROUP='My*Prod'
        $ aws ec2 describe-security-groups --filters Name=group-name,Values="$GROUP" --output json| jq '.SecurityGroups[]| .VpcId, .GroupName, .GroupId'|  awk '{printf (NR%3==0) ? $0 "\n" : $0}'| sed -e 's/""/ - /g'
        # this will print
        "vpc-xxxxxx - My Prod - sg-yyyy"
        "vpc-xxxxxx - My deprecated Prod - sg-yyyy"
        "vpc-xxxxxx - My whatever Prod - sg-yyyy"