Friday, December 20, 2019

Tcpdump on docker interfaces

This post shows how you can inspect docker containers traffic with tcpdump on linux.

First find the docker names and the mac addresses.


bash $ for c in `sudo docker ps| grep -v CON| awk '{print $1}'`; do sudo docker inspect $c| jq ". |map({ (.Name): .NetworkSettings.Networks[].MacAddress })"; done

[
  {
    "/docker-demo_cortex2_1": "02:42:ac:12:00:08"
  }
]
[
  {
    "/docker-demo_consul_1": "02:42:ac:12:00:05"
  }
]
[
  {
    "/docker-demo_prometheus2_1": "02:42:ac:12:00:03"
  }
]
[
  {
    "/docker-demo_cortex3_1": "02:42:ac:12:00:09"
  }
]
[
  {
    "/docker-demo_cortex1_1": "02:42:ac:12:00:06"
  }
]
[
  {
    "/docker-demo_prometheus3_1": "02:42:ac:12:00:04"
  }
]
[
  {
    "/docker-demo_prometheus1_1": "02:42:ac:12:00:02"
  }
]
[
  {
    "/docker-demo_grafana_1": "02:42:ac:12:00:07"
  }
]

I want to inspect on /docker-demo_cortex1_1 so I list the forward table (fdb)

bash $ /sbin/bridge fdb |grep 02:42:ac:12:00:06

02:42:ac:12:00:06 dev vethee0ca4e master br-f9c7e5b79104
This says that the dev `vethee0ca4e` forwards to the master bridge `br-f9c7e5b79104`

List what interfaces are into the system

bash$ sbin/ip link show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:6f:ce:6d brd ff:ff:ff:ff:ff:ff
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:cf:95:1a:17 brd ff:ff:ff:ff:ff:ff
4: br-f9c7e5b79104: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ae:f7:a0:c6 brd ff:ff:ff:ff:ff:ff
28: veth47b30a5@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether b2:23:05:8a:cd:4e brd ff:ff:ff:ff:ff:ff link-netnsid 0
30: veth95ec404@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether ba:41:85:94:67:39 brd ff:ff:ff:ff:ff:ff link-netnsid 1
32: veth246e156@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether 92:26:8e:09:97:af brd ff:ff:ff:ff:ff:ff link-netnsid 2
34: veth426ba55@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether 6a:c0:12:86:30:0f brd ff:ff:ff:ff:ff:ff link-netnsid 5
38: veth91e2bee@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether de:53:75:37:b0:88 brd ff:ff:ff:ff:ff:ff link-netnsid 6
40: veth9199c33@if39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether e2:d1:fa:61:83:cd brd ff:ff:ff:ff:ff:ff link-netnsid 3
42: vethdb6a7ca@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether ea:51:60:cc:6f:e8 brd ff:ff:ff:ff:ff:ff link-netnsid 4
44: vethee0ca4e@if43: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether ca:b1:72:d1:c7:e2 brd ff:ff:ff:ff:ff:ff link-netnsid 7

As you can see the interface that I want to inspect is listed as 44.

At this point just start a tcpdump on the interface

bash$ sudo tcpdump -nvv -s0 -A -i vethee0ca4e
In case you have multiple bridges configured onto the system it will help to fist find the master bridge you want to find.
bash$ sudo docker network ls

NETWORK ID          NAME                         DRIVER              SCOPE
bedcfa44fe2b        bridge                       bridge              local
f9c7e5b79104        docker-demo_cortex_network   bridge              local
0d3a96789a7f        host                         host                local
1ecffcd51252        none                         null                local

Sunday, April 14, 2019

Making use of Ansible vault from fabric(fabfile)

Ansible provides a convenient solution to encrypt sensitive data such as passwords, secrets, etc. - Ansible Vault. This post shows how to use the ansible vault from Fabric. First you would think why ? First I thought is a crazy idea :) however since I've been using Fabric and Ansible for a long while I said why not - they are both written in python right ?!. So how to use it, you need to have installed Fabric and Ansible obviously. Create a fabfile at the top import a few Ansible modules

from ansible.cli import CLI
from ansible.parsing.vault import VaultLib
from ansible.parsing.dataloader import DataLoader
import yaml
import os

This allows to interface with the VaultLib which in turns will unencrypt the vault. And this is how you use them from a function


def gef_vault_data(vault_pass_file, vault_file):
    secrets = CLI.setup_vault_secrets(
            DataLoader(),
            vault_ids=[],
            vault_password_files=[vault_pass_file])

    v = VaultLib(secrets=secrets)

    data = v.decrypt(open(vault_file, 'rb').read())
    return yaml.load(data)

# in case you keep the password file into your home directory - adjust as required
HOME = os.environ.get("HOME")
VAULT_PASSWORD_FILE = os.path.join(HOME, ".ansible/vault_password_file")

my_vault = get_vault_data(VAULT_PASSWORD_FILE, "/etc/ansible/vault.yml")  

print(my_vault)  # this is the data from the encrypted Ansible vault. 

Monday, May 14, 2018

Python pip install from git with specific revision

There are times when you want to try a specific revision of a package that is under a specific git revision.

The general syntax is

pip install -e git://github.com/{ username }/{ reponame }.git@{ tag name }#egg={ desired egg name }
An this is how to install from tag 3.7.0b0 from github via https

# install
pip install git+https://github.com/mongodb/mongo-python-driver.git@3.7.0b0#egg=pymongo

# use pymongo
import pymongo
pymongo.MongoClient()

# MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True)

Tuesday, November 28, 2017

CentOS 7 Postfix relay (gmail)

How to send emails trough a smart relay that uses SASL and TLS

I used:

  • CentOS Linux release 7.3.1611
  • postfix-2.10.1-6.el7.x86_64
The rpm comes from CentOS yum Base.

The setup

File: /etc/postfix/main.cf
This is the main configuration for postfix in regards to how you would like to behave.

smtpd_banner = $myhostname ESMTP $mail_name
biff = no
append_dot_mydomain = no
readme_directory = no
smtpd_tls_session_cache_timeout=3600s
tls_random_source=dev:/dev/urandom
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/sasl/password
smtp_use_tls = yes
smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.trust.crt
smtp_tls_loglevel = 1
smtp_tls_security_level = encrypt
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = ${OPTIONAL_HOSTNAME}
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = $myhostname localhost.$mydomain
relayhost = [${mail.RELAY}]:587
mynetworks = 127.0.0.0/8
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = localhost
inet_protocols = ipv4

# comment these two when done
debug_peer_list = ${mail.RELAY}
debug_peer_level = 3

File: /etc/postfix/sasl/password
Write into the file the username and password that you use to authenticate.
[${mail.RELAY}]    ${user@domain}:${PASSWORD}  
Once you save the file you need to create the database, in this case it's hash
cd /etc/postfix/salsl && postmap password
At this point restart postfix
systemctl restart postfix

The problem

Since all that is configured is ok ... you would expect that now you can send email however ...
smtp_sasl_authenticate: mail.RELAY[IPV4]:587: SASL mechanisms PLAIN LOGIN
warning: SASL authentication failure: No worthy mechs found
...
send attr reason = SASL authentication failed; cannot authenticate to server mail.RELAY[IPV4]: no mechanism available 
The main problem is that the username and password works fine ... you can test by using telnet
# First compute the base64 encoded string. \0 is a null terminated string
printf '${user@domain}\0${user@domain}\0${PASSWORD}' | base64

# telnet to the smtp relay

telnet ${mail.RELAY}
EHLO ${OPTIONAL_HOSTNAME}
250-server.example.com
250-PIPELINING
250-SIZE 10240000
250-ETRN
250-AUTH DIGEST-MD5 PLAIN CRAM-MD5
250 8BITMIME
AUTH PLAIN ${COMPUTED_STRING_FROM_PRINTF}
235 Authentication successful
So what is not working ?! Based on the errors we've seen postfix complains that there is no worthy mechs ... that may lead you to read more into the source code. Bottom line since Postfix uses Cyrus SASL library as per Postfix documentation you actually need to install cyrus-sasl-lib
yum install -y  cyrus-sasl cyrus-sasl-lib cyrus-sasl-plain

# restart postfix

systemctl restart postfix 
At this point if you keep the debug on you will see
....
smtp_sasl_authenticate: ${mail.RELAY}[${IPV4}]:587: SASL mechanisms PLAIN LOGIN
xsasl_cyrus_client_get_user: ${user@domain}
xsasl_cyrus_client_get_passwd: ${PASSWORD}
...
... 235 2.7.0 Authentication successful
 
Note: all symbols ${} should be replace with your relevant information. The value of myhostname is optional into /etc/postfix/main.cf if not present postfix uses your hostname.

Wednesday, November 1, 2017

Zabbix server under Selinux (Centos 7)

Zabbix server under Selinux (CentOS 7)

When running zabbix server under Selinux out of the box when you start
systemctl start zabbix-server
you will get an error like this into /var/log/zabbix/zabbix_server.log


 using configuration file: /etc/zabbix/zabbix_server.conf
 cannot set resource limit: [13] Permission denied
 cannot disable core dump, exiting...
 Starting Zabbix Server. Zabbix 3.0.12 (revision 73586).

 

The problem is related to zabbix policy under Selinux.

How to Fix it

First as the message says zabbix server needs to set some resource limits.
To do so will need to have permissions from selinux. Run the following to see the error and transform it into a format that selinux can load later.
cat /var/log/audit/audit.log | grep zabbix_server | grep denied | audit2allow -M zabbix_server.limits

Two files are created a .pp and a .pe. The .pe file should have content similar to

 module zabbi_server.limits 1.0;

require {
        type zabbix_t;
        class process setrlimit;
}

#============= zabbix_t ==============
allow zabbix_t self:process setrlimit;

 

Load this policy with semodule -i zabbix_server.limits.pp

At this point zabbix server can be started systemctl start zabbix-server
If you need to connect to a database such as mysql/postgress you will need to allow zabbix server again ... (note: I used mysql/mariadb)

cat /var/log/audit/audit.log | grep zabbix_server | grep denied | audit2allow -M zabbix_server.ports

This will create again two files, the .pe file should look like

module zabbix_server_ports 1.0;

require {
        type mysqld_port_t;
        type zabbix_t;
        class process setrlimit;
        class tcp_socket name_connect;
}

#============= zabbix_t ==============

#!!!! This avc can be allowed using the boolean 'zabbix_can_network'
allow zabbix_t mysqld_port_t:tcp_socket name_connect;

#!!!! This avc is allowed in the current policy
allow zabbix_t self:process setrlimit;

    
As you can see the setrlimits is already present and you will need to allow the socket access.
To do so semodule -i zabbix_server.ports.pp

At this point you have two policies loaded and you should restart zabbix server systemctl restart zabbix-server
Note: This may apply to any other version of Linux distros/versions that use Selinux though I only tried on CentOS 7.

Friday, February 10, 2017

MongoDB shell - query collections with special characters

From time to time I found in MongoDB collections that have characters that get interpreted by the mongo shell in a different way and you can't use it as is.

Some example: If your collection name is Items:SubItems and you try to query as you would normally do


mongos> db.Items:SubItems.findOne()
2017-02-10T14:11:17.305+0000 E QUERY    SyntaxError: Unexpected token :

The 'fix' is to use a special javascript notation - so this will work
mongos> db['Items:SubItems'].stats()
{
... 
}

This is called 'Square bracket notation' in javascript.
See Property_accessors for more info.

Tuesday, December 6, 2016

Password recovery on Zabbix server UI

In case you need it ...

Obtain access to the database for read/write (for mysql this is what you need)

update zabbix.users set passwd=md5('mynewpassword') where alias='Admin';