Tcpdump on docker interfaces

This post shows how you can inspect docker containers traffic with tcpdump on linux.

First find the docker names and the mac addresses.

bash $ for c in `sudo docker ps| grep -v CON| awk '{print $1}'`; do sudo docker inspect $c| jq ". |map({ (.Name): .NetworkSettings.Networks[].MacAddress })"; done

[
  {
    "/docker-demo_cortex2_1": "02:42:ac:12:00:08"
  }
]
[
  {
    "/docker-demo_consul_1": "02:42:ac:12:00:05"
  }
]
[
  {
    "/docker-demo_prometheus2_1": "02:42:ac:12:00:03"
  }
]
[
  {
    "/docker-demo_cortex3_1": "02:42:ac:12:00:09"
  }
]
[
  {
    "/docker-demo_cortex1_1": "02:42:ac:12:00:06"
  }
]
[
  {
    "/docker-demo_prometheus3_1": "02:42:ac:12:00:04"
  }
]
[
  {
    "/docker-demo_prometheus1_1": "02:42:ac:12:00:02"
  }
]
[
  {
    "/docker-demo_grafana_1": "02:42:ac:12:00:07"
  }
]

I want to inspect on /docker-demo_cortex1_1 so I list the forward table (fdb)

bash $ /sbin/bridge fdb |grep 02:42:ac:12:00:06

02:42:ac:12:00:06 dev vethee0ca4e master br-f9c7e5b79104

This says that the dev `vethee0ca4e` forwards to the master bridge `br-f9c7e5b79104`

List what interfaces are into the system

bash$ sbin/ip link show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:6f:ce:6d brd ff:ff:ff:ff:ff:ff
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:cf:95:1a:17 brd ff:ff:ff:ff:ff:ff
4: br-f9c7e5b79104: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ae:f7:a0:c6 brd ff:ff:ff:ff:ff:ff
28: veth47b30a5@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether b2:23:05:8a:cd:4e brd ff:ff:ff:ff:ff:ff link-netnsid 0
30: veth95ec404@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether ba:41:85:94:67:39 brd ff:ff:ff:ff:ff:ff link-netnsid 1
32: veth246e156@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether 92:26:8e:09:97:af brd ff:ff:ff:ff:ff:ff link-netnsid 2
34: veth426ba55@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether 6a:c0:12:86:30:0f brd ff:ff:ff:ff:ff:ff link-netnsid 5
38: veth91e2bee@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether de:53:75:37:b0:88 brd ff:ff:ff:ff:ff:ff link-netnsid 6
40: veth9199c33@if39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether e2:d1:fa:61:83:cd brd ff:ff:ff:ff:ff:ff link-netnsid 3
42: vethdb6a7ca@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether ea:51:60:cc:6f:e8 brd ff:ff:ff:ff:ff:ff link-netnsid 4
44: vethee0ca4e@if43: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f9c7e5b79104 state UP mode DEFAULT group default 
    link/ether ca:b1:72:d1:c7:e2 brd ff:ff:ff:ff:ff:ff link-netnsid 7

As you can see the interface that I want to inspect is listed as 44.

At this point just start a tcpdump on the interface

bash$ sudo tcpdump -nvv -s0 -A -i vethee0ca4e

In case you have multiple bridges configured onto the system it will help to fist find the master bridge you want to find.

bash$ sudo docker network ls

NETWORK ID          NAME                         DRIVER              SCOPE
bedcfa44fe2b        bridge                       bridge              local
f9c7e5b79104        docker-demo_cortex_network   bridge              local
0d3a96789a7f        host                         host                local
1ecffcd51252        none                         null                local

Tracing a telnet session with strace

Sometime you just need to know if a port is open on a remote system. The simplest way to find out if the port is open is to just telnet into the host and the port number.
This should look like:

shell$ telnet localhost  23
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host: Connection refused

# let's redo it with the strace enabled.
strace -vo strace.telnet telnet localhost  23
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host: Connection refused
# the error is the same but now we do have the strace.telnet file to find more info

shell$ cat strace.telnet
execve("/usr/kerberos/bin/telnet", ["telnet", "localhost", "23"], ["HOSTNAME=localhost.localdomain", "TERM=xterm-color", "SHELL=/bin/bash", "HISTSIZE=1000", "SSH_CLIENT=10.211.55.2 62489 22", "SSH_TTY=/dev/pts/0", "USER=root", "LS_COLORS=no=00:fi=00:di=01;34:l", "MAIL=/var/spool/mail/root", "PATH=/usr/kerberos/sbin:/usr/ker", "INPUTRC=/etc/inputrc", "PWD=/root", "LANG=en_US.UTF-8", "SHLVL=1", "HOME=/root", "LOGNAME=root", "SSH_CONNECTION=10.211.55.2 62489", "LESSOPEN=|/usr/bin/lesspipe.sh %", "G_BROKEN_FILENAMES=1", "_=/usr/bin/strace", "OLDPWD=/usr/src"]) = 0
brk(0)                                  = 0x9c8d000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_dev=makedev(3, 1), st_ino=129997, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=56, st_size=26551, st_atime=2011/04/21-14:46:38, st_mtime=2011/04/21-07:09:59, st_ctime=2011/04/21-07:09:59}) = 0
mmap2(NULL, 26551, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7fe5000
close(3)                                = 0
open("/usr/lib/libkrb4.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0pB(\0004\0\0\0"..., 512) = 512
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fe4000
fstat64(3, {st_dev=makedev(3, 1), st_ino=239248, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=208, st_size=100960, st_atime=2011/04/21-14:46:38, st_mtime=2010/01/12-19:22:52, st_ctime=2011/04/08-04:02:50}) = 0
mmap2(NULL, 117948, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6d3000
mmap2(0x6ea000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17) = 0x6ea000
mmap2(0x6eb000, 19644, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x6eb000
close(3)                                = 0
open("/usr/lib/libdes425.so.3", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\300\354)\0004\0\0\0"..., 512) = 512
fstat64(3, {st_dev=makedev(3, 1), st_ino=236616, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=32, st_size=12816, st_atime=2011/04/21-14:46:38, st_mtime=2010/01/12-19:22:52, st_ctime=2011/04/08-04:02:41}) = 0
mmap2(NULL, 13868, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x1d9000
mmap2(0x1dc000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0x1dc000
close(3)                                = 0
.......
.......
fstat64(1, {st_dev=makedev(0, 12), st_ino=2, st_mode=S_IFCHR|0620, st_nlink=1, st_uid=0, st_gid=5, st_blksize=4096, st_blocks=0, st_rdev=makedev(136, 0), st_atime=2011/04/21-14:46:38, st_mtime=2011/04/21-14:46:38, st_ctime=2011/04/21-04:36:50}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7feb000
write(1, "Trying 127.0.0.1...\r\n", 21) = 21
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_IP, IP_TOS, [16], 4)  = 0
connect(3, {sa_family=AF_INET, sin_port=htons(23), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)
write(2, "telnet: connect to address 127.0"..., 57) = 57
close(3)                                = 0
write(2, "telnet: Unable to connect to rem"..., 61) = 61
exit_group(1) 

# as you can see there is a lot of information and some i replaced with ..... 
# the line of interest will be 

connect(3, {sa_family=AF_INET, sin_port=htons(23), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)

# this tells when the socket is actually trying to connect onto the remote host and had a return code of -1, all after it is just output from the telnet program that formats it very carefully.

Ssh dynamic port(SOCKS) - firefox with different profiles

I have a data center network that is accessible only from a host - everything that is
behind it has to go trough it - this is usually what it happens to access  data centers and different solution being a site to site vpn (or road warrior - basically from your laptop to a firewall or access server).

I do deployments onto the servers that are behind this bastion host and I need to access the web applications ... how to do it ?!

Use openssh with a dynamic tunnel or putty (I use openssh) !

How to do it:

This is the diagram

/ web01 port 9000
me ---- internet ---- bastion_host --- web02 port 9001
                                     \ web03 port 9002
me$ ssh -v -D 8080 bastion_host

This has created a SOCKS proxy on my machine port 8080. If you don't know what is a SOCKS proxy - read on it http://en.wikipedia.org/wiki/SOCKS


Configure firefox to use a SOCKS proxy. Go to Edit -> Preferences -> Advanced -> Network -> Settings

Now you would like to use the browser to read as well your gmail or just surf the web -  so you need two different profiles

- one that uses the SOCKS proxy
- one for general use

To do so start firefox like this:

firefox -ProfileManager -no-remote

-no-remote is to start a separate process for firefox(which is not default)
-ProfileManager is to choose your profile - you can create a new one and from there take off the SOCKS proxy to use it for web surfing.


Now when you are into the profile with SOCKS proxy type in

http://web01:9000 and you have a direct connection !

Fire up a new firefox and choose the no proxy profile and just use it.

Port redirect in linux to remote host

I had an application that needs to connect to a LDAP port (tcp:389) but the problem was that the LDAP was on an external network and on a non standard port (tcp:1389).
Pointing direct to remote_ip:1389 was not and option because the application is taking the LDAP port from the openldap libs (LDAP_PORT) which is a constant integer = 389 ...

The solution proved to be very simple - install a small port redirect program rinted do a small configuration into .

# config
/etc/rinetd.conf
   
127.0.0.1     389        remote_ip        1389


# start the daemon (only if you install it from rpm - if not just start manually)
/etc/init.d/rinetd start

And this is it - all works just fine.

Note that doing an iptables PREROUTING and DNAT will not work in my case because iptables can do
redirects ONLY into the local network.