Monday, May 14, 2018

Python pip install from git with specific revision

There are times when you want to try a specific revision of a package that is under a specific git revision.

The general syntax is

pip install -e git://{ username }/{ reponame }.git@{ tag name }#egg={ desired egg name }
An this is how to install from tag 3.7.0b0 from github via https

# install
pip install git+

# use pymongo
import pymongo

# MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True)

Tuesday, November 28, 2017

CentOS 7 Postfix relay (gmail)

How to send emails trough a smart relay that uses SASL and TLS

I used:

  • CentOS Linux release 7.3.1611
  • postfix-2.10.1-6.el7.x86_64
The rpm comes from CentOS yum Base.

The setup

File: /etc/postfix/
This is the main configuration for postfix in regards to how you would like to behave.

smtpd_banner = $myhostname ESMTP $mail_name
biff = no
append_dot_mydomain = no
readme_directory = no
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/sasl/password
smtp_use_tls = yes
smtp_tls_CAfile = /etc/ssl/certs/
smtp_tls_loglevel = 1
smtp_tls_security_level = encrypt
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = ${OPTIONAL_HOSTNAME}
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = $myhostname localhost.$mydomain
relayhost = [${mail.RELAY}]:587
mynetworks =
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = localhost
inet_protocols = ipv4

# comment these two when done
debug_peer_list = ${mail.RELAY}
debug_peer_level = 3

File: /etc/postfix/sasl/password
Write into the file the username and password that you use to authenticate.
[${mail.RELAY}]    ${user@domain}:${PASSWORD}  
Once you save the file you need to create the database, in this case it's hash
cd /etc/postfix/salsl && postmap password
At this point restart postfix
systemctl restart postfix

The problem

Since all that is configured is ok ... you would expect that now you can send email however ...
smtp_sasl_authenticate: mail.RELAY[IPV4]:587: SASL mechanisms PLAIN LOGIN
warning: SASL authentication failure: No worthy mechs found
send attr reason = SASL authentication failed; cannot authenticate to server mail.RELAY[IPV4]: no mechanism available 
The main problem is that the username and password works fine ... you can test by using telnet
# First compute the base64 encoded string. \0 is a null terminated string
printf '${user@domain}\0${user@domain}\0${PASSWORD}' | base64

# telnet to the smtp relay

telnet ${mail.RELAY}
250-SIZE 10240000
235 Authentication successful
So what is not working ?! Based on the errors we've seen postfix complains that there is no worthy mechs ... that may lead you to read more into the source code. Bottom line since Postfix uses Cyrus SASL library as per Postfix documentation you actually need to install cyrus-sasl-lib
yum install -y  cyrus-sasl cyrus-sasl-lib cyrus-sasl-plain

# restart postfix

systemctl restart postfix 
At this point if you keep the debug on you will see
smtp_sasl_authenticate: ${mail.RELAY}[${IPV4}]:587: SASL mechanisms PLAIN LOGIN
xsasl_cyrus_client_get_user: ${user@domain}
xsasl_cyrus_client_get_passwd: ${PASSWORD}
... 235 2.7.0 Authentication successful
Note: all symbols ${} should be replace with your relevant information. The value of myhostname is optional into /etc/postfix/ if not present postfix uses your hostname.

Wednesday, November 1, 2017

Zabbix server under Selinux (Centos 7)

Zabbix server under Selinux (CentOS 7)

When running zabbix server under Selinux out of the box when you start
systemctl start zabbix-server
you will get an error like this into /var/log/zabbix/zabbix_server.log

 using configuration file: /etc/zabbix/zabbix_server.conf
 cannot set resource limit: [13] Permission denied
 cannot disable core dump, exiting...
 Starting Zabbix Server. Zabbix 3.0.12 (revision 73586).


The problem is related to zabbix policy under Selinux.

How to Fix it

First as the message says zabbix server needs to set some resource limits.
To do so will need to have permissions from selinux. Run the following to see the error and transform it into a format that selinux can load later.
cat /var/log/audit/audit.log | grep zabbix_server | grep denied | audit2allow -M zabbix_server.limits

Two files are created a .pp and a .pe. The .pe file should have content similar to

 module zabbi_server.limits 1.0;

require {
        type zabbix_t;
        class process setrlimit;

#============= zabbix_t ==============
allow zabbix_t self:process setrlimit;


Load this policy with semodule -i zabbix_server.limits.pp

At this point zabbix server can be started systemctl start zabbix-server
If you need to connect to a database such as mysql/postgress you will need to allow zabbix server again ... (note: I used mysql/mariadb)

cat /var/log/audit/audit.log | grep zabbix_server | grep denied | audit2allow -M zabbix_server.ports

This will create again two files, the .pe file should look like

module zabbix_server_ports 1.0;

require {
        type mysqld_port_t;
        type zabbix_t;
        class process setrlimit;
        class tcp_socket name_connect;

#============= zabbix_t ==============

#!!!! This avc can be allowed using the boolean 'zabbix_can_network'
allow zabbix_t mysqld_port_t:tcp_socket name_connect;

#!!!! This avc is allowed in the current policy
allow zabbix_t self:process setrlimit;

As you can see the setrlimits is already present and you will need to allow the socket access.
To do so semodule -i zabbix_server.ports.pp

At this point you have two policies loaded and you should restart zabbix server systemctl restart zabbix-server
Note: This may apply to any other version of Linux distros/versions that use Selinux though I only tried on CentOS 7.

Friday, February 10, 2017

MongoDB shell - query collections with special characters

From time to time I found in MongoDB collections that have characters that get interpreted by the mongo shell in a different way and you can't use it as is.

Some example: If your collection name is Items:SubItems and you try to query as you would normally do

mongos> db.Items:SubItems.findOne()
2017-02-10T14:11:17.305+0000 E QUERY    SyntaxError: Unexpected token :

The 'fix' is to use a special javascript notation - so this will work
mongos> db['Items:SubItems'].stats()

This is called 'Square bracket notation' in javascript.
See Property_accessors for more info.

Tuesday, December 6, 2016

Password recovery on Zabbix server UI

In case you need it ...

Obtain access to the database for read/write (for mysql this is what you need)

update zabbix.users set passwd=md5('mynewpassword') where alias='Admin';

Wednesday, November 16, 2016

Netcat HTTP server

Netcat is a very versatile program used for network communications - the place to find it is .

Often I need to test different programs with a dummy HTTP server, so using netcat for this is very easy.

Lt's say you want to respond with HTTP code 200 ... this is what you do with netcat into a shell

 nc -k  -lp 9000 -c 'echo "HTTP/1.1 200 OK\nContent-Length:0\nContent-Type: text/html; charset=utf-8"' -vvv -o session.txt

To explain the switches used:
  • -k accept multiple connections, won't stop netcat after first connection(default)
  • -l listen TCP on the all interfaces
  • -p the port number to bind
  • -c 'echo "HTTP/1.1 200 OK\nContent-Length:0\nContent-Type: text/html; charset=utf-8"' is the most interesting one ... this responds back to the client with a minimal http header and sets code 200 OK
  • -vvv verbosity level
  • -o session.txt netcat will write into this file all the input and output
Now you have a dummy http server running on port 9000 that will answer 200 OK ALL the time :)

Monday, March 28, 2016

Backups with Duplicity and Dropbox

Dropbox is a very popular service for file storage, the way the service works will synchronize by default
all your files across your devices. This is important to know since you will be backing up data into
Dropbox and you don't want to download the backups on every device you have connected.

What we want to do is to backup files, encrypt them and send them to Dropbox.
All this is achieved with Duplicity.

This is the setup

  • Linux OS, any distro will work I guess but I tried on Ubuntu 14.04 LTS
  • Dropbox account (going pro or business is recommended since backups will typical grow over 2GB basic account)

To encrypt files you will need GPG, in case you don't have a key on your system
we need to do a bit of work, if you do have a gpg key you can skip the next section.

GPG Setup

In this section will create GPG public key/private keys that will be used to encrypt the data you backup to Dropbox.

# install
$ sudo apt-get install gnupg
# check if you have any keys
$ gpg --list-keys
# if this is empty than you need to create a set of keys
# follow the wizard to create keys
$ gpg --gen-key
gpg (GnuPG) 1.4.16; Copyright (C) 2013 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: keyring `/home/yourname/.gnupg/secring.gpg' created
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) "

Real name: Your Name
Email address:
You selected this USER-ID:
    "Your Name "

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

gpg: checking the trustdb

# At this point the keys are created and saved into your keyring
# list keys
$ gpg --list-keys
pub   2048R/999B4B79 2016-03-26
            ^^^^^^^^ /used by duplicity
uid                  Your Name 
sub   2048R/99917D12 2016-03-26 

# Note 999B4B79 which is your keyid

Duplicity install

$ sudo apt-get install duplicity

After installation if you are on Ubuntu 14.04 LTS you will need to apply this patch
to /usr/lib/python2.7/dist-packages/duplicity/backends/
If you don't know how to apply the patch is simpler to open the file at line 75 and write the following

 72 def command(login_required=True):
 73     """a decorator for handling authentication and exceptions"""
 74     def decorate(f):
 75         def wrapper(self, *args):
 76             from dropbox import rest  ## line to add
 77             if login_required and not self.sess.is_linked():
 78               log.FatalError("dpbx Cannot login: check your credentials",log.ErrorCode.dpbx_nologin)

Dropbox and duplicity setup

You need to have an account first. Open your browser and login.

Backups with duplicity and dropbox

Since this is the first time you run it need to make a authorization token, this is done as follow

$ duplicity --encrypt-key 999B4B79 full SOURCE dpbx:///
Please authorize in the browser. After you're done, press enter.

Now into your browser authorize the application. This will create an access token into dropbox.
You can see the apps you have going to Security
Should see under Apps linked backend for duplicity
In case you need to know what token is in use you can see it onto you system ~/.dropbox.token_store.txt

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase: 
Retype passphrase to confirm: 
--------------[ Backup Statistics ]--------------
StartTime 1459031263.59 (Sat Mar 26 18:27:43 2016)
EndTime 1459031263.73 (Sat Mar 26 18:27:43 2016)
ElapsedTime 0.14 (0.14 seconds)
SourceFiles 2
SourceFileSize 1732720 (1.65 MB)
NewFiles 2
NewFileSize 1732720 (1.65 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 2
RawDeltaSize 1728624 (1.65 MB)
TotalDestinationSizeChange 388658 (380 KB)
Errors 0


When the first full backup finished you can start making incremental backups, list the backups etc.
# list the backup files
duplicity --encrypt-key 999B4B79 list-current-files dpbx:///

## Make an incremental backup

duplicity --encrypt-key 999B4B79 incr SOURCE dpbx:///

duplicity --encrypt-key 999B4B79 list-current-files dpbx:///


During a backup if you see something like

Attempt 1 failed. NameError: global name 'rest' is not defined
Attempt 2 failed. NameError: global name 'rest' is not defined

See the note about Ubuntu 14.04 because you need to patch the file


If you use multiple computers and don't want to download from dropbox all
the backups you need to enable selective sync and exclude the Apps/duplicity
folder from Dropbox.
I haven't used duplicity for long time and heard some mix opinions, some say is excellent and some
say has some design flows (didn't checked) where your full backup will be taken after a while even if
you just do incremental. Remains to be seen.
I guess if this doesn't work well I would look into Borg Backup which seems to be the best these days since
has dedup built in and many other features. One thing that doesn't though is many backends as duplicity which
can use pretty much all cloud storage solutions around :).