Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Sunday, April 14, 2019

Making use of Ansible vault from fabric(fabfile)

Ansible provides a convenient solution to encrypt sensitive data such as passwords, secrets, etc. - Ansible Vault. This post shows how to use the ansible vault from Fabric. First you would think why ? First I thought is a crazy idea :) however since I've been using Fabric and Ansible for a long while I said why not - they are both written in python right ?!. So how to use it, you need to have installed Fabric and Ansible obviously. Create a fabfile at the top import a few Ansible modules

from ansible.cli import CLI
from ansible.parsing.vault import VaultLib
from ansible.parsing.dataloader import DataLoader
import yaml
import os

This allows to interface with the VaultLib which in turns will unencrypt the vault. And this is how you use them from a function


def gef_vault_data(vault_pass_file, vault_file):
    secrets = CLI.setup_vault_secrets(
            DataLoader(),
            vault_ids=[],
            vault_password_files=[vault_pass_file])

    v = VaultLib(secrets=secrets)

    data = v.decrypt(open(vault_file, 'rb').read())
    return yaml.load(data)

# in case you keep the password file into your home directory - adjust as required
HOME = os.environ.get("HOME")
VAULT_PASSWORD_FILE = os.path.join(HOME, ".ansible/vault_password_file")

my_vault = get_vault_data(VAULT_PASSWORD_FILE, "/etc/ansible/vault.yml")  

print(my_vault)  # this is the data from the encrypted Ansible vault. 

Monday, May 14, 2018

Python pip install from git with specific revision

There are times when you want to try a specific revision of a package that is under a specific git revision.

The general syntax is

pip install -e git://github.com/{ username }/{ reponame }.git@{ tag name }#egg={ desired egg name }
An this is how to install from tag 3.7.0b0 from github via https

# install
pip install git+https://github.com/mongodb/mongo-python-driver.git@3.7.0b0#egg=pymongo

# use pymongo
import pymongo
pymongo.MongoClient()

# MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True)

Thursday, December 18, 2014

Supervisor (python supervisord) email alerts

The program supervisor written in python is used to supervise long running processes. In case that a long running process will stop (crash) supervisor will detect it and will restart it, you will get entries into the log files however unless you have a log aggregation tool or you login into the server or have some other monitoring tool you will not know that your process has crashed.

However there is hope :) - you can setup an event listener into supervisor which can email you in case that a process has exit. To do so you will need to install a python package superlance This is how the setup is done.

# install superlance
$ sudo pip install superlance  # if you don't have pip install try easy_install 

# configure supervisor to send events to crashmail

$ sudo vim /etc/supervisor/supervisord.conf  # change according to your setup

[eventlistener:crashmail]
command=crashmail -a -m root@localhost
events=PROCESS_STATE_EXITED

$ sudo supervisor stop && sudo supervisor start
# done :)

In the example above if a process will crash (exit) an event will be sent to crashmail which in turn will email to root@localhost - of course you can change the email address, crashmail uses actually sendmail to send email (postfix and qmail come with a sendmail like program so no worries).
Also the email alert will be sent out for any program that crashed but if you want to filter out you can choose just the program you want by specifying -p program_name instead if -a, for more info you can see Crashmail section on the superlance docs.

Sunday, March 2, 2014

Getting started with the new AWS tools

AWS replaced their java based tool with a neat python package for linux (didn't try the windows based ones yet ...). Why are these tools nice ?! - written in python - support from one tool for all services - wizard configuration To get started

# use virtualenv or global
# this example shows virtualenv

$ mkdir AWS
$ virtualenv AWS 
...
$ source AWS/bin/activate

# install the tools from pypi

$ pip install awscli
...
# configure

$ aws configure
AWS Access Key ID [None]: XXXXXX
AWS Secret Access Key [None]: XXXXXX
Default region name [None]: us-west-1
Default output format [None]: json

$ aws ec2 describe-regions
{
    "Regions": [
        {
            "Endpoint": "ec2.eu-west-1.amazonaws.com", 
            "RegionName": "eu-west-1"
        }, 
        {
            "Endpoint": "ec2.sa-east-1.amazonaws.com", 
            "RegionName": "sa-east-1"
        }, 
        {
            "Endpoint": "ec2.us-east-1.amazonaws.com", 
            "RegionName": "us-east-1"
        }, 
        {
            "Endpoint": "ec2.ap-northeast-1.amazonaws.com", 
            "RegionName": "ap-northeast-1"
        }, 
        {
            "Endpoint": "ec2.us-west-2.amazonaws.com", 
            "RegionName": "us-west-2"
        }, 
        {
            "Endpoint": "ec2.us-west-1.amazonaws.com", 
            "RegionName": "us-west-1"
        }, 
        {
            "Endpoint": "ec2.ap-southeast-1.amazonaws.com", 
            "RegionName": "ap-southeast-1"
        }, 
        {
            "Endpoint": "ec2.ap-southeast-2.amazonaws.com", 
            "RegionName": "ap-southeast-2"
        }
    ]
}

# Done!

For more info the project is hosted at github.com The reference table Aws tools references and the home page at aws.amazon.com/cli.

Wednesday, August 28, 2013

Why schematics is awesome

Schematics it's a python library that has primary use to validate json data.

Why is this awsome versus other validation tools like validictory or jsonschema ?

The workflow by design is based on the django/sqlalchemy, so you get back an object that has fields and each field can have it's own type, even complex types like objects that contain their own fields and so on.

One more thing that schematics has is - default values. This comes very very handy if you want your data to be normalized,

This is a simple example on how it works:

>>> from schematics import models
>>> from schematics import types
>>>
>>> class Client(models.Model):
>>>     name = types.StringType(required=True, min_length=1, max_length=255)
>>>     email = types.EmailType(required=True)
>>>     active = types.IntType(default=1)
>>>
>>> c = Client(raw_data={'name': 'John', 'email': 'john@example.com'})
>>> c.validate()
>>> c.serialize()
>>> {'active': 1, 'email': u'john@example.com', 'name': u'John'}

There are many other options to validate data - see Schematics.

Wednesday, February 29, 2012

Tricks with nose and python

Nose is a very useful tool for running unittest in python. These are a few tricks you can use. Bellow is my test file - I called it test.py

# dummy case test 

class Test():

    def test_algo(self):
        assert 0 == 0, '0 is not equal to 0'

    def test_failed(self):
        print 'this will fail'
        assert 1 == 0, '0 is not equal to 1'

    def test_fail_inpdb(self):
        # div by 0
        1/0

Now let's see what is this about

  1. first I use assert to check if the results match
  2. based on assert the first function will pass and second will fail
  3. the last function will trigger and Error not a Failure

# running with pdb so any Error not Failure will drop me into python debugger
$ nosetests  --pdb  test.py
.> /home/silviud/PROGS/PYTHON/wal/tests/test.py(14)test_fail_inpdb()
-> 1/0
(Pdb) l
  9             print 'this will fail'
 10             assert 1 == 0, '0 is not equal to 1'
 11     
 12         def test_fail_inpdb(self):
 13             # div by 0
 14  ->         1/0          #### this is the line that triggers the error
[EOF]
(Pdb) c
EF
======================================================================
ERROR: tests.test.Test.test_fail_inpdb
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/silviud/Environments/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/silviud/PROGS/PYTHON/wal/tests/test.py", line 14, in test_fail_inpdb
    1/0
ZeroDivisionError: integer division or modulo by zero

======================================================================
FAIL: tests.test.Test.test_failed
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/silviud/Environments/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/silviud/PROGS/PYTHON/wal/tests/test.py", line 10, in test_failed
    assert 1 == 0, '0 is not equal to 1'
AssertionError: 0 is not equal to 1
-------------------- >> begin captured stdout << ---------------------
this will fail

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 3 tests in 8.453s

FAILED (errors=1, failures=1)

Now let's pretended that I run this regular and is part of my Continuous Integration server which so happen
to be running Jenkins. How can I integrate the python unittests with it ?!
Simple - nose has many plugins and one of them is xunit.

$ nosetests  --with-xunit test.py
....
$ cat nosetests.xml
<?xml version="1.0" encoding="UTF-8"?><testsuite name="nosetests" tests="3" errors="1" failures="1" skip="0"><testcase classname="tests.test.Test" name="test_algo" time="0.000" /><testcase classname="tests.test.Test" name="test_fail_inpdb" time="0.000"><error type="exceptions.ZeroDivisionError" message="integer division or modulo by zero"><![CDATA[Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 321, in run
    testMethod()
  File "/home/silviud/Environments/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/silviud/PROGS/PYTHON/wal/tests/test.py", line 14, in test_fail_inpdb
    1/0
ZeroDivisionError: integer division or modulo by zero
]]></error></testcase><testcase classname="tests.test.Test" name="test_failed" time="0.001"><failure type="exceptions.AssertionError" message="0 is not equal to 1&#10;-------------------- &gt;&gt; begin captured stdout &lt;&lt; ---------------------&#10;this will fail&#10;&#10;--------------------- &gt;&gt; end captured stdout &lt;&lt; ----------------------"><![CDATA[Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 321, in run
    testMethod()
  File "/home/silviud/Environments/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/silviud/PROGS/PYTHON/wal/tests/test.py", line 10, in test_failed
    assert 1 == 0, '0 is not equal to 1'
AssertionError: 0 is not equal to 1
-------------------- >> begin captured stdout << ---------------------
this will fail

--------------------- >> end captured stdout << ----------------------
]]></failure></testcase></testsuite>

Just by adding --with-xunit made nose to active the xunit plugin and it generated an xml file into nosetests.xml - this file can be used by Jenkins to take decisions if the build failed or not !

Monday, January 23, 2012

Install cProfile on debian 6.0.3 (squeeze)

So you just seen this when tried to profile something on debian squeeze

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/cProfile.py", line 36, in run
    result = prof.print_stats(sort)
  File "/usr/lib/python2.6/cProfile.py", line 80, in print_stats
    import pstats
ImportError: No module named pstats

Well all is need to fix it is to enable a repository and then install python-profile

echo 'deb http://ftp.ca.debian.org/debian squeeze main non-free' >> /etc/apt/sources.list
# replace .ca. with your country code
apt-get update
aptitude install  python-profiler
python

>>> import cProfile
>>> def f():
...     print 'called'
...
>>> cProfile.run('f()')
called
         3 function calls in 0.000 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 :1(f)
        1    0.000    0.000    0.000    0.000 :1()
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Tuesday, April 26, 2011

Submit puzzle to facebook's puzzle master

Facebook runs a robot that takes email attachments and runs them to solve a puzzle that is posted
at http://www.facebook.com/careers/puzzles.php#!/careers/puzzles.php .

This is what I did to submit the hoppity puzle

shell$ echo 15 > file.txt 
shell$ python hoppity.py file.txt 
Hoppity
Hohpop
Hoppity
Hoppity
Hohpop
Hoppity
Hop
shell$ cat hoppity.py
#!/usr/bin/env python


import sys
if len(sys.argv) != 2:
    print 'run it as ', __file__, 'file.txt # file.txt should contain one unsigned int'
    sys.exit(1)

_file=sys.argv[1]

try:
    f = open(_file, 'r')
except IOError, ioe:
    print "file %s does not exist " %  _file
except:
    print "can not open file %s" % _file


no = f.read() # assume ONE uint in _file
max = int(no.strip()) + 1

for i in xrange(1,max):
    if i % 3 == 0 and i % 5 == 0 :
            print 'Hop'
    elif i % 3 == 0: print 'Hoppity'
    elif i % 5 == 0: print 'Hophop'

try:
    f.close()
except:
    pass

to actually submit the program - archive it as

mv hoppity.py hoppity && tar cvfz hoppity.tar.gz hoppity.py # the bot doesn't take the extension so you have to cut it off
and send an email with the archive attached to 1051962371@fb.com

Wednesday, February 23, 2011

sqlalchemy UUID as primary key

I keep hearing about having uuid as primary keys into your database so I decided to give a try with sqlalchemy and python(of course).

So the plan is to have a table users that has the following fields

id - type UUID - 32
fname - varchar(50)
lname - varchar(50)

the code bellow builds the table for me.

1 from sqlalchemy import Table, Column, Integer, String, Sequence, MetaData, ForeignKey, CHAR
  2 from sqlalchemy.orm import mapper, sessionmaker, scoped_session
  3 from sqlalchemy import create_engine
  4 import uuid
  5 
  6 metadata = MetaData()
  7 
  8 users = Table('users', metadata,
  9               Column('id', CHAR(32), primary_key=True, autoincrement=True),
 10               Column('fname', String(50)),
 11               Column('lname', String(50))
 12              )
 13 # orm
 14 class Users(object):
 15     def __init__(self, fname, lname):
 16         assert isinstance(fname, str), 'fname is not a string'
 17         assert isinstance(lname, str), 'lname is not a string'
 18         self.fname = fname
 19         self.lname = lname
 20 
 21     
 22     
 23 mapper(Users, users, version_id_col=users.c.id, version_id_generator = lambda version:uuid.uuid4().hex)



line of interest will be
4 - import the uuid module (standard with python 2.6 or higher)
23 - the version_id_col=users.c.id and version_id_generator = lambda version:uuid.uuid4().hex)
the explanation for it is http://www.sqlalchemy.org/docs/orm/mapper_config.html?highlight=uuid#sqlalchemy.orm.mapper
basically you map a temporary integer used my sqlaclhemy and then you transform it into the UUID with a 32 chars in hex.

the rest of the program


24 
 25 engine = create_engine('sqlite:///:memory:', echo=True)
 26 Session = scoped_session(sessionmaker(bind=engine))
 27 metadata.drop_all(engine)
 28 metadata.create_all(engine)
 29 # my test 
 30 session = Session()
 31 
 32 
 33 u = Users('s', 'd');
 34 u1 = Users('s1', 'd2');
 35 
 36 session.add_all([u1, u])
 37 session.commit()
 38 
 39 session.query(Users).all()

if uuid are better then integers as primary keys I don't think so - at least with mysql taking
into consideration the following article http://www.mysqlperformanceblog.com/2007/03/13/to-uuid-or-not-to-uuid/
but they 'hide' your data from outside and seem to do the job.

Wednesday, February 16, 2011

MySQLdb (mysql-python) install on OSX 10.6 Snow Leopard (32 bits)

Ok - you have mysql server installed into /usr/local/mysql and you are thinking - yes I can connect from python to it like on my linux box ... but on 10.6 OSX is a bit different.
First a bit of light of what is happening:

  • the python you run from /usr/bin/python is compiled for 64 and 32 bits ! that is a fat binary as is called. do a file /usr/bin/python and you will see something like
    usr/bin/python: Mach-O universal binary with 3 architectures
    /usr/bin/python (for architecture x86_64): Mach-O 64-bit executable x86_64
    /usr/bin/python (for architecture i386): Mach-O executable i386
    /usr/bin/python (for architecture ppc7400): Mach-O executable ppc
    

  • the mysql server that you installed is 32 bits only !

  • the code for MySQLdb can be compiled for either architecture but not two at ones as into the fat binary above


Steps to instal

  • have the mysql server installed - source, archive or dmg - the best location to install is /usr/local/mysql
  • if you use virtual environment it is best to extract the 32 bits version from the fat python into your environment. same goes for 64 bits if you use it.
    to extract do something like this after you have your virtual environment -
    cp /my_virtual/env/bin/python /my_virtual/env/bin/python.fat
    lipo -remove x86_64 /my_virtual/env/bin/python.fat -output /my_virtual/env/bin/python
    
    -- to check if you are using 32 bits
    python
    >>> import sys
    >>> sys.maxint
    2147483647
    
  • install mysql-python wit pip/easy_install or from source

errors you may see and how to solve them
  • >>> import MySQLdb
    Traceback (most recent call last):
      File "", line 1, in 
      File "/Users/silviud/PROGS/PYTHON/Environments/2.6/lib/python2.6/site-packages/MySQLdb/__init__.py", line 19, in 
        import _mysql
    ImportError: dlopen(/Users/silviud/PROGS/PYTHON/Environments/2.6/lib/python2.6/site-packages/_mysql.so, 2): Library not loaded: libmysqlclient.16.dylib
      Referenced from: /Users/silviud/PROGS/PYTHON/Environments/2.6/lib/python2.6/site-packages/_mysql.so
      Reason: image not found
    
    This is because the dynamic loader can not find the library libmysqlclient.16.dylib which is located into /usr/local/mysql/lib - to solve it add this to your .profile file

    export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/usr/local/mysql/lib
    


This is what I have done to make it work !
I've seen other solutions where you would have to choose the python architecture with an environment variable as

export VERSIONER_PYTHON_PREFER_32_BIT=yes
or
to have it system wide available with
defaults write com.apple.versioner.python Prefer-32-Bit -bool yes
but NONE worked for me except what I shown above.
I even tried to load static the mysql library into the mysql-python by changing the site.cfg from the dist but no luck.

In any case I don't suggest you do this for the a system wide installation - use virtual environment!

Tuesday, September 7, 2010

Python re multiline match (inline)

Compilation Flags -- inline

Python regex flags effect matching. For example: re.MULTILINE, re.IGNORECASE, re.DOTALL. Unfortunately, passing these flags is awkward if want to put regex patterns in a config file or database or otherwise want to have the user be able to enter in regex patterns. You don't want to have to make the user pass in flags separately. Luckily, you can pass flags in the pattern itself. This is a very poorly documented feature of Python regular expressions. At the start of the pattern add flags like this:
(?i) for re.IGNORECASE
 (?L) for re.LOCALE (Make \w, \W, \b, \B, \s and \S dependent on the current locale.)
 (?m) for re.MULTILINE  (Makes ^ and $ match before and after newlines)
 (?s) for re.DOTALL  (Makes . match newlines)
 (?u) for re.UNICODE (Make \w, \W, \b, \B, \d, \D, \s and \S dependent on the Unicode character properties database.)
 (?x) for re.VERBOSE
For example, the following pattern ignores case (case insensitive):
re.search ("(?i)password", string)
The flags can be combined. The following ignores case and matches DOTALL:
re.search ("(?is)username.*?password", string)

Saturday, July 24, 2010

Django on Google App Engine

In case that you want to run the head (latest version) of django on the google app engine this will help you. The info bellow is for a new project (in case you need information on how to make it work with an
existing project see the README file from the appengine_helper).

First you need to understand what is different between running django on your server where you have a RDBMS like mysql and the datastore that google provides. To do this look into the following link
http://code.google.com/appengine/articles/appengine_helper_for_django.html

After you are done reading do the following:
(i'm using /tmp as an example, you can change it to whatever you want)

cd /tmp/

- download the appengine_helper_for_django (if the link changed because there is a new version replace bellow the names for helper with the new names)

curl -L -o 'appengine_helper_for_django-r105.zip' 'http://google-app-engine-django.googlecode.com/files/appengine_helper_for_django-r105.zip'


- download  the latest version of django (you can download it or use the svn to checkout)

curl -L -o 'Django-1.2.1.tar.gz' 'http://www.djangoproject.com/download/1.2.1/tarball/'


- untar the django

tar xvfz Django-1.2.1.tar.gz


- make a new temporary directory for your application

mkdir myapp_tmp


- extract the django helper 

cd myapp_tmp
unzip ../appengine_helper_for_django-r105.zip


- rename the helper

mv appengine_helper_for_django/ myapp


- copy the django framework to have it available

cp  -r ../Django-1.2.1/django  myapp


- configure the app and run the webserver

cd myapp
ls    (you should have something like this)


HANGES VERSION appengine_django manage.py urls.pyc
COPYING __init__.py django settings.py
KNOWN_ISSUES __init__.pyc index.yaml settings.pyc
README app.yaml main.py urls.py


- edit the app.yaml to reflect your application name etc.


- run the webserver to test

python manage.py runserver
(you should see the output from the app engine starting)

"""

/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore_file_stub.py:40: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5
/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/memcache/__init__.py:31: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha
WARNING:root:Could not read datastore data from /var/folders/TI/TIS6v9poFPyMZ3GaFkfkjE+++TQ/-Tmp-/django_vpsup.datastore
WARNING:root:Could not initialize images API; you are likely missing the Python "PIL" module. ImportError: No module named _imaging
INFO:google.appengine.tools.appengine_rpc:Server: appengine.google.com
INFO:root:Checking for updates to the SDK.
INFO:root:The SDK is up to date.
"""

that's it ! you can start using django as you normally do.

(in case you want to change the location of your myapp directory all you have to do is to move the myapp to the new destination - mv /tmp/myapp_tmp/myapp /home/myuser/applications/myapp)