Puppet run_list (like Chef)

Chef has a very nice concept where whatever you need to run is part of a run_list. run_list. This provides an enforced way to run it as opposite to let control of the execution to the configuration management framework. Puppet lacks such run list - can use operators as -> but there is no global list. This trick can simulate a run_list with the same results. Create a file called test.pp with the content:

class one() {
    notice('class one')
}

class two() {
    notice('class two')
}

class three() {
    notice('class three')
}

define include_list  {
    $cls = $name
    notice("including $name")
    include $name
}

if $fqdn == 'debian.localdomain'{
    $clss = ['one', 'two', 'three']
    include_list { $clss:; }
}

And then run it as

 puppet apply test.pp
 notice: Scope(Include_list[one]): including one
notice: Scope(Class[One]): class one
notice: Scope(Include_list[two]): including two
notice: Scope(Class[Two]): class two
notice: Scope(Include_list[three]): including three
notice: Scope(Class[Three]): class three
notice: Finished catalog run in 0.04 seconds

Ec2 (aws) - delete snapshots

Ec2 snapshots are a way to make backups of your data into the amazon cloud. To do snapshots you will need the ec2-api-tools, your access key and secret or the x509 certificates for your aws account. Obviously after you snapshot you will need eventually to delete snapshots that you don't need anymore. This example shows how to use the ec2-api-tools into a shell to delete snapshots that are not part of the current month. You can have a cronjob that runs every last day of the month, this will give you almost 30 days of snapshots.

# describe snapshots and sort by date
ec2-describe-snapshots -C cert.pem  -K key.pem | sort -k 5

# delete all but current month (not the last 30 days)
ec2-describe-snapshots -C cert.pem  -K key.pem | grep -v $(date +%Y-%M-) |  awk '{print $2}' | xargs -n 1 -t ec2-delete-snapshot -K key.pem -C cert.pem

Couchbase recover web console password

You will need to have access to the config.dat file that resides onto the couchbase server (can be any of them if is into a cluster).

/opt/couchbase/bin/erl \
-noinput -eval \
'case file:read_file("/opt/couchbase/var/lib/couchbase/config/config.dat") of {ok, B}  -> io:format("~p~n", [binary_to_term(B)]) end.' \
-run init stop | grep cred
  {rest_creds,
         {creds,[{"Administrator",[{password,"Administrator"}]}]}]},

There you go
username : Administrator
password : Administrator

Puppet install rpms via http sources

The redhat package manager - rpm has the capability to install packages
from an url. As simple as

rpm -ivh http://example.com/package.rpm

Taking this in consideration we can use this into puppet to install packages
from an url.

Save this text as test.pp

class examplerpm ( $src ) {

  package { 'package':
     provider => 'rpm',
     ensure => installed,
     source => "${examplerpm::rpm}"
 }
}

class { 'examplerpm':
  src => 'https://example.com/package.rpm',
}


Apply the manifest with puppet

puppet apply --debug --no-daemonize test.pp

Voila - the package is installed via puppet->rpm provider.
The key to all this is to specify the provider into the Package section of
examplerpm class. This ensures that rpm will go fetch the source and installs
it.

How to calculate the MySQL database size

Connect to mysql and run the command bellow

# total db size
SELECT table_schema "Data Base Name", SUM( data_length + index_length) / 1024 / 1024 
"Data Base Size in MB" FROM information_schema.TABLES GROUP BY table_schema ;

# total per db size
SELECT TABLE_NAME, table_rows, data_length, index_length, 
round(((data_length + index_length) / 1024 / 1024),2) "Size in MB"
FROM information_schema.TABLES WHERE table_schema = "schema_name";

Tricks with nose and python

Nose is a very useful tool for running unittest in python. These are a few tricks you can use. Bellow is my test file - I called it test.py

# dummy case test 

class Test():

    def test_algo(self):
        assert 0 == 0, '0 is not equal to 0'

    def test_failed(self):
        print 'this will fail'
        assert 1 == 0, '0 is not equal to 1'

    def test_fail_inpdb(self):
        # div by 0
        1/0

Now let's see what is this about

1. first I use assert to check if the results match
2. based on assert the first function will pass and second will fail
3. the last function will trigger and Error not a Failure

# running with pdb so any Error not Failure will drop me into python debugger
$ nosetests  --pdb  test.py
.> /home/silviud/PROGS/PYTHON/wal/tests/test.py(14)test_fail_inpdb()
-> 1/0
(Pdb) l
  9             print 'this will fail'
 10             assert 1 == 0, '0 is not equal to 1'
 11     
 12         def test_fail_inpdb(self):
 13             # div by 0
 14  ->         1/0          #### this is the line that triggers the error
[EOF]
(Pdb) c
EF
======================================================================
ERROR: tests.test.Test.test_fail_inpdb
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/silviud/Environments/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/silviud/PROGS/PYTHON/wal/tests/test.py", line 14, in test_fail_inpdb
    1/0
ZeroDivisionError: integer division or modulo by zero

======================================================================
FAIL: tests.test.Test.test_failed
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/silviud/Environments/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/silviud/PROGS/PYTHON/wal/tests/test.py", line 10, in test_failed
    assert 1 == 0, '0 is not equal to 1'
AssertionError: 0 is not equal to 1
-------------------- >> begin captured stdout << ---------------------
this will fail

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 3 tests in 8.453s

FAILED (errors=1, failures=1)

Now let's pretended that I run this regular and is part of my Continuous Integration server which so happen
to be running Jenkins. How can I integrate the python unittests with it ?!
Simple - nose has many plugins and one of them is xunit.

$ nosetests  --with-xunit test.py
....
$ cat nosetests.xml
<?xml version="1.0" encoding="UTF-8"?><testsuite name="nosetests" tests="3" errors="1" failures="1" skip="0"><testcase classname="tests.test.Test" name="test_algo" time="0.000" /><testcase classname="tests.test.Test" name="test_fail_inpdb" time="0.000"><error type="exceptions.ZeroDivisionError" message="integer division or modulo by zero"><![CDATA[Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 321, in run
    testMethod()
  File "/home/silviud/Environments/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/silviud/PROGS/PYTHON/wal/tests/test.py", line 14, in test_fail_inpdb
    1/0
ZeroDivisionError: integer division or modulo by zero
]]></error></testcase><testcase classname="tests.test.Test" name="test_failed" time="0.001"><failure type="exceptions.AssertionError" message="0 is not equal to 1&#10;-------------------- &gt;&gt; begin captured stdout &lt;&lt; ---------------------&#10;this will fail&#10;&#10;--------------------- &gt;&gt; end captured stdout &lt;&lt; ----------------------"><![CDATA[Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 321, in run
    testMethod()
  File "/home/silviud/Environments/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/silviud/PROGS/PYTHON/wal/tests/test.py", line 10, in test_failed
    assert 1 == 0, '0 is not equal to 1'
AssertionError: 0 is not equal to 1
-------------------- >> begin captured stdout << ---------------------
this will fail

--------------------- >> end captured stdout << ----------------------
]]></failure></testcase></testsuite>

Just by adding --with-xunit made nose to active the xunit plugin and it generated an xml file into nosetests.xml - this file can be used by Jenkins to take decisions if the build failed or not !

Shell parallel processing

This is the description of the tool from the gnu site - Parallel
GNU parallel is a shell tool for executing jobs in parallel using one
or more computers. A job it can be a single command or a small script
that has to be run for each of the lines in the input. The typical
input is a list of files, a list of hosts, a list of users, a list of
URLs, or a list of tables. A job can also be a command that reads from
a pipe. GNU parallel can then split the input into blocks and pipe a
lock into each command in parallel.

The tool can do many things and has some very useful tools that come with it
see sql and niceload.
Bellow you can see some example on how to use it.

#!/bin/sh

# tail log files on different computers
# create a hosts file with all the computers you want to connect

echo '10.100.218.79' >> host.file
echo '107.22.24.219' >> host.file
cat host.file | parallel ssh {} "tail /var/log/php-fpm/error.log | awk '{print \$1,\$2,\$3,\$4,\$5,\$6}'"

[20-Feb-2012 15:05:05.323178] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:06.324028] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:07.324877] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:08.325727] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:09.326568] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:10.327418] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:11.328265] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:12.329118] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:13.329960] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:14.330806] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),

GNU's site has lots more example see Examples

Install cProfile on debian 6.0.3 (squeeze)

So you just seen this when tried to profile something on debian squeeze

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/cProfile.py", line 36, in run
    result = prof.print_stats(sort)
  File "/usr/lib/python2.6/cProfile.py", line 80, in print_stats
    import pstats
ImportError: No module named pstats

Well all is need to fix it is to enable a repository and then install python-profile

echo 'deb http://ftp.ca.debian.org/debian squeeze main non-free' >> /etc/apt/sources.list
# replace .ca. with your country code
apt-get update
aptitude install  python-profiler
python

>>> import cProfile
>>> def f():
...     print 'called'
...
>>> cProfile.run('f()')
called
         3 function calls in 0.000 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 :1(f)
        1    0.000    0.000    0.000    0.000 :1()
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

MySQL backup in time

Run mysqldump on master database (needs to have innodb)

# replication point in time
mysqldump --single-transaction --flush-logs \
--master-data=2 --all-databases > backup.sql

# Note the position and the log file from the backup.sql and insert it into the slave</>

shell# mysql -u USER -p PASS
mysql> stop slave;
shell# mysql < backup.sql
shell# mysql -u USER -p PASS 
mysql> CHANGE MASTER TO MASTER_LOG_FILE='the_log_file_written_into_dump',
mysql> MASTER_LOG_POS = xxx ;
mysql> start slave;

Create storage infrastructure with libvirt

Libvirt is defacto library to manage virtual machines. This will show how you can create a storage pool that can be used later to allocate space for vms.

my storage space is located under /storage and the type is a regular directory as i want to store image files in there. this is low performance but will do for my dev machines.

first you need to create the directory

mkdir /storage

# then you need to have an xml file as follow - see type is dir and path is /storage

<pool type="dir">
        <name>storage_01</name>
        <target>
          <path>/storage</path>
        </target>

</pool>

# than go into virsh and define the new pool - define will make the pool  persistent into the /etc/libvirt/

virsh# pool-list
virsh# pool-autostart pool_name
<name>sparse.img</name>
<allocation>0</allocation>
<capacity unit="T">1</capacity>
<target>
  <path>/var/lib/virt/images/sparse.img</path>
  <permissions>
    <owner>107</owner>
    <group>107</group>
    <mode>0744</mode>
    <label>virt_image_t</label>
  </permissions>
</target>
</volume>