Install cProfile on debian 6.0.3 (squeeze)

So you just seen this when tried to profile something on debian squeeze

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/cProfile.py", line 36, in run
    result = prof.print_stats(sort)
  File "/usr/lib/python2.6/cProfile.py", line 80, in print_stats
    import pstats
ImportError: No module named pstats

Well all is need to fix it is to enable a repository and then install python-profile

echo 'deb http://ftp.ca.debian.org/debian squeeze main non-free' >> /etc/apt/sources.list
# replace .ca. with your country code
apt-get update
aptitude install  python-profiler
python

>>> import cProfile
>>> def f():
...     print 'called'
...
>>> cProfile.run('f()')
called
         3 function calls in 0.000 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 :1(f)
        1    0.000    0.000    0.000    0.000 :1()
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Quick start with GlusterFS

The software that www.gluster.org makes allows to have distributed file systems with commodity hardware.

Rpm
===

http://download.gluster.com/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-fuse-3.2.0-1.x86_64.rpm
http://download.gluster.com/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-core-3.2.0-1.x86_64.rpm
http://download.gluster.com/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-rdma-3.2.0-1.x86_64.rpm


Gluster Daemon
==============

/etc/init.d/glusterd start


Firewall
========

iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24007:24008 -j ACCEPT
iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24009:24014 -j ACCEPT


Bricks
======

gluster peer probe SERVER_NAME_OR_IP


Volumes
=======

    Creation
    ========
    # replicated (mirror onto 2 hosts)
    gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2

    Start
    =====
    gluster volume start test-volume

    List
    ====
    gluster volume info (all|vol_name)

    Mount
    =====
    mount -t glusterfs HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR

Ssh execute remote commands

Ssh is a very very useful tool and some tricks with it will help you a lot of time and typing. For example I want to transfer a file remote like my ssh-key file. I have a few options:

• use scp to transfer the file, then login into the remote system and execute the commands.
  
• use a different mechanism to transfer the files(ftp etc), login and execute the commands.
  
• do it all in one command ... this is the cool one - see bellow
  

shell$ cat ~sd/.ssh/id_rsa.pub | ssh root@192.168.0.105 'cat - > .ssh/authorized_keys'
shell$ cat ~sd/.ssh/id_rsa.pub | ssh root@192.168.0.105 'cat - > .ssh/authorized_keys2'

# usually the authorized_keys is a link to authorized_keys2 but in this I just have two separate files.
# as you can see is nothing else to do :)

Am I hacked ?

You do a ps -ef and you think is all good ... but perhaps what you see is not exactly what is really running ... This is a simple but effective way to compare the running processes reported by ps with what is into /proc

shell$ ps ax | wc -l
shell$ 30
shell$ ls -d /proc/* | grep [0-9]|wc -l
shell$ 31 # there is one extra root kit perhaps :)

What happens when you do kill a program in linux ?

I had two simple questions:

• 
  q1: how to you stop(kill) a program in linux ?
  
• 
  a1: i use the kill command as in

  kill 99 or kill -9 99

• 
  q2: ok ... so what does it really happens ?
  
• 
  a2: himm ... good question - well i send a signal to the program via a system call and then the kernel will take care of the rest ... as in will kill the program
  q2.1: himm so how does it kill it ?! what does it really happens
  a2.1: you know what let me think about it ... yeah i didn't look into this - well let me trace it and will find out ...
  

  shell$ bash &
  [2] 29120
  shell$ strace kill 29120
  execve("/usr/bin/kill", ["kill", "29120"], [/* 23 vars */]) = 0
  brk(0)                                  = 0x8849000
  access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
  open("/etc/ld.so.cache", O_RDONLY)      = 3
  fstat64(3, {st_mode=S_IFREG|0644, st_size=24036, ...}) = 0
  mmap2(NULL, 24036, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f1e000
  close(3)                                = 0
  open("/lib/libc.so.6", O_RDONLY)        = 3
  read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340\17K\0004\0\0\0"..., 512) = 512
  fstat64(3, {st_mode=S_IFREG|0755, st_size=1611564, ...}) = 0
  mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f1d000
  mmap2(0x49b000, 1332676, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x110000
  mprotect(0x24f000, 4096, PROT_NONE)     = 0
  mmap2(0x250000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13f) = 0x250000
  mmap2(0x253000, 9668, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x253000
  close(3)                                = 0
  mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f1c000
  set_thread_area({entry_number:-1 -> 6, base_addr:0xb7f1c6c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
  mprotect(0x250000, 8192, PROT_READ)     = 0
  mprotect(0x497000, 4096, PROT_READ)     = 0
  munmap(0xb7f1e000, 24036)               = 0
  brk(0)                                  = 0x8849000
  brk(0x886a000)                          = 0x886a000
  kill(29120, SIGTERM)                    = 0
  exit_group(0)                           = ?
  
  

  from the second line at the bottom i can see that a kill(PID, SIGTERM) was sent to the process and the return code is 0 (meaning success), but does it really happens into the kernel ?! - it will take me a lot more to explain but I found a good article about it at http://www.ibm.com/developerworks/library/l-linux-process-management/
  

What pid has my shell ?

Sometimes you are logged into a system on different terminals and you want to figure it out what process id you have on a specific terminal. The commands to do so are very simple:

shell$ tty
/dev/pts/1
# i'm on pts/1 meaning remote

shell$ ps -p $$
 PID TTY          TIME CMD
10044 pts/1    00:00:00 bash
# $$ means current process pid

Recursive grep with filters - command line

Problem:
you have a list o directories that you want to search a pattern BUT you want to exclude a few directories

Solution:
use ls and grep recursive and apply a filter for the directory that you want to exlude

shell$ pwd
/home/
shell$ ls
work documents logs bin
# you want to search on all except logs for pattern 'invoice'

shell$ for i in `ls` ;do  if [ $i != 'logs' ]; then grep -r 'invoice' $i; fi; done
... output

what it happens is this:

the for loop executes based on the command `ls`
all the output is taken and feed it to the do
inside the do we are filtering off 'logs' (with the if) and then search for invoice with grep -r

Delete empty lines into a file

Problem - you have a file that has a huge number of records and in between you have
empty lines.
This is my example.txt file:

asd

asd
as


sd
as

asd

asd

sd
g

To take the spaces off you have at least two simple tools to do it.

• sed with this tool you can achieve your goal just by doing this - open a command line prompt and type the following
  linux$ cat example.txt | sed -e '/^$/d' you will see on your screen the following
  

  asd
  asd
  as
  sd
  as
  asd
  asd
  sd
  g
  

• grep with this we use a trick saying that we don't want certain patterns from the file - the empty lines - open a command line prompt and type the following
  linux$ grep -v '^$' example.txt
  

asd
asd
as
sd
as
asd
asd
sd
g

Html to XHTML/XML with tidy

As you all know the browser will perform in quirks mode
www.quirksmode.org if you don't specify into your html file what type of document is it.
This will make your life super difficult because browser will display your pages as they wish.
The best thing to do then is to declare what type of document you present.

If there is only one page to change of course it can be done manually but if you have
many pages to do is always better to use a programmatic approach.
So tidy is to rescue !
Tidy is a small program that runs on Linux and can help on this. More info can be found at tidy.sourceforge.net

So let's get practical:
problem - you have a large number of files you want to convert from simple html to xhtml or xml.
solution - open a terminal (xterm. gterm or anything you want - works on direct console as well)
and type the following

for i in `find ./*.html -type f`; do tidy -asxhtml  < $i > $i.xhtml ; done 

Q: What just happened ?
A1: The command looked for all the *.html (all files ending with html extension) into the current directory and then feed tidy with the files matched.
Tidy took the file as input converted to xhtml (option -asxhtml) and wrote it back to the disk with the original_name.html.xhtml
A2: Into each file that was wrote by tidy you can see the new tags that are necessary for xhtml.

For example into the original file you had
<html>
        <head>
                <title> me - home page  </title>
        </head>
        <body>
                <h1> my home page </h1>
                <p> Lorem ipsum dolor sit amet, consectetur adipiscing elit.
                        Morbi vitae lorem justo. Cum sociis natoque penatibus et
                        magnis dis parturient montes, nascetur ridiculus mus.
                        Sed sed consectetur massa. Morbi ac erat sit amet eros
                        malesuada pellentesque tempus ultricies dui. Nam laoreet nibh
                        sit amet massa pellentesque fermentum. Aenean suscipit laoreet
                        lorem sed dignissim. Integer dapibus pulvinar lorem ac placerat.
                        Mauris suscipit, risus commodo tempor feugiat, mi metus facilisis
                        quam, auctor sagittis magna mi vitae lacus. Suspendisse purus velit,
                        ultrices at iaculis dictum, rutrum nec tortor.
                        Nullam a libero ut erat semper mollis in et lectus.
                        Sed suscipit lorem eu mi tristique volutpat. Proin lobortis
                        vehicula nunc vel condimentum. Aliquam mattis, lacus eu adipiscing
                        posuere, dui enim sodales felis, et tincidunt nulla urna lacinia velit.
                </p>
        </body>
</html>


Now you have

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta name="generator" content=
"HTML Tidy for Linux/x86 (vers 25 March 2009), see www.w3.org" />
<title>me - home page</title>
</head>
<body>
<h1>my home page</h1>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi
vitae lorem justo. Cum sociis natoque penatibus et magnis dis
parturient montes, nascetur ridiculus mus. Sed sed consectetur
massa. Morbi ac erat sit amet eros malesuada pellentesque tempus
ultricies dui. Nam laoreet nibh sit amet massa pellentesque
fermentum. Aenean suscipit laoreet lorem sed dignissim. Integer
dapibus pulvinar lorem ac placerat. Mauris suscipit, risus commodo
tempor feugiat, mi metus facilisis quam, auctor sagittis magna mi
vitae lacus. Suspendisse purus velit, ultrices at iaculis dictum,
rutrum nec tortor. Nullam a libero ut erat semper mollis in et
lectus. Sed suscipit lorem eu mi tristique volutpat. Proin lobortis
vehicula nunc vel condimentum. Aliquam mattis, lacus eu adipiscing
posuere, dui enim sodales felis, et tincidunt nulla urna lacinia
velit.</p>
</body>
</html>


That's it - now you have xhtml !
You can change the above command to make tidy do other useful things as preserving the indentation level, clean the file by some tags etc.

Disable at runtime selinux

There are times when you need to test something quick and selinux is in your way ... what do you do then ?
Instead of going with a full reboot you can just do the following

echo 0 > /selinux/enforce

This will disable the selinux at runtime. If the system is configured with selinux enabled into his config file
(on redhat/centos /etc/sysconfig/selinux ) next time you reboot it will be enable.

To enable at runtime

echo 1 > /selinux/enforce

Lvm linux - how to create logical volumes

Steps to have lvm running on your system.

- find a partition or disk that is not used - I have /dev/sda10
- pvcreate /dev/sda10 (creates the physical volume)
- vgcreate VolGroup00 /dev/sda10 (creates a volume group)
- lvcreate -L 40GB -nTempVolume VolGroup00 (create a volume with 40GB with the name TempVolume under the volume group VolGroup00)
- mkfs.ext3 /dev/mapper/VolGroup00-TempVolume (creates the filesystem ext3 type on the TempVolume)
- mkdir /mnt/IMAGE (create a directory to mount the volume)
- mount /dev/mapper/VolGroup00-TempVolume /mnt/IMAGE
- mount (shows /dev/mapper/VolGroup00-TempVolume on /mnt/IMAGE type ext3 (rw))

That's it! You can start using the new mount point.

Rsync from linux mirrors without warning

Did you try to rsync and get lots of warnings because it can't change the owner or
the group ? This is caused by the fact that the rsync servers have a user id as owner
that doesn't exists on your system. Same goes for the group.

So if you want to have the verbose -v and no warnings you will need to use the
--no-o meaning no owner
--no-g meaning no group

This tells rsync on your server not to preserve the above

rsync -avz --no-g --no-o rsync://centos.mirror/centos/5/os/i386/ /my/repo

Build xen templates images with yum and rpm

Great example

http://www.virtuatopia.com/index.php/Building_a_Xen_Guest_Root_Filesystem_using_yum_and_rpm

Screen as window manager

In case that you didn't know screen is a window manager and a very useful one however for consoles ...
This is my .screenrc file

hardstatus alwayslastline
hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{=kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B}%Y-%m-%d %{W}%c %{g}]'
# Default screens
screen -t shell1 0
screen -t shell2 1
screen -t shell3 2
screen -t shell4 3

To switch between screens type CTRL+a 0 - goes on screen 0 - replace 0 with 1-3 to end up on a different window(screen)

To detach is a simple CTRL+a d - this will put screen into the background and you can reatach with
screen -r or if you have multiple sessions
screen -ls will list the screens that you have running and then
screen -r sessionName

Create from scratch a Xen image [Part-1]

This is a mini howto on creating from scratch a xen domU guest system cloning the
hostOS.

What you need:

- have a host os which runs xen in dom0 (you can install a centOS and choose virtualization as profile)
- some disk space (i put 20Gb for the / and 2 for swap)
- run the following two commands to set the volumes

pvcreate /dev/sda7 -- change the /dev/sda7 with whatever you want to
vgcreate VolGroup00 /dev/sda7 -- change the /dev/sda7 with whatever you want to

if unsure what to do look it on on how to create volumes in linux (google it ... )



The following script will create you and image:

#!/bin/sh

##########################################
echo $0 "Starting"                       #
echo "This script will clone the guestOS #
creating a new xen image"                #
#
# echo "Author: dioda11@gmail.com"       #
##########################################

if [ $@ ne 1 ]; then
echo "please give a numeric identifier for the clone
between 10 and 100"
exit 1;
# if ![ $1 lt 100 ]  
fi

IMGID=$1
BASEDIRMOUNT=/mnt

VOLGROUP=VolGroup00

lvcreate  -L 20G -n   LV_VM$IMGID_RH5.3_ROOT   $VOLGROUP
lvcreate  -L 2G -n   LV_VM$IMGID_RH5.3_SWAP  $VOLGROUP

mkfs.ext3  /dev/$VOLGROUP/LV_VM$IMGID_RH5.3_ROOT
mkswap  /dev/$VOLGROUP/LV_VM$IMGID_RH5.3_SWAP


if [ -d $BASEDIRMOUNT/$IMGID ]; then
mount /dev/$VOLGROUP/LV_VM$IMGID_RH5.3_ROOT $BASEDIRMOUNT/$IMGID
echo "Mounting the /dev/$VOLGROUP/LV_VM$IMGID_RH5.3_ROOT was " $? 
else
mkdir -p $BASEDIRMOUNT/$IMGID
echo "Creating $BASEDIRMOUNT/$IMGID was  " $?
mount /dev/$VOLGROUP/LV_VM$IMGID_RH5.3_ROOT $BASEDIRMOUNT/$IMGID
echo "Mounting the /dev/$VOLGROUP/LV_VM$IMGID_RH5.3_ROOT was " $? 
fi


echo "Cloning the host OS === "

ARCH=`uname -m`

if [ $ARCH == "x86_64" ]; then 
cp -ax /{bin,dev,etc,lib,lib64,root,sbin,usr,var} $BASEDIRMOUNT/$IMGID
else 
cp -ax /{bin,dev,etc,lib,root,sbin,usr,var} $BASEDIRMOUNT/$IMGID
fi

mkdir $BASEDIRMOUNT/$IMGID/{home,proc,opt,sys,tmp}
chmod 777 $BASEDIRMOUNT/$IMGID/tmp

echo "Cloning finished === " $?


echo "At this point you need to fix the /etc/fstab and create the xen config file"

# umount $BASEDIRMOUNT/$IMGID

------------------------

My .vimrc file for vim text editor

I tried a few big names editors but I tell you what for *nix console is nothing else like vi (new vi called vim) ... ok I admit perhaps emacs-nox but I don't use it.

In order to get a little from a lot of options of vim I'll show you my .vimrc file

set softtabstop=2
set shiftwidth=2
set expandtab
set cindent
set smartindent
set autoindent
syntax enable

This enables the indentation and puts spaces instead of tab characters into the file.

For more info see - the following

https://github.com/dioda11/dotfiles