Monday, June 13, 2011

Quick start with GlusterFS

The software that www.gluster.org makes allows to have distributed file systems with commodity hardware.


Rpm
===

http://download.gluster.com/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-fuse-3.2.0-1.x86_64.rpm
http://download.gluster.com/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-core-3.2.0-1.x86_64.rpm
http://download.gluster.com/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-rdma-3.2.0-1.x86_64.rpm


Gluster Daemon
==============

/etc/init.d/glusterd start


Firewall
========

iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24007:24008 -j ACCEPT
iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24009:24014 -j ACCEPT


Bricks
======

gluster peer probe SERVER_NAME_OR_IP


Volumes
=======

    Creation
    ========
    # replicated (mirror onto 2 hosts)
    gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2

    Start
    =====
    gluster volume start test-volume

    List
    ====
    gluster volume info (all|vol_name)

    Mount
    =====
    mount -t glusterfs HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR




Saturday, May 7, 2011

Ssh execute remote commands

Ssh is a very very useful tool and some tricks with it will help you a lot of time and typing. For example I want to transfer a file remote like my ssh-key file. I have a few options:

  • use scp to transfer the file, then login into the remote system and execute the commands.
  • use a different mechanism to transfer the files(ftp etc), login and execute the commands.
  • do it all in one command ... this is the cool one - see bellow


shell$ cat ~sd/.ssh/id_rsa.pub | ssh root@192.168.0.105 'cat - > .ssh/authorized_keys'
shell$ cat ~sd/.ssh/id_rsa.pub | ssh root@192.168.0.105 'cat - > .ssh/authorized_keys2'

# usually the authorized_keys is a link to authorized_keys2 but in this I just have two separate files.
# as you can see is nothing else to do :)


Thursday, April 28, 2011

Am I hacked ?

You do a ps -ef and you think is all good ... but perhaps what you see is not exactly what is really running ... This is a simple but effective way to compare the running processes reported by ps with what is into /proc

shell$ ps ax | wc -l
shell$ 30
shell$ ls -d /proc/* | grep [0-9]|wc -l
shell$ 31 # there is one extra root kit perhaps :)

Tuesday, April 26, 2011

What happens when you do kill a program in linux ?

I had two simple questions:


  • q1: how to you stop(kill) a program in linux ?

  • a1: i use the kill command as in
    kill 99 or kill -9 99

  • q2: ok ... so what does it really happens ?

  • a2: himm ... good question - well i send a signal to the program via a system call and then the kernel will take care of the rest ... as in will kill the program
    q2.1: himm so how does it kill it ?! what does it really happens
    a2.1: you know what let me think about it ... yeah i didn't look into this - well let me trace it and will find out ...
    shell$ bash &
    [2] 29120
    shell$ strace kill 29120
    execve("/usr/bin/kill", ["kill", "29120"], [/* 23 vars */]) = 0
    brk(0)                                  = 0x8849000
    access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
    open("/etc/ld.so.cache", O_RDONLY)      = 3
    fstat64(3, {st_mode=S_IFREG|0644, st_size=24036, ...}) = 0
    mmap2(NULL, 24036, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f1e000
    close(3)                                = 0
    open("/lib/libc.so.6", O_RDONLY)        = 3
    read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340\17K\0004\0\0\0"..., 512) = 512
    fstat64(3, {st_mode=S_IFREG|0755, st_size=1611564, ...}) = 0
    mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f1d000
    mmap2(0x49b000, 1332676, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x110000
    mprotect(0x24f000, 4096, PROT_NONE)     = 0
    mmap2(0x250000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13f) = 0x250000
    mmap2(0x253000, 9668, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x253000
    close(3)                                = 0
    mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f1c000
    set_thread_area({entry_number:-1 -> 6, base_addr:0xb7f1c6c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
    mprotect(0x250000, 8192, PROT_READ)     = 0
    mprotect(0x497000, 4096, PROT_READ)     = 0
    munmap(0xb7f1e000, 24036)               = 0
    brk(0)                                  = 0x8849000
    brk(0x886a000)                          = 0x886a000
    kill(29120, SIGTERM)                    = 0
    exit_group(0)                           = ?
    
    
    from the second line at the bottom i can see that a kill(PID, SIGTERM) was sent to the process and the return code is 0 (meaning success), but does it really happens into the kernel ?! - it will take me a lot more to explain but I found a good article about it at http://www.ibm.com/developerworks/library/l-linux-process-management/

Submit puzzle to facebook's puzzle master

Facebook runs a robot that takes email attachments and runs them to solve a puzzle that is posted
at http://www.facebook.com/careers/puzzles.php#!/careers/puzzles.php .

This is what I did to submit the hoppity puzle

shell$ echo 15 > file.txt 
shell$ python hoppity.py file.txt 
Hoppity
Hohpop
Hoppity
Hoppity
Hohpop
Hoppity
Hop
shell$ cat hoppity.py
#!/usr/bin/env python


import sys
if len(sys.argv) != 2:
    print 'run it as ', __file__, 'file.txt # file.txt should contain one unsigned int'
    sys.exit(1)

_file=sys.argv[1]

try:
    f = open(_file, 'r')
except IOError, ioe:
    print "file %s does not exist " %  _file
except:
    print "can not open file %s" % _file


no = f.read() # assume ONE uint in _file
max = int(no.strip()) + 1

for i in xrange(1,max):
    if i % 3 == 0 and i % 5 == 0 :
            print 'Hop'
    elif i % 3 == 0: print 'Hoppity'
    elif i % 5 == 0: print 'Hophop'

try:
    f.close()
except:
    pass

to actually submit the program - archive it as

mv hoppity.py hoppity && tar cvfz hoppity.tar.gz hoppity.py # the bot doesn't take the extension so you have to cut it off
and send an email with the archive attached to 1051962371@fb.com

Monday, April 25, 2011

What pid has my shell ?

Sometimes you are logged into a system on different terminals and you want to figure it out what process id you have on a specific terminal. The commands to do so are very simple:


shell$ tty
/dev/pts/1
# i'm on pts/1 meaning remote

shell$ ps -p $$
 PID TTY          TIME CMD
10044 pts/1    00:00:00 bash
# $$ means current process pid

Sunday, April 24, 2011

Tracing a telnet session with strace

Sometime you just need to know if a port is open on a remote system. The simplest way to find out if the port is open is to just telnet into the host and the port number.
This should look like:

shell$ telnet localhost  23
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host: Connection refused

# let's redo it with the strace enabled.
strace -vo strace.telnet telnet localhost  23
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host: Connection refused
# the error is the same but now we do have the strace.telnet file to find more info

shell$ cat strace.telnet
execve("/usr/kerberos/bin/telnet", ["telnet", "localhost", "23"], ["HOSTNAME=localhost.localdomain", "TERM=xterm-color", "SHELL=/bin/bash", "HISTSIZE=1000", "SSH_CLIENT=10.211.55.2 62489 22", "SSH_TTY=/dev/pts/0", "USER=root", "LS_COLORS=no=00:fi=00:di=01;34:l", "MAIL=/var/spool/mail/root", "PATH=/usr/kerberos/sbin:/usr/ker", "INPUTRC=/etc/inputrc", "PWD=/root", "LANG=en_US.UTF-8", "SHLVL=1", "HOME=/root", "LOGNAME=root", "SSH_CONNECTION=10.211.55.2 62489", "LESSOPEN=|/usr/bin/lesspipe.sh %", "G_BROKEN_FILENAMES=1", "_=/usr/bin/strace", "OLDPWD=/usr/src"]) = 0
brk(0)                                  = 0x9c8d000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_dev=makedev(3, 1), st_ino=129997, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=56, st_size=26551, st_atime=2011/04/21-14:46:38, st_mtime=2011/04/21-07:09:59, st_ctime=2011/04/21-07:09:59}) = 0
mmap2(NULL, 26551, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7fe5000
close(3)                                = 0
open("/usr/lib/libkrb4.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0pB(\0004\0\0\0"..., 512) = 512
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fe4000
fstat64(3, {st_dev=makedev(3, 1), st_ino=239248, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=208, st_size=100960, st_atime=2011/04/21-14:46:38, st_mtime=2010/01/12-19:22:52, st_ctime=2011/04/08-04:02:50}) = 0
mmap2(NULL, 117948, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6d3000
mmap2(0x6ea000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17) = 0x6ea000
mmap2(0x6eb000, 19644, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x6eb000
close(3)                                = 0
open("/usr/lib/libdes425.so.3", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\300\354)\0004\0\0\0"..., 512) = 512
fstat64(3, {st_dev=makedev(3, 1), st_ino=236616, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=32, st_size=12816, st_atime=2011/04/21-14:46:38, st_mtime=2010/01/12-19:22:52, st_ctime=2011/04/08-04:02:41}) = 0
mmap2(NULL, 13868, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x1d9000
mmap2(0x1dc000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0x1dc000
close(3)                                = 0
.......
.......
fstat64(1, {st_dev=makedev(0, 12), st_ino=2, st_mode=S_IFCHR|0620, st_nlink=1, st_uid=0, st_gid=5, st_blksize=4096, st_blocks=0, st_rdev=makedev(136, 0), st_atime=2011/04/21-14:46:38, st_mtime=2011/04/21-14:46:38, st_ctime=2011/04/21-04:36:50}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7feb000
write(1, "Trying 127.0.0.1...\r\n", 21) = 21
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_IP, IP_TOS, [16], 4)  = 0
connect(3, {sa_family=AF_INET, sin_port=htons(23), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)
write(2, "telnet: connect to address 127.0"..., 57) = 57
close(3)                                = 0
write(2, "telnet: Unable to connect to rem"..., 61) = 61
exit_group(1) 

# as you can see there is a lot of information and some i replaced with ..... 
# the line of interest will be 

connect(3, {sa_family=AF_INET, sin_port=htons(23), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)

# this tells when the socket is actually trying to connect onto the remote host and had a return code of -1, all after it is just output from the telnet program that formats it very carefully.

Thursday, April 21, 2011

Check a linux filesystem with an alternate superblock

A filesystem contains different data structures after is created and one of the most important things that is present is the superblock - because is that important there is more then one superblock.
How to find the superblocks and how to do the filesystem check will be shown bellow.

Since some partitions are labeled will we need to find the associated device of the label. How to look on for the label/device association is to follow.


# this is my /etc/fstab

LABEL=/                 /                       ext3    defaults        1 1
LABEL=/opt              /opt                    ext3    defaults        1 2
LABEL=/tmp              /tmp                    ext3    defaults        1 2
LABEL=/usr              /usr                    ext3    defaults        1 2
LABEL=/home             /home                   ext3    defaults        1 2
LABEL=/logs             /logs                   ext3    defaults        1 2
LABEL=/var              /var                    ext3    defaults        1 2
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda9         swap                    swap    defaults        0 0

# i want to check the partition with label /opt  
# first step - find what device is associated with the label /opt
# to do this we use the command e2label
# if you have one or two partitions is as simple as running e2label on those
# i have a few partitions so i made a small chained command 

root# for i in `mount | awk '{print $1}' | grep '/'`; do echo -n "$i=" && e2label $i   ;done
/dev/hda5=/
/dev/hda8=/tmp
/dev/hda7=/usr
/dev/hda6=/home
/dev/hda3=/logs
/dev/hda2=/var
/dev/hda1=/boot
/dev/hda10=/opt # this is the one I need

# running dumpe2fs
root# dumpe2fs /dev/hda10 |grep 'Backup superblock'
  Backup superblock at 32768, Group descriptors at 32769-32769
  Backup superblock at 98304, Group descriptors at 98305-98305
  Backup superblock at 163840, Group descriptors at 163841-163841
  Backup superblock at 229376, Group descriptors at 229377-229377
  Backup superblock at 294912, Group descriptors at 294913-294913
  Backup superblock at 819200, Group descriptors at 819201-819201
  Backup superblock at 884736, Group descriptors at 884737-884737
  Backup superblock at 1605632, Group descriptors at 1605633-1605633

# note that you may have a different output
# the number after Backup superblock at is the superblock you want

# run fsck.ext3 or fsck.ext2 ... or any other command for your filesystem (reiser etc)
# first umount the partition

root# umount /opt
# thenn fsck  
root# fsck.ext3  -b 32768 /dev/hda10

# after you are done mount back the partition



Wednesday, April 20, 2011

Transfer files between two host with nc

Problem: you have two hosts that you can access but there is no mechanism
to transfer files between them - no ssh(scp/sftp), no ftp etc.
How to do it ?!

Solution: use nc and tar/dd/echo ...

# transfer by tar of a directory
|destination_host|                     |source_host|
nc -l 9000 | tar xvf -                  tar cvf - /my_dir  | nc destination_host 9000

# we listen on all interfaces          # we tar my_dir to STDOUT(-) and all is piped to 
# on port 9000, all that comes in      # nc that will connect on destination_host on port 9000
# is piped to tar xvf (will extract)   # and will transfer what ever is given
# - means take the STDIN

# transfer by dd of a partition /dev/sda3
|destination_host|                     |source_host|
nc -l 9000 | dd of=/backup_device      dd if=/dev/sda3  | nc destination_host 9000

# we listen on all interfaces          # we dd in /dev/sda3 (reading) all is piped to 
# on port 9000, all that comes in      # nc that will connect on destination_host on port 9000
# is piped to dd and dd will write it  # and will transfer what ever is given
# all to /backup_device

As you can see this becomes very useful because you can open the destination port as you need it
and even transfer from a block device as /dev/sda3 in the example. Once the transfer is done
on the destination host nc stops to listen for the port you asked(there is a switch to make it
to continue listening -k, this won't work -l however).

Wednesday, February 23, 2011

sqlalchemy UUID as primary key

I keep hearing about having uuid as primary keys into your database so I decided to give a try with sqlalchemy and python(of course).

So the plan is to have a table users that has the following fields

id - type UUID - 32
fname - varchar(50)
lname - varchar(50)

the code bellow builds the table for me.

1 from sqlalchemy import Table, Column, Integer, String, Sequence, MetaData, ForeignKey, CHAR
  2 from sqlalchemy.orm import mapper, sessionmaker, scoped_session
  3 from sqlalchemy import create_engine
  4 import uuid
  5 
  6 metadata = MetaData()
  7 
  8 users = Table('users', metadata,
  9               Column('id', CHAR(32), primary_key=True, autoincrement=True),
 10               Column('fname', String(50)),
 11               Column('lname', String(50))
 12              )
 13 # orm
 14 class Users(object):
 15     def __init__(self, fname, lname):
 16         assert isinstance(fname, str), 'fname is not a string'
 17         assert isinstance(lname, str), 'lname is not a string'
 18         self.fname = fname
 19         self.lname = lname
 20 
 21     
 22     
 23 mapper(Users, users, version_id_col=users.c.id, version_id_generator = lambda version:uuid.uuid4().hex)



line of interest will be
4 - import the uuid module (standard with python 2.6 or higher)
23 - the version_id_col=users.c.id and version_id_generator = lambda version:uuid.uuid4().hex)
the explanation for it is http://www.sqlalchemy.org/docs/orm/mapper_config.html?highlight=uuid#sqlalchemy.orm.mapper
basically you map a temporary integer used my sqlaclhemy and then you transform it into the UUID with a 32 chars in hex.

the rest of the program


24 
 25 engine = create_engine('sqlite:///:memory:', echo=True)
 26 Session = scoped_session(sessionmaker(bind=engine))
 27 metadata.drop_all(engine)
 28 metadata.create_all(engine)
 29 # my test 
 30 session = Session()
 31 
 32 
 33 u = Users('s', 'd');
 34 u1 = Users('s1', 'd2');
 35 
 36 session.add_all([u1, u])
 37 session.commit()
 38 
 39 session.query(Users).all()

if uuid are better then integers as primary keys I don't think so - at least with mysql taking
into consideration the following article http://www.mysqlperformanceblog.com/2007/03/13/to-uuid-or-not-to-uuid/
but they 'hide' your data from outside and seem to do the job.

Wednesday, February 16, 2011

MySQLdb (mysql-python) install on OSX 10.6 Snow Leopard (32 bits)

Ok - you have mysql server installed into /usr/local/mysql and you are thinking - yes I can connect from python to it like on my linux box ... but on 10.6 OSX is a bit different.
First a bit of light of what is happening:

  • the python you run from /usr/bin/python is compiled for 64 and 32 bits ! that is a fat binary as is called. do a file /usr/bin/python and you will see something like
    usr/bin/python: Mach-O universal binary with 3 architectures
    /usr/bin/python (for architecture x86_64): Mach-O 64-bit executable x86_64
    /usr/bin/python (for architecture i386): Mach-O executable i386
    /usr/bin/python (for architecture ppc7400): Mach-O executable ppc
    

  • the mysql server that you installed is 32 bits only !

  • the code for MySQLdb can be compiled for either architecture but not two at ones as into the fat binary above


Steps to instal

  • have the mysql server installed - source, archive or dmg - the best location to install is /usr/local/mysql
  • if you use virtual environment it is best to extract the 32 bits version from the fat python into your environment. same goes for 64 bits if you use it.
    to extract do something like this after you have your virtual environment -
    cp /my_virtual/env/bin/python /my_virtual/env/bin/python.fat
    lipo -remove x86_64 /my_virtual/env/bin/python.fat -output /my_virtual/env/bin/python
    
    -- to check if you are using 32 bits
    python
    >>> import sys
    >>> sys.maxint
    2147483647
    
  • install mysql-python wit pip/easy_install or from source

errors you may see and how to solve them
  • >>> import MySQLdb
    Traceback (most recent call last):
      File "", line 1, in 
      File "/Users/silviud/PROGS/PYTHON/Environments/2.6/lib/python2.6/site-packages/MySQLdb/__init__.py", line 19, in 
        import _mysql
    ImportError: dlopen(/Users/silviud/PROGS/PYTHON/Environments/2.6/lib/python2.6/site-packages/_mysql.so, 2): Library not loaded: libmysqlclient.16.dylib
      Referenced from: /Users/silviud/PROGS/PYTHON/Environments/2.6/lib/python2.6/site-packages/_mysql.so
      Reason: image not found
    
    This is because the dynamic loader can not find the library libmysqlclient.16.dylib which is located into /usr/local/mysql/lib - to solve it add this to your .profile file

    export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/usr/local/mysql/lib
    


This is what I have done to make it work !
I've seen other solutions where you would have to choose the python architecture with an environment variable as

export VERSIONER_PYTHON_PREFER_32_BIT=yes
or
to have it system wide available with
defaults write com.apple.versioner.python Prefer-32-Bit -bool yes
but NONE worked for me except what I shown above.
I even tried to load static the mysql library into the mysql-python by changing the site.cfg from the dist but no luck.

In any case I don't suggest you do this for the a system wide installation - use virtual environment!

Tuesday, January 18, 2011

Mysql regex query

Mysql supports regexes into their queries - this is how you would do it.

desc PENDING_ORDER

+-------------+--------------+------+-----+---------+-------+
| Field       | Type         | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+-------+
| CustomerID  | varchar(255) | NO   | PRI | NULL    |       | 
| DateCreated | datetime     | NO   |     | NULL    |       | 
| XML_DATA    | text         | NO   |     | NULL    |       | 
| Status      | varchar(20)  | NO   |     | NULL    |       | 
+-------------+--------------+------+-----+---------+-------+

-- find all values that have a new line after the last digit 
 
select XML_DATA as xml from PENDING_ORDER  where XML_DATA REGEXP '[0-9]+\n';

-- Output

< property>
< name> PartnerCustomerID </name>
< value> 999999579
                        </value>
</property>


Monday, January 17, 2011

rotate nohup out file (nohup.log)

I got this article by posting on serverfault.com on how to rotate the nohup log file.
Basically you connect the output of nohup to a pipe that is redirecting to a file - then the file can me moved around very easily.

mknod /tmp/mypipe p
cat < /tmp/mypipe >/tmp/myreallog
nohup myapplication >/tmp/mypipe


To rotate the log:
mv /tmp/myreallog /tmp/rotatedlog
kill [pid of the cat process]
cat < /tmp/mypipe >/tmp/myreallog


Tuesday, December 14, 2010

Remote mount trough ssh - sshfs

You may need to work on remote hosts file systems and copying files over with scp/sftp may be a hassle. This is why sshfs is in place - you can mount remote file systems locally on your computer


How to do it (on ubuntu) - install first the sshfs


$ sudo aptitude install sshfs
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
The following NEW packages will be installed:
  sshfs
0 packages upgraded, 1 newly installed, 0 to remove and 6 not upgraded.
Need to get 40.8kB of archives. After unpacking 143kB will be used.
Writing extended state information... Done
Get:1 http://ca.archive.ubuntu.com/ubuntu/ lucid/main sshfs 2.2-1build1 [40.8kB]
Fetched 40.8kB in 0s (44.3kB/s)
Selecting previously deselected package sshfs.
(Reading database ... 197354 files and directories currently installed.)
Unpacking sshfs (from .../sshfs_2.2-1build1_i386.deb) ...
Processing triggers for man-db ...
Setting up sshfs (2.2-1build1) ...
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Writing extended state information... Done


then is as easy as

$ mkdir /tmp/remote_host_name_yours
$ sshfs your_remote_host: /tmp/remote_host_name_yours
$ cd /tmp/remote_host_name_yours

that is it ! (you may be need to input a password or paraphrase)
to umount the remote file system you need to use


$ fusermount -u /tmp/remote_host_name_yours


for more info see http://fuse.sourceforge.net/sshfs.html

Tuesday, November 23, 2010

Recursive grep with filters - command line

Problem:
you have a list o directories that you want to search a pattern BUT you want to exclude a few directories

Solution:
use ls and grep recursive and apply a filter for the directory that you want to exlude


shell$ pwd
/home/
shell$ ls
work documents logs bin
# you want to search on all except logs for pattern 'invoice'

shell$ for i in `ls` ;do  if [ $i != 'logs' ]; then grep -r 'invoice' $i; fi; done
... output


what it happens is this:

the for loop executes based on the command `ls`
all the output is taken and feed it to the do
inside the do we are filtering off 'logs' (with the if) and then search for invoice with grep -r

Thursday, November 18, 2010

Vi(vim) column editing

In case that you want to edit by column and you use vi(vim) - this will do it.

To use it, press:

* Ctrl + V to go into column mode
* Select the columns and rows where you want to enter your text
* Shift + i to go into insert mode in column mode
* Type in the text you want to enter. You will see changes ONLY into the first row that you edit !
* Esc to apply your change (or alternately Ctrl+c)

You will now see your changed applied.

Ssh dynamic port(SOCKS) - firefox with different profiles

I have a data center network that is accessible only from a host - everything that is
behind it has to go trough it - this is usually what it happens to access  data centers and different solution being a site to site vpn (or road warrior - basically from your laptop to a firewall or access server).

I do deployments onto the servers that are behind this bastion host and I need to access the web applications ... how to do it ?!

Use openssh with a dynamic tunnel or putty (I use openssh) !

How to do it:

This is the diagram

/ web01 port 9000
me ---- internet ---- bastion_host --- web02 port 9001
                                     \ web03 port 9002
me$ ssh -v -D 8080 bastion_host

This has created a SOCKS proxy on my machine port 8080. If you don't know what is a SOCKS proxy - read on it http://en.wikipedia.org/wiki/SOCKS


Configure firefox to use a SOCKS proxy. Go to Edit -> Preferences -> Advanced -> Network -> Settings






Now you would like to use the browser to read as well your gmail or just surf the web -  so you need two different profiles

- one that uses the SOCKS proxy
- one for general use

To do so start firefox like this:

firefox -ProfileManager -no-remote

-no-remote is to start a separate process for firefox(which is not default)
-ProfileManager is to choose your profile - you can create a new one and from there take off the SOCKS proxy to use it for web surfing.


Now when you are into the profile with SOCKS proxy type in

http://web01:9000 and you have a direct connection !

Fire up a new firefox and choose the no proxy profile and just use it.

Thursday, November 11, 2010

Port redirect in linux to remote host

I had an application that needs to connect to a LDAP port (tcp:389) but the problem was that the LDAP was on an external network and on a non standard port (tcp:1389).
Pointing direct to remote_ip:1389 was not and option because the application is taking the LDAP port from the openldap libs (LDAP_PORT) which is a constant integer = 389 ...

The solution proved to be very simple - install a small port redirect program rinted do a small configuration into .

# config
/etc/rinetd.conf
   
127.0.0.1     389        remote_ip        1389


# start the daemon (only if you install it from rpm - if not just start manually)
/etc/init.d/rinetd start



And this is it - all works just fine.

Note that doing an iptables PREROUTING and DNAT will not work in my case because iptables can do
redirects ONLY into the local network.

Saturday, November 6, 2010

Delete empty lines into a file

Problem - you have a file that has a huge number of records and in between you have
empty lines.
This is my example.txt file:
asd

asd
as


sd
as

asd

asd

sd
g
To take the spaces off you have at least two simple tools to do it.
  • sed with this tool you can achieve your goal just by doing this - open a command line prompt and type the following
    linux$ cat example.txt | sed -e '/^$/d' you will see on your screen the following
    asd
    asd
    as
    sd
    as
    asd
    asd
    sd
    g
    
  • grep with this we use a trick saying that we don't want certain patterns from the file - the empty lines - open a command line prompt and type the following
    linux$ grep -v '^$' example.txt
  • asd
    asd
    as
    sd
    as
    asd
    asd
    sd
    g
    

Friday, October 15, 2010

Html to XHTML/XML with tidy

As you all know the browser will perform in quirks mode
www.quirksmode.org
if you don't specify into your html file what type of document is it.
This will make your life super difficult because browser will display your pages as they wish.
The best thing to do then is to declare what type of document you present.

If there is only one page to change of course it can be done manually but if you have
many pages to do is always better to use a programmatic approach.
So tidy is to rescue !
Tidy is a small program that runs on Linux and can help on this. More info can be found at tidy.sourceforge.net

So let's get practical:
problem - you have a large number of files you want to convert from simple html to xhtml or xml.
solution - open a terminal (xterm. gterm or anything you want - works on direct console as well)
and type the following

for i in `find ./*.html -type f`; do tidy -asxhtml  < $i > $i.xhtml ; done 

Q: What just happened ?
A1: The command looked for all the *.html (all files ending with html extension) into the current directory and then feed tidy with the files matched.
Tidy took the file as input converted to xhtml (option -asxhtml) and wrote it back to the disk with the original_name.html.xhtml
A2: Into each file that was wrote by tidy you can see the new tags that are necessary for xhtml.

For example into the original file you had
<html>
        <head>
                <title> me - home page  </title>
        </head>
        <body>
                <h1> my home page </h1>
                <p> Lorem ipsum dolor sit amet, consectetur adipiscing elit.
                        Morbi vitae lorem justo. Cum sociis natoque penatibus et
                        magnis dis parturient montes, nascetur ridiculus mus.
                        Sed sed consectetur massa. Morbi ac erat sit amet eros
                        malesuada pellentesque tempus ultricies dui. Nam laoreet nibh
                        sit amet massa pellentesque fermentum. Aenean suscipit laoreet
                        lorem sed dignissim. Integer dapibus pulvinar lorem ac placerat.
                        Mauris suscipit, risus commodo tempor feugiat, mi metus facilisis
                        quam, auctor sagittis magna mi vitae lacus. Suspendisse purus velit,
                        ultrices at iaculis dictum, rutrum nec tortor.
                        Nullam a libero ut erat semper mollis in et lectus.
                        Sed suscipit lorem eu mi tristique volutpat. Proin lobortis
                        vehicula nunc vel condimentum. Aliquam mattis, lacus eu adipiscing
                        posuere, dui enim sodales felis, et tincidunt nulla urna lacinia velit.
                </p>
        </body>
</html>


Now you have

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta name="generator" content=
"HTML Tidy for Linux/x86 (vers 25 March 2009), see www.w3.org" />
<title>me - home page</title>
</head>
<body>
<h1>my home page</h1>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi
vitae lorem justo. Cum sociis natoque penatibus et magnis dis
parturient montes, nascetur ridiculus mus. Sed sed consectetur
massa. Morbi ac erat sit amet eros malesuada pellentesque tempus
ultricies dui. Nam laoreet nibh sit amet massa pellentesque
fermentum. Aenean suscipit laoreet lorem sed dignissim. Integer
dapibus pulvinar lorem ac placerat. Mauris suscipit, risus commodo
tempor feugiat, mi metus facilisis quam, auctor sagittis magna mi
vitae lacus. Suspendisse purus velit, ultrices at iaculis dictum,
rutrum nec tortor. Nullam a libero ut erat semper mollis in et
lectus. Sed suscipit lorem eu mi tristique volutpat. Proin lobortis
vehicula nunc vel condimentum. Aliquam mattis, lacus eu adipiscing
posuere, dui enim sodales felis, et tincidunt nulla urna lacinia
velit.</p>
</body>
</html>


That's it - now you have xhtml !
You can change the above command to make tidy do other useful things as preserving the indentation level, clean the file by some tags etc.