Remote mount trough ssh - sshfs

You may need to work on remote hosts file systems and copying files over with scp/sftp may be a hassle. This is why sshfs is in place - you can mount remote file systems locally on your computer

How to do it (on ubuntu) - install first the sshfs

$ sudo aptitude install sshfs
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
The following NEW packages will be installed:
  sshfs
0 packages upgraded, 1 newly installed, 0 to remove and 6 not upgraded.
Need to get 40.8kB of archives. After unpacking 143kB will be used.
Writing extended state information... Done
Get:1 http://ca.archive.ubuntu.com/ubuntu/ lucid/main sshfs 2.2-1build1 [40.8kB]
Fetched 40.8kB in 0s (44.3kB/s)
Selecting previously deselected package sshfs.
(Reading database ... 197354 files and directories currently installed.)
Unpacking sshfs (from .../sshfs_2.2-1build1_i386.deb) ...
Processing triggers for man-db ...
Setting up sshfs (2.2-1build1) ...
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Writing extended state information... Done

then is as easy as

$ mkdir /tmp/remote_host_name_yours
$ sshfs your_remote_host: /tmp/remote_host_name_yours
$ cd /tmp/remote_host_name_yours

that is it ! (you may be need to input a password or paraphrase)
to umount the remote file system you need to use

$ fusermount -u /tmp/remote_host_name_yours

for more info see http://fuse.sourceforge.net/sshfs.html

Recursive grep with filters - command line

Problem:
you have a list o directories that you want to search a pattern BUT you want to exclude a few directories

Solution:
use ls and grep recursive and apply a filter for the directory that you want to exlude

shell$ pwd
/home/
shell$ ls
work documents logs bin
# you want to search on all except logs for pattern 'invoice'

shell$ for i in `ls` ;do  if [ $i != 'logs' ]; then grep -r 'invoice' $i; fi; done
... output

what it happens is this:

the for loop executes based on the command `ls`
all the output is taken and feed it to the do
inside the do we are filtering off 'logs' (with the if) and then search for invoice with grep -r

Vi(vim) column editing

In case that you want to edit by column and you use vi(vim) - this will do it.

To use it, press:

* Ctrl + V to go into column mode
* Select the columns and rows where you want to enter your text
* Shift + i to go into insert mode in column mode
* Type in the text you want to enter. You will see changes ONLY into the first row that you edit !
* Esc to apply your change (or alternately Ctrl+c)

You will now see your changed applied.

Ssh dynamic port(SOCKS) - firefox with different profiles

I have a data center network that is accessible only from a host - everything that is
behind it has to go trough it - this is usually what it happens to access  data centers and different solution being a site to site vpn (or road warrior - basically from your laptop to a firewall or access server).

I do deployments onto the servers that are behind this bastion host and I need to access the web applications ... how to do it ?!

Use openssh with a dynamic tunnel or putty (I use openssh) !

How to do it:

This is the diagram

/ web01 port 9000
me ---- internet ---- bastion_host --- web02 port 9001
                                     \ web03 port 9002
me$ ssh -v -D 8080 bastion_host

This has created a SOCKS proxy on my machine port 8080. If you don't know what is a SOCKS proxy - read on it http://en.wikipedia.org/wiki/SOCKS


Configure firefox to use a SOCKS proxy. Go to Edit -> Preferences -> Advanced -> Network -> Settings

Now you would like to use the browser to read as well your gmail or just surf the web -  so you need two different profiles

- one that uses the SOCKS proxy
- one for general use

To do so start firefox like this:

firefox -ProfileManager -no-remote

-no-remote is to start a separate process for firefox(which is not default)
-ProfileManager is to choose your profile - you can create a new one and from there take off the SOCKS proxy to use it for web surfing.


Now when you are into the profile with SOCKS proxy type in

http://web01:9000 and you have a direct connection !

Fire up a new firefox and choose the no proxy profile and just use it.

Port redirect in linux to remote host

I had an application that needs to connect to a LDAP port (tcp:389) but the problem was that the LDAP was on an external network and on a non standard port (tcp:1389).
Pointing direct to remote_ip:1389 was not and option because the application is taking the LDAP port from the openldap libs (LDAP_PORT) which is a constant integer = 389 ...

The solution proved to be very simple - install a small port redirect program rinted do a small configuration into .

# config
/etc/rinetd.conf
   
127.0.0.1     389        remote_ip        1389


# start the daemon (only if you install it from rpm - if not just start manually)
/etc/init.d/rinetd start

And this is it - all works just fine.

Note that doing an iptables PREROUTING and DNAT will not work in my case because iptables can do
redirects ONLY into the local network.

Delete empty lines into a file

Problem - you have a file that has a huge number of records and in between you have
empty lines.
This is my example.txt file:

asd

asd
as


sd
as

asd

asd

sd
g

To take the spaces off you have at least two simple tools to do it.

• sed with this tool you can achieve your goal just by doing this - open a command line prompt and type the following
  linux$ cat example.txt | sed -e '/^$/d' you will see on your screen the following
  

  asd
  asd
  as
  sd
  as
  asd
  asd
  sd
  g
  

• grep with this we use a trick saying that we don't want certain patterns from the file - the empty lines - open a command line prompt and type the following
  linux$ grep -v '^$' example.txt
  

asd
asd
as
sd
as
asd
asd
sd
g

Html to XHTML/XML with tidy

As you all know the browser will perform in quirks mode
www.quirksmode.org if you don't specify into your html file what type of document is it.
This will make your life super difficult because browser will display your pages as they wish.
The best thing to do then is to declare what type of document you present.

If there is only one page to change of course it can be done manually but if you have
many pages to do is always better to use a programmatic approach.
So tidy is to rescue !
Tidy is a small program that runs on Linux and can help on this. More info can be found at tidy.sourceforge.net

So let's get practical:
problem - you have a large number of files you want to convert from simple html to xhtml or xml.
solution - open a terminal (xterm. gterm or anything you want - works on direct console as well)
and type the following

for i in `find ./*.html -type f`; do tidy -asxhtml  < $i > $i.xhtml ; done 

Q: What just happened ?
A1: The command looked for all the *.html (all files ending with html extension) into the current directory and then feed tidy with the files matched.
Tidy took the file as input converted to xhtml (option -asxhtml) and wrote it back to the disk with the original_name.html.xhtml
A2: Into each file that was wrote by tidy you can see the new tags that are necessary for xhtml.

For example into the original file you had
<html>
        <head>
                <title> me - home page  </title>
        </head>
        <body>
                <h1> my home page </h1>
                <p> Lorem ipsum dolor sit amet, consectetur adipiscing elit.
                        Morbi vitae lorem justo. Cum sociis natoque penatibus et
                        magnis dis parturient montes, nascetur ridiculus mus.
                        Sed sed consectetur massa. Morbi ac erat sit amet eros
                        malesuada pellentesque tempus ultricies dui. Nam laoreet nibh
                        sit amet massa pellentesque fermentum. Aenean suscipit laoreet
                        lorem sed dignissim. Integer dapibus pulvinar lorem ac placerat.
                        Mauris suscipit, risus commodo tempor feugiat, mi metus facilisis
                        quam, auctor sagittis magna mi vitae lacus. Suspendisse purus velit,
                        ultrices at iaculis dictum, rutrum nec tortor.
                        Nullam a libero ut erat semper mollis in et lectus.
                        Sed suscipit lorem eu mi tristique volutpat. Proin lobortis
                        vehicula nunc vel condimentum. Aliquam mattis, lacus eu adipiscing
                        posuere, dui enim sodales felis, et tincidunt nulla urna lacinia velit.
                </p>
        </body>
</html>


Now you have

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta name="generator" content=
"HTML Tidy for Linux/x86 (vers 25 March 2009), see www.w3.org" />
<title>me - home page</title>
</head>
<body>
<h1>my home page</h1>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi
vitae lorem justo. Cum sociis natoque penatibus et magnis dis
parturient montes, nascetur ridiculus mus. Sed sed consectetur
massa. Morbi ac erat sit amet eros malesuada pellentesque tempus
ultricies dui. Nam laoreet nibh sit amet massa pellentesque
fermentum. Aenean suscipit laoreet lorem sed dignissim. Integer
dapibus pulvinar lorem ac placerat. Mauris suscipit, risus commodo
tempor feugiat, mi metus facilisis quam, auctor sagittis magna mi
vitae lacus. Suspendisse purus velit, ultrices at iaculis dictum,
rutrum nec tortor. Nullam a libero ut erat semper mollis in et
lectus. Sed suscipit lorem eu mi tristique volutpat. Proin lobortis
vehicula nunc vel condimentum. Aliquam mattis, lacus eu adipiscing
posuere, dui enim sodales felis, et tincidunt nulla urna lacinia
velit.</p>
</body>
</html>


That's it - now you have xhtml !
You can change the above command to make tidy do other useful things as preserving the indentation level, clean the file by some tags etc.

Python re multiline match (inline)

Compilation Flags -- inline

Python regex flags effect matching. For example: re.MULTILINE, re.IGNORECASE, re.DOTALL. Unfortunately, passing these flags is awkward if want to put regex patterns in a config file or database or otherwise want to have the user be able to enter in regex patterns. You don't want to have to make the user pass in flags separately. Luckily, you can pass flags in the pattern itself. This is a very poorly documented feature of Python regular expressions. At the start of the pattern add flags like this:

(?i) for re.IGNORECASE
 (?L) for re.LOCALE (Make \w, \W, \b, \B, \s and \S dependent on the current locale.)
 (?m) for re.MULTILINE  (Makes ^ and $ match before and after newlines)
 (?s) for re.DOTALL  (Makes . match newlines)
 (?u) for re.UNICODE (Make \w, \W, \b, \B, \d, \D, \s and \S dependent on the Unicode character properties database.)
 (?x) for re.VERBOSE

For example, the following pattern ignores case (case insensitive):

re.search ("(?i)password", string)

The flags can be combined. The following ignores case and matches DOTALL:

re.search ("(?is)username.*?password", string)

Django on Google App Engine

In case that you want to run the head (latest version) of django on the google app engine this will help you. The info bellow is for a new project (in case you need information on how to make it work with an
existing project see the README file from the appengine_helper).

First you need to understand what is different between running django on your server where you have a RDBMS like mysql and the datastore that google provides. To do this look into the following link
http://code.google.com/appengine/articles/appengine_helper_for_django.html

After you are done reading do the following:
(i'm using /tmp as an example, you can change it to whatever you want)

cd /tmp/

- download the appengine_helper_for_django (if the link changed because there is a new version replace bellow the names for helper with the new names)

curl -L -o 'appengine_helper_for_django-r105.zip' 'http://google-app-engine-django.googlecode.com/files/appengine_helper_for_django-r105.zip'


- download  the latest version of django (you can download it or use the svn to checkout)

curl -L -o 'Django-1.2.1.tar.gz' 'http://www.djangoproject.com/download/1.2.1/tarball/'


- untar the django

tar xvfz Django-1.2.1.tar.gz


- make a new temporary directory for your application

mkdir myapp_tmp


- extract the django helper 

cd myapp_tmp
unzip ../appengine_helper_for_django-r105.zip


- rename the helper

mv appengine_helper_for_django/ myapp


- copy the django framework to have it available

cp  -r ../Django-1.2.1/django  myapp


- configure the app and run the webserver

cd myapp
ls    (you should have something like this)


HANGES VERSION appengine_django manage.py urls.pyc
COPYING __init__.py django settings.py
KNOWN_ISSUES __init__.pyc index.yaml settings.pyc
README app.yaml main.py urls.py

- edit the app.yaml to reflect your application name etc.

- run the webserver to test

python manage.py runserver
(you should see the output from the app engine starting)

"""

/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore_file_stub.py:40: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5
/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/memcache/__init__.py:31: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha
WARNING:root:Could not read datastore data from /var/folders/TI/TIS6v9poFPyMZ3GaFkfkjE+++TQ/-Tmp-/django_vpsup.datastore
WARNING:root:Could not initialize images API; you are likely missing the Python "PIL" module. ImportError: No module named _imaging
INFO:google.appengine.tools.appengine_rpc:Server: appengine.google.com
INFO:root:Checking for updates to the SDK.
INFO:root:The SDK is up to date.

"""

that's it ! you can start using django as you normally do.

(in case you want to change the location of your myapp directory all you have to do is to move the myapp to the new destination - mv /tmp/myapp_tmp/myapp /home/myuser/applications/myapp)

URL - safe and unsafe characters

RFC - http://www.rfc-editor.org/rfc/rfc1738.txt

"Unsafe characters"
	Why:	Some characters present the possibility of being misunderstood within URLs for various reasons. These characters should also always be encoded.
	Characters:	Character Code Points (Hex) Code Points (Dec) Why encode? Space 20 32 Significant sequences of spaces may be lost in some uses (especially multiple spaces) Quotation marks 'Less Than' symbol ("<") 'Greater Than' symbol (">") 22 3C 3E 34 60 62 These characters are often used to delimit URLs in plain text. 'Pound' character ("#") 23 35 This is used in URLs to indicate where a fragment identifier (bookmarks/anchors in HTML) begins. Percent character ("%") 25 37 This is used to URL encode/escape other characters, so it should itself also be encoded. Misc. characters:    Left Curly Brace ("{")    Right Curly Brace ("}")    Vertical Bar/Pipe ("|")    Backslash ("\")    Caret ("^")    Tilde ("~")    Left Square Bracket ("[")    Right Square Bracket ("]")    Grave Accent ("`") 7B 7D 7C 5C 5E 7E 5B 5D 60 123 125 124 92 94 126 91 93 96 Some systems can possibly modify these characters.

"Reserved characters"
	Why:	URLs use some characters for special use in defining their syntax. When these characters are not used in their special role inside a URL, they need to be encoded.
	Characters:	Character Code Points (Hex) Code Points (Dec)  Dollar ("$")  Ampersand ("&")  Plus ("+")  Comma (",")  Forward slash/Virgule ("/")  Colon (":")  Semi-colon (";")  Equals ("=")  Question mark ("?")  'At' symbol ("@") 24 26 2B 2C 2F 3A 3B 3D 3F 40 36 38 43 44 47 58 59 61 63 64

Xen resize disk (image) into DomU

The objective is to increase the storage allocated to a xen DomU vm.
Let's say the situation before we do anything looks like this:

The virtual machine name (as seen into xm list)  rhel-5.3-HARMONY
The vm is using tapio disk image files (not lvm) for the hard drive.

root@localhost.localdomain[~]11:12:00# df -k

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/xvda 9048256 3198968 5481516 37% /
tmpfs 1048576 0 1048576 0% /dev/shm

I want to add an extra 1GB space to this the 9GB already in there (the red label above).

Steps to do this:

• Shutdown the virtual machine (vm) - xm shutdown rhel-5.3-HARMONY
• Make a backup of the disk  - cp rhel-5.3-HARMONY.img rhel-5.3-HARMONY.img.backup
• Create an empty file with the desired size (I choosed 1GB) - dd if=/dev/zero of=ZeroContentFile bs=1024 count=1000000
• Append the new empty file to the initial image file - cat ZeroContentFile >> rhel-5.3-HARMONY.img -  before we did the append on the filesystem (doing ls -al):

          -rwxr-xr-x 1 root root 9414967296 May 3 11:19 rhel-5.3-HARMONY.img
          and after:
          -rwxr-xr-x 1 root root 10438967296 May 3 11:24 rhel-5.3-HARMONY.img

• Delete the temporary empty file - rm ZeroContentFile
• Resize the image file - resize2fs -f rhel-5.3-HARMONY.img
• Check with fsck the file - fsck.ext3 rhel-5.3-HARMONY.img
• Start the vm to see if the new disk has been added - xm create /etc/xen/rhel-5.3-HARMONY.cfg
• Check the disk space and you will see

root@localhost.localdomain[~]11:31:48# df -k
Filesystem 1K-blocks Used Available Use% Mounted on

/dev/xvda 10033864 3198928 6529108 33% /
tmpfs 1048576 0 1048576 0% /dev/shm


That's it you just increased your space by 1GB for the vm.
One thing to keep in mind is to do the backup of the original disk (just in case something goes wrong).

PHP - zend framework coding standars

Looking into the zend framework I noticed their coding standards at

http://framework.zend.com/manual/en/coding-standard.php-file-formatting.html#coding-standard.php-file-formatting.max-line-length

Far as I'm concern looks very good to me (except perhaps the auto loading as this can be done in different ways).

The autoload function can be then done like this:


function __autoload($class)
{
      $file = str_replace('_','/',substr($class,2)).'.php';    
      require_once(FR_BASE_PATH.'/includes/'.$file);
  }

or like where you can change the $parts and $path with more options


function __autoload($class)
{
  $parts = explode('_', $class);
  $path = implode(DIRECTORY_SEPARATOR, $parts);
  require_once (DIRECTORY_SEPARATOR .  $path . '.php');
}

NFS server setup (getport(nfs): request from unauthorized host| dump(): request from unauthorized host ) RedHat EL

I tried to make a simple nfs server on a redhat el 5.4 that uses /etc/hosts.allow and /etc/hosts.deny

I have the following nfs setup:

server

into /etc/hosts
10.0.0.2  client
10.0.0.1 server

 into /etc/exports
 /home/nfs-server client(rw,no_root_squash)

into /etc/hosts.deny

# wildcard that denies all
ALL:ALL

into /etc/hosts.allow

mountd: 10.0.0.2
statd: 10.0.0.2
portmap: 10.0.0.2
rquotad: 10.0.0.2

I start the portmap (service portmap start) on both machines and try to mount the server from client ... by my surprise there is no luck && try to do a rpcinfo -p  - it failed with - No remote programs registered
Looking into the server logs I can see

portmap[3058]: connect from x.x.x.x  to getport(nfs): request from unauthorized host

portmap[3061]: connect from x.x.x.x to dump(): request from unauthorized host


Doing different searches on the subject I came accross a bug redhat has on their site
https://bugzilla.redhat.com/show_bug.cgi?id=465412

So ... the solution seems to be the following - you need to add the ip address of the client AND the client host name into /etc/hosts.allow - even though portmap has the tcp_wrappers / libwrap compiled static and doesn't read directly the files /etc/hosts.allow|deny.

This is what I added on the server /etc/hosts.allow


mountd: client
statd: client
portmap: client
rquotad: client


After this - happy NFS.

Disable at runtime selinux

There are times when you need to test something quick and selinux is in your way ... what do you do then ?
Instead of going with a full reboot you can just do the following

echo 0 > /selinux/enforce

This will disable the selinux at runtime. If the system is configured with selinux enabled into his config file
(on redhat/centos /etc/sysconfig/selinux ) next time you reboot it will be enable.

To enable at runtime

echo 1 > /selinux/enforce

Lvm linux - how to create logical volumes

Steps to have lvm running on your system.

- find a partition or disk that is not used - I have /dev/sda10
- pvcreate /dev/sda10 (creates the physical volume)
- vgcreate VolGroup00 /dev/sda10 (creates a volume group)
- lvcreate -L 40GB -nTempVolume VolGroup00 (create a volume with 40GB with the name TempVolume under the volume group VolGroup00)
- mkfs.ext3 /dev/mapper/VolGroup00-TempVolume (creates the filesystem ext3 type on the TempVolume)
- mkdir /mnt/IMAGE (create a directory to mount the volume)
- mount /dev/mapper/VolGroup00-TempVolume /mnt/IMAGE
- mount (shows /dev/mapper/VolGroup00-TempVolume on /mnt/IMAGE type ext3 (rw))

That's it! You can start using the new mount point.

Rsync from linux mirrors without warning

Did you try to rsync and get lots of warnings because it can't change the owner or
the group ? This is caused by the fact that the rsync servers have a user id as owner
that doesn't exists on your system. Same goes for the group.

So if you want to have the verbose -v and no warnings you will need to use the
--no-o meaning no owner
--no-g meaning no group

This tells rsync on your server not to preserve the above

rsync -avz --no-g --no-o rsync://centos.mirror/centos/5/os/i386/ /my/repo