Monday, February 20, 2012

Shell parallel processing

This is the description of the tool from the gnu site - Parallel
GNU parallel is a shell tool for executing jobs in parallel using one
or more computers. A job it can be a single command or a small script
that has to be run for each of the lines in the input. The typical
input is a list of files, a list of hosts, a list of users, a list of
URLs, or a list of tables. A job can also be a command that reads from
a pipe. GNU parallel can then split the input into blocks and pipe a
lock into each command in parallel.

The tool can do many things and has some very useful tools that come with it
see sql and niceload.
Bellow you can see some example on how to use it.

#!/bin/sh

# tail log files on different computers
# create a hosts file with all the computers you want to connect

echo '10.100.218.79' >> host.file
echo '107.22.24.219' >> host.file
cat host.file | parallel ssh {} "tail /var/log/php-fpm/error.log | awk '{print \$1,\$2,\$3,\$4,\$5,\$6}'"

[20-Feb-2012 15:05:05.323178] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:06.324028] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:07.324877] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:08.325727] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:09.326568] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:10.327418] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:11.328265] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:12.329118] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:13.329960] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),
[20-Feb-2012 15:05:14.330806] DEBUG: pid 7812, fpm_pctl_perform_idle_server_maintenance(),

GNU's site has lots more example see Examples

2 comments:

DanaLapid said...

I still didn't quite understand what's the shell parallel processing? Is it a php code generator?

Jack Dowson said...

Fintech improvement has detonated throughout the last ten years. Its effect on business processes is enormous and it made an extraordinary impact on different circles of our regular day to day existence. It influences a wide range of organizations, from new businesses and little firms to worldwide undertakings. Due to fintech, organizations and clients gain admittance to an immense measure of arrangements that assistance to deal with their funds. These days a great deal of fintech combinations are utilized in state-of-the-art applications for producing speculation guidance, crowdfunding, versatile installments and so on. Scaling becomes more straightforward and less expensive. Fintech permits business visionaries to get subsidizing without searching for financial speculators and financial backers. Attempts to close the deal are presently made effectively on the web and can arrive at potential banks everywhere. Fintech firms help to transmit monetary resources across borders, this cycle has had a couple of boundaries previously. Fintech applications and arrangements are created by and for 3a number of organizations gaining practical experience in different circles. Fintech programming has become profoundly practical as well as easy to use because of endeavors of each fintech organization front end designer>> best sites to hire fintech developers