Thursday, December 29, 2011

Globus Provision

I tested globus-provision which is an easy way to have a globus+condor cluster ready for number crunching on Amazon’s EC2 cloud.
My front end was my laptop which runs Windows. Python on Windows does not behave exactly as on Linux when it comes to SIGINT and therefore os.fork() generated an error message. Thanks to the support of Borja Sotomayor the SIGINT issues has resolved and the solution was to by-pass the SIGINT so I could proceed with the test. You can read more about this issue from here:
I tried the example from:
I decided not to use globusonline for the eBooks transfer and I used wget instead.
Below are a few screen captures that show the test.

My configuration file:
 Preparing the instance:

after a minute or two:

checking the instance:

Watching my cluster at the AWS management console from my browser:
Configuring Putty for SSH:

Connecting to one of the nodes:
Submitting a condor task (multiple jobs):

(unfortunately this specific test didn’t reached to the final destination due to an NFS failure of the /nfs/scratch/ mount point which I think is not directly related to this post…)

Monday, December 19, 2011

"Big Data" - The New Big Kid in Town

A year ago almost no one knew this term. With a doubling time of less than 6 months "Big Data" is becoming a hot hype, as can be seen below (by Google Trends)