On 04/04/2014 09:18 AM, Veli-Pekka Kestilä wrote:
On 3.4.2014 23:41, Adrian Sevcenco wrote:
> On 04/02/2014 09:54 PM, Bill Oliver wrote:
>> Just to see if I can do it, I thought I'd set up a small grid/cluster
>> using fedora. Does anybody know of a good step-by-step guide for this
>> (preferably free and online :-) )?
> As for the answer to your initial inquiry i would highly recommend ROCKS
> Clusters that is based on CENTOS. it will automatically install and
> manage your nodes from a single point (the FrontEnd server)(with NFS
> shared home).
>
I have tried ROCKS before (granted it was maybe 6 years ago) and it
seems good if you have very lage number of nodes. If not then I would
recommend setting the things yourself. Most programs run in scientific
that would
sum to: set up common authentication, common logging,
monitoring, nfs shared home (optional but very useful), the resource
manager and individual management of settings and packages of each
worker node. But this is up to the admin preferences (i use a personal
cluster with 4 nodes (32 cores) and from my point of view it saves me a
lot of time instead administering each node by hand) (it takes 10
minutes to re-install all nodes with different settings and new updated
packages) (for the 40 nodes grid cluster it takes the same 10 minutes ;) )
clusters use MPI for shared communication so you will just need MPI
i dont know what is meaning of "most" .. in my world of high energy
physics we use only distributed computing (so no MPI)
setup I would not waste time job management and such as long as there
is
only one person running on the cluster.
you might (i am sure you will) be hit by a
lots of problems .. mpi does
not have a knowledge of nodes and their capacities nor about their
availability and status. also if you need to start multiple parallel
jobs (eg. you have a 32 cores cluster. a job (on all 32 cores) will
take 16 hours to complete. if you launch in the morning at 8 am, the job
will will end at 0:00 so until the next morning the cluster will just
sit idle) you will need a resource manager.
Setting up maybe 10 to 20 machines with kickstart using fedora with
openmpi, ssh-keys for easy access to nodes and nfs sharing form frontend
shouldn't be that big job. You will take same amount time learning ROCKS
and it will have lots of features you don't need.
this is dependent of the
previous experiences .. i use rocks without
problems .. but i am using it for almost 10 years... YMMV
Also if you don't use the machines all the time and don't
have the
hardware yet it might be beneficial to use AWS nodes to do the
calculations. In big HPC setups the real cost of the cluster usually
comes from electricity consumption and cooling so you should factor
these costs in when deciding what to do.
I cannot agree more! the advent of cheap
IaaS clouds brings a lot of
useful opportunities :)
Adrian