starting problems with Pelican

[meckart]

Hello,

wie tried to setup a cluster with Pelican hpc as in the tutroial. The head is running fine, the nodes are booting over lan. The setup script is finding the nodes, so we are pushing "yes" - but then if we execute "lamnodes" it says: lamd is not running, exceute lamboot or something... so we start lamboot, the pelican_restart_hpc, search for nodes and the lamnodes shows the 10.11.12.* - so we are thinking: ready to run.

Now the problems: if we're running "kernel_example" we've got errors in octave. If we're running "pea_example" it runs the example, but only with 2 nodes (it says 2 nodes in ocatve and if we're watching the nodes we find only in one top "octave".

So, what are we doing wrong? We stepped through the tutorial several times.

At the end we want to try · RAxML-VI-HPC (version 2.2.3)
http://icwww.epfl.ch/~stamatak/index-Dateien/Page443.htm

So, we figured out to make the executing - but it is only running on the head...
If sombody wants to help us, feel free :)

Martin
University of Jena

[Michael Creel]

meckart wrote:

wie tried to setup a cluster with Pelican hpc as in the tutroial. The head is running fine, the nodes are booting over lan. The setup script is finding the nodes, so we are pushing "yes" - but then if we execute "lamnodes" it says: lamd is not running, exceute lamboot or something... so we start lamboot, the pelican_restart_hpc, search for nodes and the lamnodes shows the 10.11.12.* - so we are thinking: ready to run.

So, the setup process goes as it appears in the Tutorial or the screencast, and seems to complete normally. The setup process reports finding the compute nodes, as in this shot:


but when you finish the setup process and  call "lamnodes" is says that lamd is not running?  Is that an accurate summary of the problem?

What version of PelicanHPC are you using?

After you re-run pelican_restarthpc things seem to work correctly?

meckart wrote:

Now the problems: if we're running "kernel_example" we've got errors in octave. If we're running "pea_example" it runs the example, but only with 2 nodes (it says 2 nodes in ocatve and if we're watching the nodes we find only in one top "octave".

So, what are we doing wrong? We stepped through the tutorial several times.

kernel_example has a bug, thanks for the report. I'll fix that for the next release. (EDIT: actually, this is already fixed in the v1.99.0 release.). I think that pea_example is running fine. This is an indication that re-running pelican_restarthpc in fact worked. If 2 MPI ranks are used, it is normal that top only shows 1 running on the frontend, because the other is running on a compute node. To see it, ssh into the compute node and run top there (or set up a cluster monitor - see the homepage for a link how to do it).

meckart wrote:

At the end we want to try · RAxML-VI-HPC (version 2.2.3)
http://icwww.epfl.ch/~stamatak/index-Dateien/Page443.htm

So, we figured out to make the executing - but it is only running on the head...
If sombody wants to help us, feel free :)

sorry, I don't know anything about that.

[meckart]

Hello Michael, thank you for the fast answer...

Michael Creel wrote:

So, the setup process goes as it appears in the Tutorial or the screencast, and seems to complete normally. The setup process reports finding the compute nodes, as in this shot:
but when you finish the setup process and  call "lamnodes" is says that lamd is not running?  Is that an accurate summary of the problem?

That's correct. The screens are the same like in the turtorial and at the end the screen says: Found 8 nodes (we're running 8 at the moment)

Michael Creel wrote:

What version of PelicanHPC are you using?

From this mirror: http://download.mi.hs-heilbronn.de/pelicanhpc/ 
pelicanhpc-v1.9.1-32bit.iso  

Michael Creel wrote:

After you re-run pelican_restarthpc things seem to work correctly?

Interestingly it's "pelican_restart_hpc" (with underscore), but yes & no, I did the following:
1.) lamboot
2.) lamnodes shows only the host
3.) pelican_restart_hpc
4.) the host finds 8 nodes, but lamnodes has still only 1
5.) pelican_setup again
6.) the host finds 8 nodes, lamnodes shows all 8 IPs of the nodes
7.) startx and then I tried some examples

Michael Creel wrote:

kernel_example has a bug, thanks for the report. I'll fix that for the next release. (EDIT: actually, this is already fixed in the v1.99.0 release.). I think that pea_example is running fine. This is an indication that re-running pelican_restarthpc in fact worked. If 2 MPI ranks are used, it is normal that top only shows 1 running on the frontend, because the other is running on a compute node. To see it, ssh into the compute node and run top there (or set up a cluster monitor - see the homepage for a link how to do it).

Ok, here we go: v1.99 - We do have AMD Semprons 3000+ - and i'm not sure if they support 64bit - so maybe here settles the problem we do have? We tried running the v1.9.1-32bit.iso as I mentioned.

Michael Creel wrote:

sorry, I don't know anything about that.

No problem, but actually MPI Programs should be supported by Pelican?  I couldn't find any "howto" for MPI Porgrams to run it on al cluster... it's quite new for me... maybe you have a hint.

[Michael Creel]

Well, it seems that things worked after the restart. Now that I think about it, this is a circumstance that does occur occasionally - if you click yes while a compute node is just finishing booting, sometimes you see what has happened to you. The solution is to just re-run pelican_restart_hpc. So this is normal. The way to never see this problem is to wait a bit longer before clicking yes, so that the compute nodes all finish booting. Perhaps I could add a little sleep time to the script to avoid this.

In general, I don't recommend using v1.99.0, unless you need the newer Linux kernel, or unless you really want to use kernel regression or density smoothing in Octave. v1.9.1 has had a lot more testing. I believe that newer Semprons are 64 bit, so it should work, if you decide to try it. It depends on whether or not the CPUs are 64 bit.

PelicanHPC should let you use any MPI programs that work with LAM/MPI or OpenMPI. It's main purpose is precisely to provide an easy to set up platform for working with MPI. I think that the code you mention should work fine, but I have no experience with it.

Cheers, M.

[meckart]

Michael Creel wrote:

Well, it seems that things worked after the restart. Now that I think about it, this is a circumstance that does occur occasionally - if you click yes while a compute node is just finishing booting, sometimes you see what has happened to you. The solution is to just re-run pelican_restart_hpc. So this is normal. The way to never see this problem is to wait a bit longer before clicking yes, so that the compute nodes all finish booting. Perhaps I could add a little sleep time to the script to avoid this.

In general, I don't recommend using v1.99.0, unless you need the newer Linux kernel, or unless you really want to use kernel regression or density smoothing in Octave. v1.9.1 has had a lot more testing. I believe that newer Semprons are 64 bit, so it should work, if you decide to try it. It depends on whether or not the CPUs are 64 bit.

If 2 MPI ranks are used, it is normal that top only shows 1 running on the frontend, because the other is running on a compute node. To see it, ssh into the compute node and run top there (or set up a cluster monitor - see the homepage for a link how to do it)

Ok, I remember: somewhere it's written like "better use 1.9.1" - and actually I'm impressed how easy it was to setup a cluster with pelican. Now the only concern I do have is that not all nodes are computing. So I did what you told me and logged me into the nodes and run "top" - but only 1 node showed octave at the pea_example. I can ping and ssh the nodes. Interestingly sometimes if I start a example, the failban2server appears in top. Actually all examples should use all achievable nodes?

[Michael Creel]

No, the examples don't necessarily use all available nodes, the number is often hardcoded. With pea_example, you can change the number of nodes by calling "edit pea_example" from inside octave, and increasing the number of compute nodes in the line

nodes = 1; # number of compute nodes to use (0 for serial)

If you increase that number, then you should see activity on more nodes.

Cheers, M.

[meckart]

Thank you, I really appreciate your work and support. I'll try this and give you a feedback. We only tried to edit the kernel_example.

[meckart]

So, I edited the pea example to the count of nodes we do have and it's working. All nodes run a command "octave". funnily enough the cpu load of the nodes just lays between 25 - 30% - only node 1 has a cpu load over 90% - why come so? And I thought, the calculation is faster, like if you had 10 seconds for 60 iterations with 2 nodes, it's only 5 or someting with 8 nodes... but it isn't.

[Michael Creel]

OK, I'm glad it's working. I guess that the node that has CPU usage at 90% has a slower CPU than those of the other nodes. They are mostly idle because they are waiting for the slow node to finish. The speedup you get depends on the relative speeds of the nodes, and on the latency and bandwidth of the networking that connects them. The results you obtain depend very much on the details of the cluster. Doing work on a single fast computer will often be faster than using a 2 node cluster made of the fast computer plus a slow node, unless you have load balancing built into the code. This is because if you just give the slow computer half of the computations, the power of the fast computer will not be fully used.

The pea_example code is fairly "tightly coupled" which means that it is sensitive to the latency and bandwidth of the network.

I have a paper out that discusses this example, at
http://www.springerlink.com/content/b2l43gg723781m87/

There are also some Monte Carlo examples using Octave that are not so sensitive to the networking hardware. Try running mc_example1 and mc_example2 while in Octave.

[meckart]

We're hosting 9 similar machines, it was a gift from a offical company, all have the same mothoerboard and CPU and RAM. The montecarlo example 2 is nice... I think, maxslaves is the number of maximum nodes that should be used? So if I increase the number of reps to 10,000 - the time is going down for the first 2 nodes and then it stucks there, later the time increases also, like:

0 node - 65 sec
1 node - 45 sec
2 node - 25 sec
3 node - 25 sec
...
8 node - 26 sec

So this is a problem of our slow ethernet? The mobos have only a fast and the switch is a 100mbps full duplex switch. I read in the forum someting about benchmarking, but if I'm honest, I didn't understand how to optimize it. I'm now a little confused if what I'm trying is actually working - to compute my phyolgeny stuff over night instead of 1 week...

[Michael Creel]

Your cluster should do better than that for Monte Carlo. Try the code below. This uses a large number of reps, so that the compute nodes are busy for a little while. Also, the nodes do 1000 replications at a time, rather than 100, to lower network effects. Finally, and importantly, I moved "toc;" to avoid counting the time used to make the bar graph (stupid bug!).  I think this last bug is what caused the problem.




########### cut here ###############

# this runs the test on several different configurations and reports timings.
# to see good speedups in parallel, you need to set "reps" to a fairly
# big number (say, 100000 or so).
outfile = "example2.out";
T = 1000;
dim = 5;
reps = 100000;

maxslaves = 8;
timings = zeros(maxslaves,1);
for i = 0:maxslaves;
# figure;
        tic;
        n_received = montecarlo("tracetest", {T,dim}, reps, outfile, i, 1000);
        t = toc;
        if i > 0
                bar(0:i, n_received/reps);
                xlabel("node");
                grid on;
                title("percentage of results received by node");
                drawnow();
        endif

        timings(i+1,:) = t;
endfor

for i = 0:maxslaves;
        printf("\n\nTiming results, second example\n");
        printf("Number of nodes: %d     Time (sec.): %f\n", i, timings(i+1,:));
endfor

[meckart]

you,re right...

Timing results, second example
Number of nodes: 0     Time (sec.): 626.631167
Timing results, second example
Number of nodes: 1     Time (sec.): 311.761412
Timing results, second example
Number of nodes: 2     Time (sec.): 171.927231
Timing results, second example
Number of nodes: 3     Time (sec.): 113.922598
Timing results, second example
Number of nodes: 4     Time (sec.): 92.087825
Timing results, second example
Number of nodes: 5     Time (sec.): 73.063092
Timing results, second example
Number of nodes: 6     Time (sec.): 73.621497
Timing results, second example
Number of nodes: 7     Time (sec.): 54.708440
Timing results, second example
Number of nodes: 8     Time (sec.): 54.226124
octave:34>  

It,s working great... thank you again for your help. PelicanHPC is a really nice linux distro to learn something about clusters. Now it,s working i,m going to read a little bit about MPI and try to start RAxML on the cluster.