Node boot problem

[TheSin]

First let me start with my setup and the mods I had to do to get this far.

I'm running a Dell M1000e with currently 2 M600 blades with dual quad core 3.16Ghz procs and 4gigs of ram each.  Since it's a blade server lots of the components are virtual or assigned.  Like KB, Mouse, I don't have any HDDS at all my plan is to connect a SAN or NAS to the master once I'm ready.  The Nics are BNX2 which needs the firmware-bnx2 pkg to work.  Adding this to the master was fairly easy, I first added it to the list of pkgs, but since by default live-helper only access the main branch and not the contrib or non-free and the firmware pkg is non-free I had to figure out how to add those, which I find out was adding.

<lh_config --categories "main contrib non-free">

after <lh_config -a "$ARCHITECTURE"> in both the master and the node setup in the make script.

That then got the master booting and I could run setup_pelican, once that was done I was able to test boot the node, it found the PXE and booted, but was missing the bnx2 firmware, this took me much longer to figure out, but I added a second list of pkgs to include for nodes and got that solved.  I used the same form as for the master so it's configurable at the top and I'd like to contribute these back if anyone wants.

Now the node finds the nic and starts to load, but it stops and I let it sit over night, it's not locked up ctrl-alt-del still works but it won't fulling boot either.  I have a screen shot of where it's sitting which I'll include if I figure out how but here is the text it's sitting at just incase I can't attach.

Driver 'sd' needs updating - please use bus_type methods
sd 1:0:0:0: [sda] Attached SCSI removable disk
Driver 'sr' needs updating - please use bus_type methods
sr0: scsi-1 drive
Uniform CD-ROM driver Revision: 3.20
sd 1:0:0:0: Attached scsi generic sg0 type 0
sr 0:0:0:0: Attached scsi generic sg1 type 5

Remember I have no CD-Rom, no HDD, no USB drive, no floppies, nothing but CPUs and Ram on these blades, the master is booting from a USB CD-Rom.

So my question is since the master and the node are 100% the same hardware why would one fully boot and the other won't where can I debug or find the differences in the boot process?  I really need to get the cluster up ASAP so any info or leads would help very much, Once i have this working I'm going to work on auto launching setup_pelican on the master and have nodes auto join and I"ll be sending that back upstream, but I need a node first ;)

p_49.jpg

[Michael Creel]

This is interesting, and I hope we can sort it out. Your modifications sound interesting too. Problems with booting can often be solved with cheatcodes to supply kernel boot parameters, such as noapic or noacpi. At the boot prompt, type "live" followed by a list of cheatcodes, for example "live noapic keyb=es". However, it may be the case that there is a problem between the kernel and the hardware. How to find the right cheatcodes is simply a matter of experimentation, in my experience. What I don't understand is why the frontend node will boot but not the compute node.

Another thing to try would be to set the distribution to "sid" instead of "lenny". This will only help if the kernel is more recent in Sid, which I haven't checked recently.

As I noted in a recent thread, booting PelicanHPC on the master node is the same as booting any Debian Live image, so asking for help on the Debian Live mail list will put you in contact with a larger pool of people, some of whom are a lot more knowledgable than I am. Solving boot problems on PelicanHPC should be useful for Debian Live in general.

[TheSin]

I tried a few cheats, but since I didn't need any on the master I find this odd, I tried nousb but that didn't help it still loaded all the usb.

I just rewrote the make_pelican script I wanted to separate all the file creations in it so I could get a more clear picture, in doing this I noticed one major flaw I think.  in the node there is a script 99script which makes and install rc.local, in that file is a call to pelican_boot_setup, but that file isn't installed on the node it's only on the master.  And since the file does the mount of the master files it's a circular depend (this is of course if I'm understanding how the PXE boot works which I have only played with for 2 days now).  So I'm currently making a new iso with installs that script on the nodes so it can run, I doubt that is the problem as I'd expect a bash file not found type error but figured it was worth a shot. If it works and my new build system works I'll post it here so you guys can see it and it might make contrib a little easier as I separated the scripts and configs out and it makes adding more and modifying them MUCH easier, I also added a ramdisk size variable since I saw a thread on that.

Anyhow wish me luck :D

[Michael Creel]

I agree that separating out configuration script from the make_pelican script makes it easier to understand, and that's the way I did things initially. The reason it's all lumped together in a script is to make it easy for a new user to make a new image with a different package list and no other changes. This way of arranging things flattens the learning curve at the beginning, but makes the hill go on longer.

About how the compute nodes get their filesystems - the nodes access the same squashfs file as does the frontend node, over NFS. So exactly the same software as is on the frontend node is available to the nodes. For this reason, pelican_boot_setup is available to the compute nodes. This script mounts  /home on the compute nodes, but /etc is already there before the script runs, so there is no problem.

Adding a ramdisk size variable seems like a good idea. It can be done with a cheatcode, but adding it explicitly to the script would be nice, too.

I don't understand why your frontend boots but not the compute nodes, if the hardware is the same. I have booted a frontend using a firewire CD drive, and have then gone on to boot compute nodes without trouble. The people on the Debian Live list who know more about the details of kernels, modules, and the possible differences between CD boot and netboot are likely to be of help here.

Good luck!

[TheSin]

I can totally understand the ease of use, but if a user is looking to build a cluster lets hope they know how to use tar ;)

Anyhow I almost have it done, I have the first 3 parts of pelican_setup auto work and I have it auto injecting custom network file so everything can be setup during boot.  It's just the interaction of hpc restart that I need to figure out I was thinking of just auto launching it in an xterm on the main system but I would like for nodes to auto connect, but I still have very little grasp on how it works since I still can't get a node to boot, it works 100% when I use vitalization to test it though master and all nodes connect.

Anyhow is there a place you'd like me to send what I have as a contrib back?  since this is going to be major part of our systems here I'll have lots of contributions to make ;)  but I want it to be able to boot and run un attended or if I add a new blade it should just auto join IMHO.  That is the major reason I went to a cluster so I could add power and not have down time.

But obviously I need to learn more about how the lam stuff actually works before I can figure that one out ;)

[TheSin]

I just wanted to post these 2 screen shots, this is the Master booting up and you can see it looks very close to the same (Except is has one extra scsi device which is the CDROM it's booting from that the node doesn't have) but it makes it past it and you can see what's next, thought it might help someone else help me solve this issue.
4292191662.jpg
2825066859.jpg

[Michael Creel]

TheSin wrote:

I can totally understand the ease of use, but if a user is looking to build a cluster lets hope they know how to use tar ;)

Right, there are definitely tradeoffs. For me to maintain it, the present system works well, but I agree that a more modular system would be better for advanced use and flexibility. But if you go very far in that direction, you'll end up with  Rocks :-)

TheSin wrote:

Anyhow I almost have it done, I have the first 3 parts of pelican_setup auto work and I have it auto injecting custom network file so everything can be setup during boot.  It's just the interaction of hpc restart that I need to figure out I was thinking of just auto launching it in an xterm on the main system but I would like for nodes to auto connect, but I still have very little grasp on how it works since I still can't get a node to boot, it works 100% when I use vitalization to test it though master and all nodes connect.

Having a system that will configure itself on boot is nice, but I think it will always require some customization to fit in with the local network, hardware, etc.  As long as the frontend can be accessed over the net (2 interfaces, one set to use DHCP, the other for the cluster), and the compute nodes can be made to reboot over the net, you're set. I guess your hardware satisfies these requirements.

TheSin wrote:

Anyhow is there a place you'd like me to send what I have as a contrib back?  since this is going to be major part of our systems here I'll have lots of contributions to make ;)  but I want it to be able to boot and run un attended or if I add a new blade it should just auto join IMHO.  That is the major reason I went to a cluster so I could add power and not have down time.

But obviously I need to learn more about how the lam stuff actually works before I can figure that one out ;)

Basically, the PelicanHPC download page hangs off my account on a server at my university, and I don't want to give out access to that. I would be happy to mirror stuff, though, and add links to your stuff on the PelicanHPC home page. Or if you send me a tar.gz of a directory to uncompress in the download area, with documentation, that works for me. I'll update it any time you send me a new version. I'm open to suggestions about how to do it.

About new blades auto-joining, getting them to boot is no problem. Getting them to participate in calculations that are ongoing _is_ a problem, at least as far as I know. The current way to do things is to run the restart_hpc script when all calculations have finished. Otherwise, they get interrupted, if you're using LAM. If you use OpenMPI, you have more flexibility. I use LAM in my day-to-day work, so I'm a little rusty regarding OpenMPI.

I'm looking forward to seeing your mods, and I hope you get the node to boot.
M.

[Michael Creel]

I don't see the compute node loading the aufs and squashfs modules. Those are needed to access the filesystem.squashfs file, which contains the root filesystem. Can you watch carefully when the compute node is trying to boot as see if those modules are loaded? I suspect that the compute node is not loading the correct net card module, and everything grinds to a halt as a result. Are you sure the networking card is getting support from the kernel?

[TheSin]

It's not that is where it's hanging, in virtualization it is so I think i agree with your guess, the nics are bnx2 which require a firmware to boot, I have added that to the master and the node images, that is how I started with all the script changes so I could figure out where I had to add it and since it is none free that is even harder, but it is there when I mount the pxelinux file.  Also prior to that I would get a nic error.  But I wonder if there is more that I am missing.  Perhaps I should check the kernel modules on the pxelinux.

Also as a side note in my virtualization tests I noticed one thing you same the nodes wouldn't share the master's /etc but it does so dhcp starts on the nodes too which could be a bad thing IMHO.  I'll look into this once my node actually boots ;)

[Michael Creel]

The nodes get their /etc from the same squashfs file as the master, but the contents once things are working are not the same. aufs allows files to be changed. When any node boots, dhcp does not run, in fact you'll see an error when it tries to start but fails. Then on the frontend, when pelican_setup is run, dhcp  is started. This allows the compute nodes to boot, and also is a reason to isolate your cluster from the rest of the world, so as not to conflict with other servers.

aufs is a very neat tool that allows the same compressed filesystem to be used for the frontend and the nodes.

On Monday I'll put an image I've made with the net module you need in the download area, as a beta version. Maybe that will work.

Cheers, M.

[TheSin]

Ahhh so that is the problem since I"m changing files in the squashfs, I should changing them during the boot and restart dhcpd then, okay I'll change that then.

And thanks I'm anxious to try it.

[Michael Creel]

There is an image called "testing.iso" in the download area. This has the module for your net card. Please give it a try and let me know what happens. Cheers, M.

[TheSin]

Nope same thing, BTW I'm intel not AMD, it still boots exactly the same though.  I'm almost done messing with my scripts I'll send you a copy in a bit, but the node still stalls at the same place as with mine.

[TheSin]

Okay I have an iso that I'd like you to test for me please everything should auto start now on boot, I can't test the nodes though, if I put it up on the web can you download it and test it for me?  If successful I'll send my build dir.  Now back to trying to get my nodes to boot now that I have a much much better grasp of the boot process.

[TheSin]

Also i finally got some info, this is from the frontend while the node tries to boot, it gets net then it seems to drop it or maybe the frontend has a bad route?

Jan 13 09:00:25 pelican dhcpd: DHCPDISCOVER from 00:22:19:94:91:7c via eth1
Jan 13 09:00:26 pelican dhcpd: DHCPOFFER on 10.11.12.2 to 00:22:19:94:91:7c via eth1
Jan 13 09:00:30 pelican dhcpd: DHCPREQUEST for 10.11.12.2 (10.11.12.1) from 00:22:19:94:91:7c via eth1
Jan 13 09:00:30 pelican dhcpd: DHCPACK on 10.11.12.2 to 00:22:19:94:91:7c via eth1
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.0 to 10.11.12.2:2070
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.0 to 10.11.12.2:2071
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.cfg/44454c4c-3000-1046-8032-b7c04f544831 to 10.11.12.2:57089
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.cfg/01-00-22-19-94-91-7c to 10.11.12.2:57090
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.cfg/0A0B0C02 to 10.11.12.2:57091
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.cfg/0A0B0C0 to 10.11.12.2:57092
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.cfg/0A0B0C to 10.11.12.2:57093
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.cfg/0A0B0 to 10.11.12.2:57094
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.cfg/0A0B to 10.11.12.2:57095
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.cfg/0A0 to 10.11.12.2:57096
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.cfg/0A to 10.11.12.2:57097
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.cfg/0 to 10.11.12.2:57098
Jan 13 09:00:30 pelican atftpd[4466]: Serving pxelinux.cfg/default to 10.11.12.2:57099
Jan 13 09:00:30 pelican atftpd[4466]: Serving debian-live/i386/boot-screens/header.cfg to 10.11.12.2:57100
Jan 13 09:00:30 pelican atftpd[4466]: Serving debian-live/i386/boot-screens/prompt.cfg to 10.11.12.2:57101
Jan 13 09:00:30 pelican atftpd[4466]: Serving debian-live/i386/boot-screens/boot.txt to 10.11.12.2:57102
Jan 13 09:00:30 pelican atftpd[4466]: Serving debian-live/i386/boot-screens/data/splash.rle to 10.11.12.2:57103
Jan 13 09:00:30 pelican atftpd[4466]: Serving debian-live/i386/boot-screens/menu.cfg to 10.11.12.2:57104
Jan 13 09:00:30 pelican atftpd[4466]: Serving debian-live/i386/boot-screens/help.cfg to 10.11.12.2:57105
Jan 13 09:00:30 pelican atftpd[4466]: Serving debian-live/i386/boot-screens/menu.cfg to 10.11.12.2:57106
Jan 13 09:00:30 pelican atftpd[4466]: Serving debian-live/i386/boot-screens/footer.cfg to 10.11.12.2:57107
Jan 13 09:00:31 pelican atftpd[4466]: Serving debian-live/i386/vmlinuz-2.6.26-1-686 to 10.11.12.2:57108
Jan 13 09:00:32 pelican atftpd[4466]: Serving debian-live/i386/initrd.img-2.6.26-1-686 to 10.11.12.2:57109
Jan 13 09:00:35 pelican atftpd[4466]: timeout: retrying...
Jan 13 09:00:40 pelican atftpd[4466]: timeout: retrying...
Jan 13 09:00:45 pelican atftpd[4466]: timeout: retrying...
Jan 13 09:00:50 pelican atftpd[4466]: timeout: retrying...
Jan 13 09:00:55 pelican atftpd[4466]: timeout: retrying...

[TheSin]

Okay I finally got it!  It HAS to use Eth0, where I wanted my pelican network to be on Eth1

Now is there a way I can change that, I noticed it cause I saw IP-Config: line come up before that but it was trying to use eth0 for it.

[Michael Creel]

I guess you will be testing now that you have the node booting? I'd be happy to test it too, though. Why don't you email me (michael.creel AT uab.es) with the details?

[Michael Creel]

OK, with that your node is now booting? I haven't tested using eth1 or anything other than eth0 for a while. At least that should be easy to fix if its broken. I'll check it here.

OK, back again. I tested (using virtualization) a 2 NIC frontend node, with the cluster on eth1 and NAT on eth0 to connect to the internet. It worked fine. For the compute node, it seems necessary to use eth0. If a compute node has more than 1 NIC, then it should be possible to connect to the cluster using eth0, since in principle the other NIC(s) will not be used for anything (at least, I can't think of why they would be used).

Was your compute node using eth1, or the frontend? Everything working now?

[TheSin]

sorry for the length in response I'm dealing with a some personal problems ATM, I hope to get back to this soon.

The Master and Ndoes both have 2 Nics, the Master is totally happy using ETH1 for the pelican network, Because of how the M1000e chassis is made i wanted to use eth1 on all blades for the pelican network, but it's not a huge deal I'm jsut glad to have it working.  The funny part is that the PXE boot works on eth1 untill it tries to mount the squashfs, I found that odd but I'll deal with looking deeper into that later on.

Right now I'm just adding a few more options to the config like using Xen now.

I hope to know more about the personal issues today, so I hope by tomorrow I can be full steam on this again at wish point I'd send you my setup fils and a copy of my ISO, BTW I also added the ability to make a USB-HDD which was way way more convenient and faster for me.  Once the img file is done you just do

dd if=<pelican img file> of=<usb root dev, ie: /dev/sda>

Works great and much faster boots from it as I'm using USB2 key.

[Michael Creel]

Great! I'm glad it's working for you. This issue about multiple NICs on compute nodes has never come up before. I have a cluster where the compute nodes have 2 NICs, but either I used eth0 purely by chance, or the problem is not general. For the moment, I'll make a note in the documentation recommending use of eth0 on the nodes.

Your mods sound very interesting, especially use of Xen. I'll be looking forward to trying it out.

What kind of work will you be doing? Any recommendations about packages to have installed by default?

Cheers, M.