Is the FC1 2.4.22-1.2174.nptl kernel still having problems with SMP and hanging? I'm due to install some MP servers soon and want to avoid problems (imagine that!). If said kernel is still having trouble, what's the easiest fix? Thanks in advance.

Nigel Wade wrote: > Mitch Wiedemann wrote: > >> Is the FC1 2.4.22-1.2174.nptl kernel still having problems with SMP >> and hanging? >> I'm due to install some MP servers soon and want to avoid problems >> (imagine that!). >> >> If said kernel is still having trouble, what's the easiest fix? >> >> Thanks in advance. >> > > I'm currently investigating SMP here. When I tried stock FC1 and the > latest 2.4.22-1.2174.nptl I had severe problems with LDAP. I didn't > get any hard lockup, but I would have delays of between 4 and 300 secs > in LDAP response times. I have a strong suspicion that the delays are > related to FUTEX, but that's hard to prove. > > I found that disabling hyperthreading is the answer to my problems. > When I disabled it in the BIOS LDAP started behaving itself. > > I'm currently stress testing the LDAP server and it's managing ~40,000 > requests per minute with the peak search times of around 5ms. I read > in some other threads that mounting/unmounting NFS and Samba > filesystems lead to hard lockups so I'm mounting/listing/unmounting > one of each every half hour. The test will be left running for the > next 72 hours to see what happens... > > Compile a SMP kernel from vanilla kernel.org source and it works like a dream.. Even with HT enabled.. :) Later..

Nigel Wade wrote: > > My system crashed at 23:35. The cron job to mount/unmount filesystems > was set to run at :10 and :35 every hour. Given what other people > report in this and other threads I'd say either mount or unmount was > the culprit. > > I think I'll be building my own kernel now. > I built the latest 2.4 kernel, but LDAP won't run with that kernel (more precicely, the BDB backend won't start and slapd seg. faults). So that's a non-starter. I guess I'm back to square one. The system won't operate in SMP with the FC kernel and won't operate at all without a FC kernel. So it looks like Fedora is snafu'd. Unfortunately, RH9/RHEL3 don't have the version of LDAP I require, and when I tried an upgrade to openldap I started getting FUTEX locking problems.

I guess I'm back to square one. The system won't operate in SMP with the FC kernel and won't operate at all without a FC kernel. So it looks like Fedora is snafu'd. Unfortunately, RH9/RHEL3 don't have the version of LDAP I require, and when I tried an upgrade to openldap I started getting FUTEX locking problems.

Norman Gaywood wrote: > On Fri, Apr 02, 2004 at 11:31:06AM +0100, Nigel Wade wrote: > >> I guess I'm back to square one. The system won't operate in SMP with >> the FC kernel and won't operate at all without a FC kernel. So it >> looks like Fedora is snafu'd. >> >> Unfortunately, RH9/RHEL3 don't have the version of LDAP I require, >> and when I tried an upgrade to openldap I started getting FUTEX >> locking problems. > > > > Install the RHEL3 kernel on FC1. I installed: > > kernel-smp-2.4.21-9.0.1.EL > kernel-smp-unsupported-2.4.21-9.0.1.EL > > from mirror.physics.ncsu.edu:/pub/whitebox/3.0/en/updates/i686 > Thanks, that's an interesting suggestion. I really don't want to install RPMS from an unknown source, but I'll have a go at building my own.

On Tue, Apr 06, 2004 at 03:38:21PM +0100, Nigel Wade wrote: >Nigel Wade wrote: > >>Norman Gaywood wrote: >> >>>On Fri, Apr 02, 2004 at 11:31:06AM +0100, Nigel Wade wrote: >>> >>>>Unfortunately, RH9/RHEL3 don't have the version of LDAP I require, >>>>and when I tried an upgrade to openldap I started getting FUTEX >>>>locking problems. >>> >>>Install the RHEL3 kernel on FC1. I installed: >>> >>>kernel-smp-2.4.21-9.0.1.EL >>>kernel-smp-unsupported-2.4.21-9.0.1.EL > >Ok, the bottom line is that this doesn't work either. > >I've built the RHEL3 kernel on FC1 and run the system up using this. >Ordinary ldap requests seem quite happy, but I'm still seeing the futex >lock problem with nss_ldap. E.g. when I add 'hosts: files ldap dns' to >/etc/nsswitch.conf I get: > ># strace ping host >munmap(0xb75ff000, 4096) = 0 >uname({sys="Linux", node="hostname", ...}) = 0 >futex(0x53e4ec, FUTEX_WAIT, 2, NULL Interesting. I thought the EL kernel had the fast futex code back-ported but I have not checked this. So I thought the EL kernel would work like RH9 and FC1 kernels.

Could I ask what what steps you took to build the EL kernel on FC1? I didn't have much luck building with rpmbuild when I tried this.

>Setting LD_ASSUME_KERNEL=2.4.1 cures this particular problem, but isn't a >solution for a system relying on LDAP for all authentication and NSS >functions. Another suggestion for you to try is on one of the bugzillas. That is, to rebuild the FC1 kernel with low latency scheduling turned off. Maybe that's your best option. https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=109962 Comments #42 and #44

> Another suggestion for you to try is on one of the bugzillas. That is, > to rebuild the FC1 kernel with low latency scheduling turned off. Maybe > that's your best option. > > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=109962 > > Comments #42 and #44 This could be the cure for the SMP/umount problem.

I'm running the "hangemhi" script (below in this thread) which continually mount/umounts a file via the loop device - it's passed the 1/2 million mark and counting...

>>>>> "DJ" == Dave Jones <davej(a)redhat.com&gt; writes: DJ> For now I've just turned off the lowlat patch completely in DJ> CVS. Its been a while since there's been a FC1 kernel update, so DJ> there's quite a bit of stuff pending, so I'll try and get out an DJ> update in the next week or so. Thanks; I had assumed that you folks were working exclusively on 2.6 now but it's good to know that at least one more FC1 kernel update will be forthcoming. Folks who need this fixed now but who want to keep everything in RPM are welcome to pull the packages I built from http://www.math.uh.edu/~tibbs/kernel-packages or follow the instructions posted in bug 109962 and build your own. It's not difficult. - J<

>>>> "JD" == Jay Daniels <drs(a)pointyhats.com&gt; writes:

> I've built the RHEL3 kernel on FC1 and run the system up using this. > Ordinary ldap requests seem quite happy, but I'm still seeing the > futex lock problem with nss_ldap. E.g. when I add 'hosts: files ldap > dns' to /etc/nsswitch.conf I get: > > # strace ping host > munmap(0xb75ff000, 4096) = 0 > uname({sys="Linux", node="hostname", ...}) = 0 > futex(0x53e4ec, FUTEX_WAIT, 2, NULL > > Setting LD_ASSUME_KERNEL=2.4.1 cures this particular problem, but > isn't a solution for a system relying on LDAP for all authentication > and NSS functions. The problem remains in FC2 (Test 3) with the 2.6 kernel! LD_ASSUME_KERNEL=2.4.1 seems to cure at least "ping host"

Nigel Wade wrote: > Herbert Gasiorowski wrote: > >>> I've built the RHEL3 kernel on FC1 and run the system up using this. >>> Ordinary ldap requests seem quite happy, but I'm still seeing the >>> futex lock problem with nss_ldap. E.g. when I add 'hosts: files ldap >>> dns' to /etc/nsswitch.conf I get: >>> >>> # strace ping host >>> munmap(0xb75ff000, 4096) = 0 >>> uname({sys="Linux", node="hostname", ...}) = 0 >>> futex(0x53e4ec, FUTEX_WAIT, 2, NULL >>> >>> Setting LD_ASSUME_KERNEL=2.4.1 cures this particular problem, but >>> isn't a solution for a system relying on LDAP for all authentication >>> and NSS functions. >> >> >> >> >> The problem remains in FC2 (Test 3) with the 2.6 kernel! >> >> LD_ASSUME_KERNEL=2.4.1 seems to cure at least "ping host" >> >> > > This is in some way related to nscd. > > If nscd isn't running hosts don't resolve using LDAP. AFAIK, nscd > should not be compulsory. > Thanks, this seems to solve my problem and finally I managed to install nscd!

From "Re: Latest FC kernel still have SMP bugs?" Nigel Wade wrote: > Herbert Gasiorowski wrote: > >> Nigel Wade wrote: >> >>> Herbert Gasiorowski wrote: >>> >>>>> I've built the RHEL3 kernel on FC1 and run the system up using >>>>> this. Ordinary ldap requests seem quite happy, but I'm still >>>>> seeing the futex lock problem with nss_ldap. E.g. when I add >>>>> 'hosts: files ldap dns' to /etc/nsswitch.conf I get: >>>>> >>>>> # strace ping host >>>>> munmap(0xb75ff000, 4096) = 0 >>>>> uname({sys="Linux", node="hostname", ...}) = 0 >>>>> futex(0x53e4ec, FUTEX_WAIT, 2, NULL >>>>> >>>>> Setting LD_ASSUME_KERNEL=2.4.1 cures this particular problem, but >>>>> isn't a solution for a system relying on LDAP for all >>>>> authentication and NSS functions. >>>> >>>> >>>> >>>> >>>> >>>> >>>> The problem remains in FC2 (Test 3) with the 2.6 kernel! >>>> >>>> LD_ASSUME_KERNEL=2.4.1 seems to cure at least "ping host" >>>> >>>> >>> >>> This is in some way related to nscd. >>> >>> If nscd isn't running hosts don't resolve using LDAP. AFAIK, nscd >>> should not be compulsory. >>> >> >> Thanks, this seems to solve my problem and finally I managed to >> install nscd! >> > > I do this, but I have to say I don't like it. It's very precarious - > if nscd hangs or crashes for some reason the m/c is unusable. You > can't even login as root at the console to fix nscd if host resolution > is hanging. Eventually pretty much everything hangs. Actually it does not solve my problem: I need to use "hosts: files ldap dns" in nsswitch.conf, but then again ping never returns and nscd crashes. (It only works with "hosts: files dns ldap" and nscd)

> > I just added a reply in Bugzilla to your report of this. > > Here, I can get nscd to work with "host: files ldap dns", but I had to > restart it after I changed nsswitch.conf. > > Ok: I finally got it (hopefully): I missed that some libldap does a reverse lookup on the ip number in ldap.conf (or the like). So I need to add the ldap-server in /etc/hosts avoiding the endless loop (and nscd so ping will not stop in futex).

Is the FC1 2.4.22-1.2174.nptl kernel still having problems with SMP and hanging?

I'm due to install some MP servers soon and want to avoid problems (imagine that!). If said kernel is still having trouble, what's the easiest fix?

Is the FC1 2.4.22-1.2174.nptl kernel still having problems with SMP and hanging? I'm due to install some MP servers soon and want to avoid problems (imagine that!).

If said kernel is still having trouble, what's the easiest fix? Thanks in advance. -- Mitch Wiedemann mc^2 Computer Consulting mc2(a)lightlink.com http://www.lightlink.com/mc2

On Thu, 2004-04-01 at 09:48, Mitch Wiedemann wrote: > Is the FC1 2.4.22-1.2174.nptl kernel still having problems with SMP and > hanging? I run around 100 SMP servers with no problem. Hardware. Dual Xeon servers with HT on supermicro server boards. Dual AMD on Tyan 2469,2468,2462 Dual Opteron on Tyan 2880 And I do run the fedora kernel. Anything I can do to help let me know. But I have never seen any of the problems anyone is talking about.

37700 mount/unmount and still going. I will let it run on my test box here at my office. And give a update to my findings. But let me ask you what file system do you use on your boxs? On Thu, 2004-04-01 at 19:21, Norman Gaywood wrote: > On Thu, Apr 01, 2004 at 04:31:57PM -0500, Chris Miller wrote: > > On Thu, 2004-04-01 at 09:48, Mitch Wiedemann wrote: > > > Is the FC1 2.4.22-1.2174.nptl kernel still having problems with SMP and > > > hanging? > > > > I run around 100 SMP servers with no problem. > > > > Hardware. > > Dual Xeon servers with HT on supermicro server boards. > > Dual AMD on Tyan 2469,2468,2462 > > Dual Opteron on Tyan 2880 > > > > And I do run the fedora kernel. Anything I can do to help let me know. > > But I have never seen any of the problems anyone is talking about. > > I'd be interested if you could run the script from: > > http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=109962#c40 > > and see how you go. The script creates a small file in /tmp, makes a > filesystem image out of that file, and then continuously does a > loop-back mount/umount until you get sick of it and ^C out. > > Any FC1 SMP kernel I have used on several types of systems, including > AMD single processor systems running the athlon SMP kernel, will lockup > in less than 500 mount/umounts. Usually much faster. > > Other kernels can do mount/umounts for as long as you want. I've gone > upto 20,000 before I ^C out. > > This is the script: > > #!/bin/sh > > NAME="hangemhi" > SIZE=6000 > > MNTPOINT=/tmp/${NAME} > IMAGE=/tmp/${NAME}.img > > echo Creating something to mount > dd if=/dev/zero of=${IMAGE} bs=1k count=${SIZE} > mke2fs -F -q ${IMAGE} ${SIZE} > > echo Creating the mount point > mkdir -p ${MNTPOINT} > > echo Now doing mount/umount until the system hangs > COUNT=1 > while true > do > echo -en ${COUNT} "\r" > mount -r -o loop -t ext2 ${IMAGE} ${MNTPOINT} > umount ${MNTPOINT} > COUNT=`expr ${COUNT} + 1` > done > > > -- > Norman Gaywood, Systems Administrator > School of Mathematics, Statistics and Computer Science > University of New England, Armidale, NSW 2351, Australia > > norm(a)turing.une.edu.au Phone: +61 (0)2 6773 2412 > http://turing.une.edu.au/~norm Fax: +61 (0)2 6773 3312 > > Please avoid sending me Word or PowerPoint attachments. > See http://www.fsf.org/philosophy/no-word-attachments.html >

On Fri, 2 Apr 2004 10:21:23 +1000 Norman Gaywood <norm(a)turing.une.edu.au&gt; wrote: >On Thu, Apr 01, 2004 at 04:31:57PM -0500, Chris Miller wrote: > > >>On Thu, 2004-04-01 at 09:48, Mitch Wiedemann wrote: >> >> >>>Is the FC1 2.4.22-1.2174.nptl kernel still having problems with >>>SMP and hanging? >>> >>> >>I run around 100 SMP servers with no problem. >> >>Hardware. >>Dual Xeon servers with HT on supermicro server boards. >>Dual AMD on Tyan 2469,2468,2462 >>Dual Opteron on Tyan 2880 >> >>And I do run the fedora kernel. Anything I can do to help let me >>know. But I have never seen any of the problems anyone is talking >>about. >> >> >I'd be interested if you could run the script from: > >http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=109962#c40 > >and see how you go. The script creates a small file in /tmp, makes a >filesystem image out of that file, and then continuously does a >loop-back mount/umount until you get sick of it and ^C out. > >Any FC1 SMP kernel I have used on several types of systems, including >AMD single processor systems running the athlon SMP kernel, will >lockup in less than 500 mount/umounts. Usually much faster. > > > Our servers have ~10 partitions mounted, all local ext3 and have no problems with the SMP kernel. However the workstations (with hyperthreading switched on) automount some of their disks from the servers. They tend to lock up after a few days. So one workaround appears to be to use fixed nfs mounts at boot time. FWIW, running the script, the workstations would lock up between 1000 and 1400 mounts. Regards, Chris