El Mon, 2 Jan 2012 12:55:09 -0600
Dennis Gilmore <dennis(a)ausil.us> escribió:
El Sat, 24 Dec 2011 23:25:56 -0600
Dennis Gilmore <dennis(a)ausil.us> escribió:
> When we were testing build were happening really fast, once we
> loaded up the build jobs things have become really slow
>
>
http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=238672
> started at 1:19 utc and at 5:18 utc four hours later the
> buildrequires are still being installed. australia is completely io
> bound. i think we need to see how we can spread the io load. ~30
> hosts reading and writing to the 4 spindles just saturates the disk
> io. i guess options are find a way to add more spindles.
> move /var/lib/mock to sdcard, see if we can get some kind of san
> that has alot of smaller fast disks. get some 2.5" usb drives one
> for each builder. some other idea? is there anyway we could add
> 4-8 more disks to australia. the size of the matters little.
> gaining more iops by adding more spindles would help.
>
> Just throwing out what im seeing. lets see what ideas we can come up
> with. performance is better than before. and seneca has done a
> great job and put a lot of hard work into the reorg and im thankful
> for that. We just have another bottleneck to address.
> _______________________________________________
> arm mailing list
> arm(a)lists.fedoraproject.org
>
https://admin.fedoraproject.org/mailman/listinfo/arm
http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=245410 has been
running 10 hours now. its inited the chroot and installing the srpm.
it is a big srpm but still. performance on the hfp builders while not
great is better. when looking at a sfp builder the mock process is
using 100% cpu and in a D state. there is one of two issues here. the
hfp builders are running a slightly older kernel. so it could be a
kernel ext3 nfs or loop device issue, upgrading a hfp builder for
testing would confirm it. or its some issue with decompresion of the
rpms. I do think that disk io on the nfs server is partly to blame.
the box sits at a load of 9 all the time with 30% or so io wait. Id
like to take a layer or 2 of the complexity out. switching over to
nfsv4 we should be able to run mock on it. im going to do some
compression tests with the latest 2.6.41.6 kernel.
further followup ive inited chroots on my pandaboard. ive used 2
sdcards both are 32gb transcend class 10 cards both with the same mock
configs with caching disabled. like koji.
[dennis@fedora-armhfp ~]$ time mock -r fedora-15-armhfp --init
INFO: mock.py version 1.1.18 starting...
State Changed: init plugins
INFO: selinux enabled
State Changed: start
State Changed: lock buildroot
State Changed: clean
INFO: chroot (/var/lib/mock/fedora-15-arm) unlocked and deleted
State Changed: unlock buildroot
State Changed: init
State Changed: lock buildroot
Mock Version: 1.1.18
INFO: Mock Version: 1.1.18
INFO: calling preinit hooks
State Changed: running yum
State Changed: unlock buildroot
INFO: Installed packages:
State Changed: end
real 8m14.626s
user 2m8.930s
sys 0m21.234s
[dennis@panda01-sfp ~]$ time mock -r fedora-15-arm --init
INFO: mock.py version 1.1.18 starting...
State Changed: init plugins
INFO: selinux enabled
State Changed: start
State Changed: lock buildroot
State Changed: clean
INFO: chroot (/var/lib/mock/fedora-15-arm) unlocked and deleted
State Changed: unlock buildroot
State Changed: init
State Changed: lock buildroot
Mock Version: 1.1.18
INFO: Mock Version: 1.1.18
INFO: calling preinit hooks
State Changed: running yum
State Changed: unlock buildroot
INFO: Installed packages:
State Changed: end
real 10m37.586s
user 2m13.969s
sys 0m20.406s
so i think its pretty safe to assume that compression is working ok. so
it comes down to ext3, loopback device, kernel, or something else.
Dennis