2:19 p.m.
Hi,
Paulo and me (kim0) have been working on testing a caching setup for the
wiki. A test migration to Moin 1.5 is complete, squid is now configured as a
reverse caching proxy. We've done some stress tests on the current setup.
Attached are some extracted results that (I think) are of interest. Mainly
the number of requests served per second, and the average time for serving a
request. Also, paulo pointed that caching differs per file type, the tests
have been done on three different file types (html, png image, and css)
Test Setup:
=========
1- All connections were initiated from proxy1
2- Proxy2 had squid caching turned on
3- Testing for html/png/css done, sweeping the number of concurrent
connections
4- Turn off squid caching on proxy2
5- Testing for html/png/css done, sweeping the number of concurrent
connections again
Interesting notes:
============
1- Serving PNG is 10X faster than html
2- Serving CSS is 10X faster than PNG!
3- Serving html is really the bottleneck. Unfortunately, Moin developers
acknowledged current version (1.5) is not cache friendly. Work for making
1.6 cache friendly is undergoing
4- Using squid currently only seems to double our PNG serving rate, nothing
else
5- The application server hits swapping (about 0.5GB) at full load (~300
concurrent connections), for some reason the requests/second served is still
high!! (Is our cache disks that fast)
6- The test did not stress the server bad enough to run out of swap space,
not sure if this is needed though!
I can send the full results date if anyone is interested.
Best Regards
3:49 p.m.
Hi all,
The PNG link i gave kim0 was just an example of an image. Currently we are
caching all general type of images (gif, jpg, png).
As Ahmed pointed, current version of Moin isn't still html friendly,
hopefully this will be addressed at Moin 1.6.x (our current version is Moin
1.3.4 and our testing was based on Moin 1.5.6).
Moin 1.5.6, still bring alot of improvements, so in my opinion, even though
that Moin html is not cache friendly we should still be thinking in the
migration. Also as a workaround, we can still create a static website that
with the help of mod_headers, it could be forced to be cached.
Last but not least, let's not forget that this was a local test. A more real
stress test, is still going to be prepared and should include a server
requesting to the future architecture (including load balancer and both
proxies), this will improve even more our ability to deliver the desired
content.
We would like some coments on this tests (past and future), so that we can
improve what needs to be improved. As a final note, i would like all of you
to start testing the future platform[1]. Just play with Moin, and give some
feedback on the WikiMigration page[2].
[1] http://webtest.fedora.redhat.com/wiki
[2] http://fedoraproject.org/wiki/Infrastructure/WikiMigration
Once again thanks all for the help and feedback...
Paulo
On 12/23/06, Ahmed Kamal <email.ahmedkamal(a)googlemail.com> wrote:
Hi,
Paulo and me (kim0) have been working on testing a caching setup for the
wiki. A test migration to Moin 1.5 is complete, squid is now configured as
a reverse caching proxy. We've done some stress tests on the current setup.
Attached are some extracted results that (I think) are of interest. Mainly
the number of requests served per second, and the average time for serving a
request. Also, paulo pointed that caching differs per file type, the tests
have been done on three different file types (html, png image, and css)
Test Setup:
=========
1- All connections were initiated from proxy1
2- Proxy2 had squid caching turned on
3- Testing for html/png/css done, sweeping the number of concurrent
connections
4- Turn off squid caching on proxy2
5- Testing for html/png/css done, sweeping the number of concurrent
connections again
Interesting notes:
============
1- Serving PNG is 10X faster than html
2- Serving CSS is 10X faster than PNG!
3- Serving html is really the bottleneck. Unfortunately, Moin developers
acknowledged current version ( 1.5) is not cache friendly. Work for making
1.6 cache friendly is undergoing
4- Using squid currently only seems to double our PNG serving rate,
nothing else
5- The application server hits swapping (about 0.5GB) at full load (~300
concurrent connections), for some reason the requests/second served is still
high!! (Is our cache disks that fast)
6- The test did not stress the server bad enough to run out of swap space,
not sure if this is needed though!
I can send the full results date if anyone is interested.
Best Regards
_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
9:35 p.m.
On 12/23/06, Paulo Santos <paulo.banon(a)googlemail.com> wrote:
> 3- Serving html is really the bottleneck. Unfortunately, Moin
developers
acknowledged current version ( 1.5) is not cache friendly. Work for making
It sounds like we really won't get much of a benefit till 1.6. Our
biggest issue last time appeard to be the dynamic generation of
content, converting them to static content helped cpu load on fpserv
quite a bit.
How did the appserver handle the stress test? How about the proxy
server? Not just in code generation and such but actual cpuload, that
type of thing?
-Mike
4:55 p.m.
Under full load (~200 concurrent connections), this was the status
Proxy2 machine
===========
1- RAM 100% used, machine not swapping though (Linux ram utilization is a
bit complex though)
2- Squid was saturating the CPU (80~90%)
3- Lots and lots of httpd processess, each taking 0% cpu, and minimal ram
~0.7%
4- load average: 1.06, 0.94, 0.75. top is showing squid cpu utilization >
85%, while total cpu utilization = 7%!! This seems to be a bug?! So, I cant
trust the load numbers either
App1 server
=========
1- RAM 100% used, 0.5GB swapping
2- CPU saturated by large number of httpd processes
3- load average: 150.35, 143.94, 104.04. Cacti showing load peaking to 400,
while top doesnt go that high!
https://admin.fedoraproject.org/cacti/graph.php?action=zoom&local_gra...
Best Regards
On 12/24/06, Mike McGrath <mmcgrath(a)fedoraproject.org> wrote:
On 12/23/06, Paulo Santos <paulo.banon(a)googlemail.com> wrote:
> > 3- Serving html is really the bottleneck. Unfortunately, Moin
developers
> acknowledged current version ( 1.5) is not cache friendly. Work for
making
It sounds like we really won't get much of a benefit till 1.6. Our
biggest issue last time appeard to be the dynamic generation of
content, converting them to static content helped cpu load on fpserv
quite a bit.
How did the appserver handle the stress test? How about the proxy
server? Not just in code generation and such but actual cpuload, that
type of thing?
-Mike
9:13 a.m.
On Dec 24, 2006, at 23:55, Ahmed Kamal wrote:
1- RAM 100% used, 0.5GB swapping
You should configure the web server for fewer concurrent
connections. Having it swap is most certainly bad for performance
(unless it's swapping unrelated and inactive programs).
- ask
--
http://develooper.com/ - http://askask.com/
9:16 a.m.
Instead of Squid it might be worthwhile looking at Varnish for the
reverse-proxy cache.
http://phk.freebsd.dk/pubs/varnish_tech.pdf
http://varnish.linpro.no/
- ask
9:04 p.m.
Hey Ahmed,
It's neat to see these results! A few comments:
I don't see anywhere where variables such as the file size, request
size, HTTP pipelining, etc. are taken into account. Optimally, the
test would have cases that exercise points along all these
dimensions. E.g. a load test of serving an empty file would help us
understand what the absolute best case performance is within the
constraints the current systems.
The other thing that'd be useful to have is a well-defined capacity
goal that helps everyone understand where things ought to be. For a
straw-man example, let's say 20 home page loads/second, including all
associated image/CSS/JS files, by logged-in users. I don't know what
the actual needs are, but a look at the web logs during the FC6
release should help.
From what's mentioned below, I also didn't quite understand whether
squid was sitting in front of apache serving straight HTML/PNG/CSS,
or whether it was moinmoin that contained all the content. Hopefully
that's just something I missed in an earlier message.
My current favorite treatise on optimizing overall web performance -
http://www.die.net/musings/page_load_time/
Hope this helps,
-- Elliot
On Dec 23, 2006, at 3:19 PM, Ahmed Kamal wrote:
Hi,
Paulo and me (kim0) have been working on testing a caching setup
for the wiki. A test migration to Moin 1.5 is complete, squid is
now configured as a reverse caching proxy. We've done some stress
tests on the current setup. Attached are some extracted results
that (I think) are of interest. Mainly the number of requests
served per second, and the average time for serving a request.
Also, paulo pointed that caching differs per file type, the tests
have been done on three different file types (html, png image, and
css)
Test Setup:
=========
1- All connections were initiated from proxy1
2- Proxy2 had squid caching turned on
3- Testing for html/png/css done, sweeping the number of concurrent
connections
4- Turn off squid caching on proxy2
5- Testing for html/png/css done, sweeping the number of concurrent
connections again
Interesting notes:
============
1- Serving PNG is 10X faster than html
2- Serving CSS is 10X faster than PNG!
3- Serving html is really the bottleneck. Unfortunately, Moin
developers acknowledged current version ( 1.5) is not cache
friendly. Work for making 1.6 cache friendly is undergoing
4- Using squid currently only seems to double our PNG serving rate,
nothing else
5- The application server hits swapping (about 0.5GB) at full load
(~300 concurrent connections), for some reason the requests/second
served is still high!! (Is our cache disks that fast)
6- The test did not stress the server bad enough to run out of swap
space, not sure if this is needed though!
I can send the full results date if anyone is interested.
Best Regards
<results.ods>
_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
6324
days inactive
6327
days old
infrastructure@lists.fedoraproject.org
6 comments
5 participants
participants (5)
-
Ahmed Kamal -
Ask Bjørn Hansen -
Elliot Lee -
Mike McGrath -
Paulo Santos