Hi, Paulo and me (kim0) have been working on testing a caching setup for the wiki. A test migration to Moin 1.5 is complete, squid is now configured as a reverse caching proxy. We've done some stress tests on the current setup. Attached are some extracted results that (I think) are of interest. Mainly the number of requests served per second, and the average time for serving a request. Also, paulo pointed that caching differs per file type, the tests have been done on three different file types (html, png image, and css)
Test Setup: ========= 1- All connections were initiated from proxy1 2- Proxy2 had squid caching turned on 3- Testing for html/png/css done, sweeping the number of concurrent connections 4- Turn off squid caching on proxy2 5- Testing for html/png/css done, sweeping the number of concurrent connections again
Interesting notes: ============ 1- Serving PNG is 10X faster than html 2- Serving CSS is 10X faster than PNG! 3- Serving html is really the bottleneck. Unfortunately, Moin developers acknowledged current version (1.5) is not cache friendly. Work for making 1.6 cache friendly is undergoing 4- Using squid currently only seems to double our PNG serving rate, nothing else 5- The application server hits swapping (about 0.5GB) at full load (~300 concurrent connections), for some reason the requests/second served is still high!! (Is our cache disks that fast) 6- The test did not stress the server bad enough to run out of swap space, not sure if this is needed though!
I can send the full results date if anyone is interested.
Best Regards
Hi all,
The PNG link i gave kim0 was just an example of an image. Currently we are caching all general type of images (gif, jpg, png). As Ahmed pointed, current version of Moin isn't still html friendly, hopefully this will be addressed at Moin 1.6.x (our current version is Moin 1.3.4 and our testing was based on Moin 1.5.6). Moin 1.5.6, still bring alot of improvements, so in my opinion, even though that Moin html is not cache friendly we should still be thinking in the migration. Also as a workaround, we can still create a static website that with the help of mod_headers, it could be forced to be cached. Last but not least, let's not forget that this was a local test. A more real stress test, is still going to be prepared and should include a server requesting to the future architecture (including load balancer and both proxies), this will improve even more our ability to deliver the desired content.
We would like some coments on this tests (past and future), so that we can improve what needs to be improved. As a final note, i would like all of you to start testing the future platform[1]. Just play with Moin, and give some feedback on the WikiMigration page[2].
[1] http://webtest.fedora.redhat.com/wiki [2] http://fedoraproject.org/wiki/Infrastructure/WikiMigration
Once again thanks all for the help and feedback...
Paulo
On 12/23/06, Ahmed Kamal email.ahmedkamal@googlemail.com wrote:
Hi, Paulo and me (kim0) have been working on testing a caching setup for the wiki. A test migration to Moin 1.5 is complete, squid is now configured as a reverse caching proxy. We've done some stress tests on the current setup. Attached are some extracted results that (I think) are of interest. Mainly the number of requests served per second, and the average time for serving a request. Also, paulo pointed that caching differs per file type, the tests have been done on three different file types (html, png image, and css)
Test Setup:
1- All connections were initiated from proxy1 2- Proxy2 had squid caching turned on 3- Testing for html/png/css done, sweeping the number of concurrent connections 4- Turn off squid caching on proxy2 5- Testing for html/png/css done, sweeping the number of concurrent connections again
Interesting notes:
1- Serving PNG is 10X faster than html 2- Serving CSS is 10X faster than PNG! 3- Serving html is really the bottleneck. Unfortunately, Moin developers acknowledged current version ( 1.5) is not cache friendly. Work for making 1.6 cache friendly is undergoing 4- Using squid currently only seems to double our PNG serving rate, nothing else 5- The application server hits swapping (about 0.5GB) at full load (~300 concurrent connections), for some reason the requests/second served is still high!! (Is our cache disks that fast) 6- The test did not stress the server bad enough to run out of swap space, not sure if this is needed though!
I can send the full results date if anyone is interested.
Best Regards
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
On 12/23/06, Paulo Santos paulo.banon@googlemail.com wrote:
3- Serving html is really the bottleneck. Unfortunately, Moin developers
acknowledged current version ( 1.5) is not cache friendly. Work for making
It sounds like we really won't get much of a benefit till 1.6. Our biggest issue last time appeard to be the dynamic generation of content, converting them to static content helped cpu load on fpserv quite a bit.
How did the appserver handle the stress test? How about the proxy server? Not just in code generation and such but actual cpuload, that type of thing?
-Mike
Under full load (~200 concurrent connections), this was the status
Proxy2 machine =========== 1- RAM 100% used, machine not swapping though (Linux ram utilization is a bit complex though) 2- Squid was saturating the CPU (80~90%) 3- Lots and lots of httpd processess, each taking 0% cpu, and minimal ram ~0.7% 4- load average: 1.06, 0.94, 0.75. top is showing squid cpu utilization > 85%, while total cpu utilization = 7%!! This seems to be a bug?! So, I cant trust the load numbers either
App1 server ========= 1- RAM 100% used, 0.5GB swapping 2- CPU saturated by large number of httpd processes 3- load average: 150.35, 143.94, 104.04. Cacti showing load peaking to 400, while top doesnt go that high! https://admin.fedoraproject.org/cacti/graph.php?action=zoom&local_graph_...
Best Regards
On 12/24/06, Mike McGrath mmcgrath@fedoraproject.org wrote:
On 12/23/06, Paulo Santos paulo.banon@googlemail.com wrote:
3- Serving html is really the bottleneck. Unfortunately, Moin
developers
acknowledged current version ( 1.5) is not cache friendly. Work for
making
It sounds like we really won't get much of a benefit till 1.6. Our biggest issue last time appeard to be the dynamic generation of content, converting them to static content helped cpu load on fpserv quite a bit.
How did the appserver handle the stress test? How about the proxy server? Not just in code generation and such but actual cpuload, that type of thing?
-Mike
On Dec 24, 2006, at 23:55, Ahmed Kamal wrote:
1- RAM 100% used, 0.5GB swapping
You should configure the web server for fewer concurrent connections. Having it swap is most certainly bad for performance (unless it's swapping unrelated and inactive programs).
- ask
Instead of Squid it might be worthwhile looking at Varnish for the reverse-proxy cache.
http://phk.freebsd.dk/pubs/varnish_tech.pdf
- ask
Hey Ahmed,
It's neat to see these results! A few comments:
I don't see anywhere where variables such as the file size, request size, HTTP pipelining, etc. are taken into account. Optimally, the test would have cases that exercise points along all these dimensions. E.g. a load test of serving an empty file would help us understand what the absolute best case performance is within the constraints the current systems.
The other thing that'd be useful to have is a well-defined capacity goal that helps everyone understand where things ought to be. For a straw-man example, let's say 20 home page loads/second, including all associated image/CSS/JS files, by logged-in users. I don't know what the actual needs are, but a look at the web logs during the FC6 release should help.
From what's mentioned below, I also didn't quite understand whether squid was sitting in front of apache serving straight HTML/PNG/CSS, or whether it was moinmoin that contained all the content. Hopefully that's just something I missed in an earlier message.
My current favorite treatise on optimizing overall web performance - http://www.die.net/musings/page_load_time/
Hope this helps, -- Elliot
On Dec 23, 2006, at 3:19 PM, Ahmed Kamal wrote:
Hi, Paulo and me (kim0) have been working on testing a caching setup for the wiki. A test migration to Moin 1.5 is complete, squid is now configured as a reverse caching proxy. We've done some stress tests on the current setup. Attached are some extracted results that (I think) are of interest. Mainly the number of requests served per second, and the average time for serving a request. Also, paulo pointed that caching differs per file type, the tests have been done on three different file types (html, png image, and css)
Test Setup:
1- All connections were initiated from proxy1 2- Proxy2 had squid caching turned on 3- Testing for html/png/css done, sweeping the number of concurrent connections 4- Turn off squid caching on proxy2 5- Testing for html/png/css done, sweeping the number of concurrent connections again
Interesting notes:
1- Serving PNG is 10X faster than html 2- Serving CSS is 10X faster than PNG! 3- Serving html is really the bottleneck. Unfortunately, Moin developers acknowledged current version ( 1.5) is not cache friendly. Work for making 1.6 cache friendly is undergoing 4- Using squid currently only seems to double our PNG serving rate, nothing else 5- The application server hits swapping (about 0.5GB) at full load (~300 concurrent connections), for some reason the requests/second served is still high!! (Is our cache disks that fast) 6- The test did not stress the server bad enough to run out of swap space, not sure if this is needed though!
I can send the full results date if anyone is interested.
Best Regards <results.ods> _______________________________________________ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
infrastructure@lists.fedoraproject.org