We have been having some persistent issues with kojipkgs01 lately.
kojipkgs01 is our squid proxy in front of koji builds. It allows users and builders to get fast access to packages. (When it's working).
Lately, it's been working fine at first, then in a few days or so it starts getting really slow. Downloads go from 25M/s to 200k/sec and sometimes things even just timeout.
Restarting squid seems to fix this... for a few more days. There is never any errors on the box, i/o, load and everything is fine.
I looked this morning a bunch at options and adjusted the memory cache down in case we were hitting some kind of issue with memory cache.
I'd like +1's for that change, and also to solicit ideas for what we can do to fix this once and for all (if these changes don't do so).
diff --git a/roles/kojipkgs/files/squid.conf b/roles/kojipkgs/files/squid.conf index b011143..a0d5312 100644 --- a/roles/kojipkgs/files/squid.conf +++ b/roles/kojipkgs/files/squid.conf @@ -6,8 +6,8 @@ hierarchy_stoplist cgi-bin ?
cache_swap_low 98 cache_swap_high 99 -cache_mem 50 GB -maximum_object_size 700 MB +cache_mem 10 GB +maximum_object_size 200 MB minimum_object_size 0 KB cache_replacement_policy heap LFUDA maximum_object_size_in_memory 100 MB
On Thu, Feb 26, 2015 at 08:11:50AM -0700, Kevin Fenzi wrote:
We have been having some persistent issues with kojipkgs01 lately.
kojipkgs01 is our squid proxy in front of koji builds. It allows users and builders to get fast access to packages. (When it's working).
Lately, it's been working fine at first, then in a few days or so it starts getting really slow. Downloads go from 25M/s to 200k/sec and sometimes things even just timeout.
Restarting squid seems to fix this... for a few more days. There is never any errors on the box, i/o, load and everything is fine.
I looked this morning a bunch at options and adjusted the memory cache down in case we were hitting some kind of issue with memory cache.
I'd like +1's for that change, and also to solicit ideas for what we can do to fix this once and for all (if these changes don't do so).
diff --git a/roles/kojipkgs/files/squid.conf b/roles/kojipkgs/files/squid.conf index b011143..a0d5312 100644 --- a/roles/kojipkgs/files/squid.conf +++ b/roles/kojipkgs/files/squid.conf @@ -6,8 +6,8 @@ hierarchy_stoplist cgi-bin ?
cache_swap_low 98 cache_swap_high 99 -cache_mem 50 GB -maximum_object_size 700 MB +cache_mem 10 GB +maximum_object_size 200 MB minimum_object_size 0 KB cache_replacement_policy heap LFUDA maximum_object_size_in_memory 100 MB
+1 for me
We can always revert if we see it doesn't improve
Pierre
On Thu, 26 Feb 2015 08:11:50 -0700 Kevin Fenzi kevin@scrye.com wrote:
We have been having some persistent issues with kojipkgs01 lately.
kojipkgs01 is our squid proxy in front of koji builds. It allows users and builders to get fast access to packages. (When it's working).
Lately, it's been working fine at first, then in a few days or so it starts getting really slow. Downloads go from 25M/s to 200k/sec and sometimes things even just timeout.
Restarting squid seems to fix this... for a few more days. There is never any errors on the box, i/o, load and everything is fine.
I looked this morning a bunch at options and adjusted the memory cache down in case we were hitting some kind of issue with memory cache.
I'd like +1's for that change, and also to solicit ideas for what we can do to fix this once and for all (if these changes don't do so).
diff --git a/roles/kojipkgs/files/squid.conf b/roles/kojipkgs/files/squid.conf index b011143..a0d5312 100644 --- a/roles/kojipkgs/files/squid.conf +++ b/roles/kojipkgs/files/squid.conf @@ -6,8 +6,8 @@ hierarchy_stoplist cgi-bin ?
cache_swap_low 98 cache_swap_high 99 -cache_mem 50 GB -maximum_object_size 700 MB +cache_mem 10 GB +maximum_object_size 200 MB minimum_object_size 0 KB cache_replacement_policy heap LFUDA maximum_object_size_in_memory 100 MB
+1 from me as well, hopefully this'll fix things but as pingou said, it's easy to revert if it ends up causing problems
+1 Hopefully this will fix it
On Thu, Feb 26, 2015, 11:14 AM Tim Flink tflink@redhat.com wrote:
On Thu, 26 Feb 2015 08:11:50 -0700 Kevin Fenzi kevin@scrye.com wrote:
We have been having some persistent issues with kojipkgs01 lately.
kojipkgs01 is our squid proxy in front of koji builds. It allows users and builders to get fast access to packages. (When it's working).
Lately, it's been working fine at first, then in a few days or so it starts getting really slow. Downloads go from 25M/s to 200k/sec and sometimes things even just timeout.
Restarting squid seems to fix this... for a few more days. There is never any errors on the box, i/o, load and everything is fine.
I looked this morning a bunch at options and adjusted the memory cache down in case we were hitting some kind of issue with memory cache.
I'd like +1's for that change, and also to solicit ideas for what we can do to fix this once and for all (if these changes don't do so).
diff --git a/roles/kojipkgs/files/squid.conf b/roles/kojipkgs/files/squid.conf index b011143..a0d5312 100644 --- a/roles/kojipkgs/files/squid.conf +++ b/roles/kojipkgs/files/squid.conf @@ -6,8 +6,8 @@ hierarchy_stoplist cgi-bin ?
cache_swap_low 98 cache_swap_high 99 -cache_mem 50 GB -maximum_object_size 700 MB +cache_mem 10 GB +maximum_object_size 200 MB minimum_object_size 0 KB cache_replacement_policy heap LFUDA maximum_object_size_in_memory 100 MB
+1 from me as well, hopefully this'll fix things but as pingou said, it's easy to revert if it ends up causing problems _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Thu, 26 Feb 2015 08:11:50 -0700 Kevin Fenzi kevin@scrye.com wrote:
We have been having some persistent issues with kojipkgs01 lately.
kojipkgs01 is our squid proxy in front of koji builds. It allows users and builders to get fast access to packages. (When it's working).
Lately, it's been working fine at first, then in a few days or so it starts getting really slow. Downloads go from 25M/s to 200k/sec and sometimes things even just timeout.
Restarting squid seems to fix this... for a few more days. There is never any errors on the box, i/o, load and everything is fine.
I looked this morning a bunch at options and adjusted the memory cache down in case we were hitting some kind of issue with memory cache.
I'd like +1's for that change, and also to solicit ideas for what we can do to fix this once and for all (if these changes don't do so).
So, I finally tracked down the issues with smp mode on squid. ;)
It's two slightly different bugs on Fedora and RHEL. Fedora: https://bugzilla.redhat.com/show_bug.cgi?id=1176318 RHEL: https://bugzilla.redhat.com/show_bug.cgi?id=1102842
In Fedora squid is set to use a /var/run/squid dir for pid/ipc state, but it doesn't have a tmpfiles.d config to make that dir on boot, so it only works until you reboot.
In RHEL, they compiled it to use /var/run/squid.pid for the pid file, and the ipc state files can't be written to /var/run/ as squid user, so they nicely silently fail to talk to each other and no one binds to the port and nothing works.
We can work around this issue for now on RHEL with:
* Set pid_filename /var/run/squid/squid.pid in the squid.conf.
* Add a selinux local policy or set permissive mode: #============= squid_t ============== allow squid_t var_run_t:sock_file { read write create };
* Add a /etc/tmpfiles.d/squid.conf with: D /var/run/squid 0755 squid squid -
So, I guess we can wait and see if the slowdown happens again, and if so, try the above to enable smp mode and see if it helps any?
kevin
infrastructure@lists.fedoraproject.org
-
Chaoyi Zha -
Kevin Fenzi -
Pierre-Yves Chibon -
Tim Flink