11:36 a.m.
Hi,
Recently we've run into a problem where Koji uses far too many database
connections and run over the simultaenous connections limit (of 100).
The connections only last for a few seconds. This has caused lots of
temporary "Database Outage" errors. I'd hate to keep bumping the
connection limit since I feel like that's just papering over the
problem.
The specific error I see in the postgres log is:
FATAL: connection limit exceeded for non-superusers
I also see lots of the following:
LOG: unexpected EOF on client connection
Does anyone know what might be going on? This setup has been working
fine for years and the problem has only shown up a few months ago.
Unfortunately we're still on Koji 1.11.0.
Thanks,
-Mat
--
Mátyás (Mat) Selmeci
Open Science Grid Software Team / Center for High-Throughput Computing
University of Wisconsin-Madison Department of Computer Sciences
12:17 p.m.
Can you raise the limit in postgres? Optionally, lower MaxClients in httpd
(that shifts the problem around a bit, but most Koji clients, e.g.
builders, should retry if httpd refuses connection).
Maybe crank up kojihub logging and try to see what client is overloading
your hub with rapid calls?
How many builders do you have? How many users?
On Tue, Oct 23, 2018 at 12:40 PM Mátyás Selmeci <matyas(a)cs.wisc.edu> wrote:
Hi,
Recently we've run into a problem where Koji uses far too many database
connections and run over the simultaenous connections limit (of 100).
The connections only last for a few seconds. This has caused lots of
temporary "Database Outage" errors. I'd hate to keep bumping the
connection limit since I feel like that's just papering over the
problem.
The specific error I see in the postgres log is:
FATAL: connection limit exceeded for non-superusers
I also see lots of the following:
LOG: unexpected EOF on client connection
Does anyone know what might be going on? This setup has been working
fine for years and the problem has only shown up a few months ago.
Unfortunately we're still on Koji 1.11.0.
Thanks,
-Mat
--
Mátyás (Mat) Selmeci
Open Science Grid Software Team / Center for High-Throughput Computing
University of Wisconsin-Madison Department of Computer Sciences
_______________________________________________
koji-devel mailing list -- koji-devel(a)lists.fedorahosted.org
To unsubscribe send an email to koji-devel-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahoste...
6:33 p.m.
Yeah, I'll raise the postgres limit to 200. Thing is, it's worked for
years at 25, so I'm worried that it's a symptom of a larger issue.
Hmm. I'm seeing frequent rapid accesses by kojira in ssl_access_log. I
wonder if that's it...
Thanks,
-Mat
On 10/23/18 12:17 PM, Michael McLean wrote:
Can you raise the limit in postgres? Optionally, lower MaxClients in
httpd (that shifts the problem around a bit, but most Koji clients,
e.g. builders, should retry if httpd refuses connection).
Maybe crank up kojihub logging and try to see what client is
overloading your hub with rapid calls?
How many builders do you have? How many users?
On Tue, Oct 23, 2018 at 12:40 PM Mátyás Selmeci <matyas(a)cs.wisc.edu
<mailto:matyas@cs.wisc.edu>> wrote:
Hi,
Recently we've run into a problem where Koji uses far too many
database
connections and run over the simultaenous connections limit (of
100).
The connections only last for a few seconds. This has caused
lots of
temporary "Database Outage" errors. I'd hate to keep bumping the
connection limit since I feel like that's just papering over the
problem.
The specific error I see in the postgres log is:
FATAL: connection limit exceeded for non-superusers
I also see lots of the following:
LOG: unexpected EOF on client connection
Does anyone know what might be going on? This setup has been
working
fine for years and the problem has only shown up a few months ago.
Unfortunately we're still on Koji 1.11.0.
Thanks,
-Mat
--
Mátyás (Mat) Selmeci
Open Science Grid Software Team / Center for High-Throughput
Computing
University of Wisconsin-Madison Department of Computer Sciences
_______________________________________________
koji-devel mailing list -- koji-devel(a)lists.fedorahosted.org
<mailto:koji-devel@lists.fedorahosted.org>
To unsubscribe send an email to
koji-devel-leave(a)lists.fedorahosted.org
<mailto:koji-devel-leave@lists.fedorahosted.org>
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahoste...
_______________________________________________
koji-devel mailing list -- koji-devel(a)lists.fedorahosted.org
To unsubscribe send an email to koji-devel-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahoste...
7:31 p.m.
On Tue, Oct 23, 2018 at 7:35 PM Mátyás Selmeci <matyas(a)cs.wisc.edu> wrote:
Yeah, I'll raise the postgres limit to 200. Thing is, it's
worked for
years at 25, so I'm worried that it's a symptom of a larger issue.
Hmm. I'm seeing frequent rapid accesses by kojira in ssl_access_log. I
wonder if that's it...
kojira is unlikely to be the culprit. It is only one daemon. It does have
a few
threads that each connect, so it's consuming up to three connections,
but that's a far cry from 100. Sure, there are lots of calls, but mostly
serialized.
You might try setting LogLevel = INFO in hub.conf and looking at the call
records in ssl_error_log (and/or error_log). Those will have a bit more
info than the access logs.
Otoh, perhaps your usage has grown slowly over the years and just now
reached a tipping point?
Thanks,
-Mat
On 10/23/18 12:17 PM, Michael McLean wrote:
Can you raise the limit in postgres? Optionally, lower MaxClients in httpd
(that shifts the problem around a bit, but most Koji clients, e.g.
builders, should retry if httpd refuses connection).
Maybe crank up kojihub logging and try to see what client is overloading
your hub with rapid calls?
How many builders do you have? How many users?
On Tue, Oct 23, 2018 at 12:40 PM Mátyás Selmeci <matyas(a)cs.wisc.edu>
wrote:
> Hi,
>
> Recently we've run into a problem where Koji uses far too many database
> connections and run over the simultaenous connections limit (of 100).
> The connections only last for a few seconds. This has caused lots of
> temporary "Database Outage" errors. I'd hate to keep bumping the
> connection limit since I feel like that's just papering over the
> problem.
>
> The specific error I see in the postgres log is:
>
> FATAL: connection limit exceeded for non-superusers
>
> I also see lots of the following:
>
> LOG: unexpected EOF on client connection
>
> Does anyone know what might be going on? This setup has been working
> fine for years and the problem has only shown up a few months ago.
>
> Unfortunately we're still on Koji 1.11.0.
>
> Thanks,
> -Mat
>
>
> --
> Mátyás (Mat) Selmeci
> Open Science Grid Software Team / Center for High-Throughput Computing
> University of Wisconsin-Madison Department of Computer Sciences
> _______________________________________________
> koji-devel mailing list -- koji-devel(a)lists.fedorahosted.org
> To unsubscribe send an email to koji-devel-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahoste...
>
_______________________________________________
koji-devel mailing list -- koji-devel(a)lists.fedorahosted.org
To unsubscribe send an email to koji-devel-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahoste...
_______________________________________________
koji-devel mailing list -- koji-devel(a)lists.fedorahosted.org
To unsubscribe send an email to koji-devel-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahoste...
7:56 p.m.
On 10/23/18 7:31 PM, Michael McLean wrote:
On Tue, Oct 23, 2018 at 7:35 PM Mátyás Selmeci
<matyas(a)cs.wisc.edu
<mailto:matyas@cs.wisc.edu>> wrote:
Yeah, I'll raise the postgres limit to 200. Thing is, it's
worked for years at 25, so I'm worried that it's a symptom of a
larger issue.
Hmm. I'm seeing frequent rapid accesses by kojira in
ssl_access_log. I wonder if that's it...
kojira is unlikely to be the culprit. It is only one daemon. It does
have a few threads that each connect, so it's consuming up to three
connections, but that's a far cry from 100. Sure, there are lots of
calls, but mostly serialized.
You might try setting LogLevel = INFO in hub.conf and looking at the
call records in ssl_error_log (and/or error_log). Those will have a
bit more info than the access logs.
Otoh, perhaps your usage has grown slowly over the years and just
now reached a tipping point?
Perhaps? I restarted Kojira to add a debugging line and the number of
connections dropped down to 7--10 and is still holding steady at that
range, even while I was doing a build and it was regenerating the repos.
I'll leave things going as-is for a while and see if the issue comes back.
-Mat
On 10/23/18 12:17 PM, Michael McLean wrote:
> Can you raise the limit in postgres? Optionally, lower
> MaxClients in httpd (that shifts the problem around a bit, but
> most Koji clients, e.g. builders, should retry if httpd refuses
> connection).
>
> Maybe crank up kojihub logging and try to see what client is
> overloading your hub with rapid calls?
>
> How many builders do you have? How many users?
>
> On Tue, Oct 23, 2018 at 12:40 PM Mátyás Selmeci
> <matyas(a)cs.wisc.edu <mailto:matyas@cs.wisc.edu>> wrote:
>
> Hi,
>
> Recently we've run into a problem where Koji uses far too
> many database
> connections and run over the simultaenous connections limit
> (of 100).
> The connections only last for a few seconds. This has
> caused lots of
> temporary "Database Outage" errors. I'd hate to keep
> bumping the
> connection limit since I feel like that's just papering
> over the
> problem.
>
> The specific error I see in the postgres log is:
>
> FATAL: connection limit exceeded for non-superusers
>
> I also see lots of the following:
>
> LOG: unexpected EOF on client connection
>
> Does anyone know what might be going on? This setup has
> been working
> fine for years and the problem has only shown up a few
> months ago.
>
> Unfortunately we're still on Koji 1.11.0.
>
> Thanks,
> -Mat
>
>
> --
> Mátyás (Mat) Selmeci
> Open Science Grid Software Team / Center for
> High-Throughput Computing
> University of Wisconsin-Madison Department of Computer Sciences
>
2017
days inactive
2018
days old
koji-devel@lists.fedorahosted.org
4 comments
2 participants
participants (2)
-
Michael McLean -
Mátyás Selmeci