6:44 a.m.
I fixed this now, but I could find virtually no documentation about it
online, so I'm writing this email to document what surely must be a
common problem ...
I wanted to enable HTTP/2 support in Apache on Fedora 38.
I followed the documentation here which worked [although it's way more
complicated than it needs to be, why isn't HTTP/2 the default out of
the box?]
https://httpd.apache.org/docs/2.4/howto/http2.html
Anyway the problem I had was that the server worked fine provided
there were not too many clients (and by "too many" I mean a simple
load test with 4-16 clients failed). Apache randomly threw 403
Forbidden errors, but with less load it gave a normal (2xx) response.
The first problem is the error is misleading:
[Wed Feb 22 13:24:52.013780 2023] [core:error] [pid 3047850:tid 3047899] (24)Too many
open files: [remote 192.168.0.139:53738] AH00132: file permissions deny server access:
/var/www/html/[filename]
If you concentrate on the second part "file permissions deny server
access" -- as I did -- then you'll be looking at file permissions,
SELinux, restorecon, ausearch etc. That's a red herring, there is no
permissions problem.
The real error is the first part "Too many open files".
It turns out that the default open file limit (1024!) is too low. To
change this and fix the problem:
# systemctl edit httpd
This creates an "override" file to which you should add (or you could
just create this file directly):
# cat /etc/systemd/system/httpd.service.d/override.conf
[Service]
LimitNOFILE=65536
and then restart Apache for the change to take effect.
Why on earth Apache needs > 1024 open files to serve a dozen clients
is not clear at all.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
9:03 a.m.
Do a lsof -p <pid-of-an-httpd-processw> and it will list out the files.
It may be a file handle leak.
I have seen leaks from failure to close a file when a process is done.
There can be leaks if a process uses an anonymous memory allocation
trick that relies on file handles, and there are probably others.
The lsof output will give you an idea if it is a leak (usually it will
be the same files being opened again and again without closing).
If it is a leak and the rate is high enough you may not be able to set
the open files high enough for it to work for long.
On Mon, Feb 27, 2023 at 6:45 AM Richard W.M. Jones <rjones(a)redhat.com> wrote:
>
> I fixed this now, but I could find virtually no documentation about it
> online, so I'm writing this email to document what surely must be a
> common problem ...
>
> I wanted to enable HTTP/2 support in Apache on Fedora 38.
>
> I followed the documentation here which worked [although it's way more
> complicated than it needs to be, why isn't HTTP/2 the default out of
> the box?]
>
> https://httpd.apache.org/docs/2.4/howto/http2.html
>
> Anyway the problem I had was that the server worked fine provided
> there were not too many clients (and by "too many" I mean a simple
> load test with 4-16 clients failed). Apache randomly threw 403
> Forbidden errors, but with less load it gave a normal (2xx) response.
>
> The first problem is the error is misleading:
>
> [Wed Feb 22 13:24:52.013780 2023] [core:error] [pid 3047850:tid 3047899] (24)Too
many open files: [remote 192.168.0.139:53738] AH00132: file permissions deny server
access: /var/www/html/[filename]
>
> If you concentrate on the second part "file permissions deny server
> access" -- as I did -- then you'll be looking at file permissions,
> SELinux, restorecon, ausearch etc. That's a red herring, there is no
> permissions problem.
>
> The real error is the first part "Too many open files".
>
> It turns out that the default open file limit (1024!) is too low. To
> change this and fix the problem:
>
> # systemctl edit httpd
>
> This creates an "override" file to which you should add (or you could
> just create this file directly):
>
> # cat /etc/systemd/system/httpd.service.d/override.conf
> [Service]
> LimitNOFILE=65536
>
> and then restart Apache for the change to take effect.
>
> Why on earth Apache needs > 1024 open files to serve a dozen clients
> is not clear at all.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-top is 'top' for virtual machines. Tiny program with many
> powerful monitoring features, net stats, disk stats, logging, etc.
> http://people.redhat.com/~rjones/virt-top
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
10:42 a.m.
Hi Rich,
[although it's way more
complicated than it needs to be, why isn't HTTP/2 the default out of
the box?]
HTTP/2 is insecure out-of-the-box. Remember CRIME and BREACH? The
protocol requires compression, and compression is a known attack
vector. From the abstract of RFC 7450:
This specification describes an optimized expression of the semantics
of the Hypertext Transfer Protocol (HTTP), referred to as HTTP
version 2 (HTTP/2). HTTP/2 enables a more efficient use of network
resources and a reduced perception of latency by introducing header
field compression and allowing multiple concurrent exchanges on the
same connection. It also introduces unsolicited push of
representations from servers to clients.
I am also not sure the push functionality is well understood in a
security context.
So it is probably a good idea to make HTTP/2 optional, until an
organization has an opportunity to weigh the risks versus reward.
Jeff
On Mon, Feb 27, 2023 at 7:44 AM Richard W.M. Jones <rjones(a)redhat.com> wrote:
>
> I fixed this now, but I could find virtually no documentation about it
> online, so I'm writing this email to document what surely must be a
> common problem ...
>
> I wanted to enable HTTP/2 support in Apache on Fedora 38.
>
> I followed the documentation here which worked [although it's way more
> complicated than it needs to be, why isn't HTTP/2 the default out of
> the box?]
>
> https://httpd.apache.org/docs/2.4/howto/http2.html
>
> Anyway the problem I had was that the server worked fine provided
> there were not too many clients (and by "too many" I mean a simple
> load test with 4-16 clients failed). Apache randomly threw 403
> Forbidden errors, but with less load it gave a normal (2xx) response.
>
> The first problem is the error is misleading:
>
> [Wed Feb 22 13:24:52.013780 2023] [core:error] [pid 3047850:tid 3047899] (24)Too
many open files: [remote 192.168.0.139:53738] AH00132: file permissions deny server
access: /var/www/html/[filename]
>
> If you concentrate on the second part "file permissions deny server
> access" -- as I did -- then you'll be looking at file permissions,
> SELinux, restorecon, ausearch etc. That's a red herring, there is no
> permissions problem.
>
> The real error is the first part "Too many open files".
>
> It turns out that the default open file limit (1024!) is too low. To
> change this and fix the problem:
>
> # systemctl edit httpd
>
> This creates an "override" file to which you should add (or you could
> just create this file directly):
>
> # cat /etc/systemd/system/httpd.service.d/override.conf
> [Service]
> LimitNOFILE=65536
>
> and then restart Apache for the change to take effect.
>
> Why on earth Apache needs > 1024 open files to serve a dozen clients
> is not clear at all.
>
1:56 p.m.
On Mon, Feb 27, 2023 at 11:42:49AM -0500, Jeffrey Walton wrote:
Hi Rich,
> [although it's way more
> complicated than it needs to be, why isn't HTTP/2 the default out of
> the box?]
HTTP/2 is insecure out-of-the-box. Remember CRIME and BREACH? The
protocol requires compression, and compression is a known attack
vector. From the abstract of RFC 7450:
This specification describes an optimized expression of the semantics
of the Hypertext Transfer Protocol (HTTP), referred to as HTTP
version 2 (HTTP/2). HTTP/2 enables a more efficient use of network
resources and a reduced perception of latency by introducing header
field compression and allowing multiple concurrent exchanges on the
same connection. It also introduces unsolicited push of
representations from servers to clients.
I am also not sure the push functionality is well understood in a
security context.
So it is probably a good idea to make HTTP/2 optional, until an
organization has an opportunity to weigh the risks versus reward.
Good points, thanks.
Rich.
Jeff
On Mon, Feb 27, 2023 at 7:44 AM Richard W.M. Jones <rjones(a)redhat.com> wrote:
>
> I fixed this now, but I could find virtually no documentation about it
> online, so I'm writing this email to document what surely must be a
> common problem ...
>
> I wanted to enable HTTP/2 support in Apache on Fedora 38.
>
> I followed the documentation here which worked [although it's way more
> complicated than it needs to be, why isn't HTTP/2 the default out of
> the box?]
>
> https://httpd.apache.org/docs/2.4/howto/http2.html
>
> Anyway the problem I had was that the server worked fine provided
> there were not too many clients (and by "too many" I mean a simple
> load test with 4-16 clients failed). Apache randomly threw 403
> Forbidden errors, but with less load it gave a normal (2xx) response.
>
> The first problem is the error is misleading:
>
> [Wed Feb 22 13:24:52.013780 2023] [core:error] [pid 3047850:tid 3047899] (24)Too
many open files: [remote 192.168.0.139:53738] AH00132: file permissions deny server
access: /var/www/html/[filename]
>
> If you concentrate on the second part "file permissions deny server
> access" -- as I did -- then you'll be looking at file permissions,
> SELinux, restorecon, ausearch etc. That's a red herring, there is no
> permissions problem.
>
> The real error is the first part "Too many open files".
>
> It turns out that the default open file limit (1024!) is too low. To
> change this and fix the problem:
>
> # systemctl edit httpd
>
> This creates an "override" file to which you should add (or you could
> just create this file directly):
>
> # cat /etc/systemd/system/httpd.service.d/override.conf
> [Service]
> LimitNOFILE=65536
>
> and then restart Apache for the change to take effect.
>
> Why on earth Apache needs > 1024 open files to serve a dozen clients
> is not clear at all.
>
_______________________________________________
users mailing list -- users(a)lists.fedoraproject.org
To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
2:34 a.m.
New subject: open files limit was: Re: Enabling Apache HTTP/2 => Too many open
files error
On 2/27/23 13:44, Richard W.M. Jones wrote:
It turns out that the default open file limit (1024!) is too low.
To
change this and fix the problem:
# systemctl edit httpd
This low limit can be an issue for many processes.
I don't understand why it is still so low in modern machines.
I've raised it a lot, many years ago, by editing
/etc/security/limits.conf, but I now see I get again
1024. My guess is systemd is deciding by itself, and not
respecting my settings.
Regards.
--
Roberto Ragusa mail at robertoragusa.it
5:33 a.m.
New subject: open files limit was: Re: Enabling Apache HTTP/2 => Too many open
files error
On Thu, Mar 2, 2023 at 2:35 AM Roberto Ragusa <mail(a)robertoragusa.it> wrote:
On 2/27/23 13:44, Richard W.M. Jones wrote:
> It turns out that the default open file limit (1024!) is too low. To
> change this and fix the problem:
>
> # systemctl edit httpd
This low limit can be an issue for many processes.
I don't understand why it is still so low in modern machines.
I've raised it a lot, many years ago, by editing
/etc/security/limits.conf, but I now see I get again
1024. My guess is systemd is deciding by itself, and not
respecting my settings.
Regards.
Systemd is started really early and seems to use the defaults.
Anything started from systemd needs to explicitly override in the
startup script. Looking at the initramfs I don't see limits.conf in
there so when the switchroot happens and systemd gets started it would
have the default limits.
8:17 a.m.
New subject: open files limit was: Re: Enabling Apache HTTP/2 => Too many open files error
On Mar 2, 2023, at 03:35, Roberto Ragusa <mail(a)robertoragusa.it> wrote:
On 2/27/23 13:44, Richard W.M. Jones wrote:
> It turns out that the default open file limit (1024!) is too low. To
> change this and fix the problem:
> # systemctl edit httpd
This low limit can be an issue for many processes.
I don't understand why it is still so low in modern machines.
I've raised it a lot, many years ago, by editing
/etc/security/limits.conf, but I now see I get again
1024. My guess is systemd is deciding by itself, and not
respecting my settings.
/etc /security/limits.conf is only read by pam_limits.so in the PAM stack. Systemd
processes don’t automatically use pam (although if you use ‘su’ or login somehow).
You’d change it with the LimitsNOfile systemd directive. I think there’s a default setting
in /etc/systemd/system.conf you can override the default. (Double-check man
systemd.resource-control)
--
Jonathan Billings
414
days inactive
417
days old
6 comments
5 participants
participants (5)
-
Jeffrey Walton -
Jonathan Billings -
Richard W.M. Jones -
Roberto Ragusa -
Roger Heflin