Russell and I did a bit of offline troubleshooting but unfortunately
didn't find anything.
I'm cc'ing a couple of the PKI developers. They would know know how to
verify that the webapp(s) are registered properly and may be able to
tell us why we're seeing 404's.
rob
Russell Long wrote:
> With pki-tomcatd@pki-tomcat running, the ipa-cert-show gives a
> connection error:
>
> [root@ipa-primary ~]# openssl x509 -serial -noout -in /etc/ipa/ca.crt
> serial=01
> [root@ipa-primary ~]# ipa cert-show 01
> ipa: ERROR: Request failed with status 404: Non-2xx response from CA
> REST API: 404. (404)
>
> --Russ
>
> On Wed, Feb 5, 2025 at 2:06 PM Rob Crittenden <rcritten(a)redhat.com
> <mailto:rcritten@redhat.com>> wrote:
>
> Russell Long wrote:
> > Here is the obfuscated sosreport.
>
> It looks like the CA restart that happened immediately before the first
> 404 was successful. At least it doesn't report any errors beyond the
> usual LDAP connection failures at startup.
>
> The startup looks to be basically done around 2025-02-03 14:37:56
>
> The CA returned 404's at 2025-02-03T19:39:34Z in the upgrade log.
>
> We don't have the tomcat access logs in the sosreport so we can't see
> the requests but they would likely also report 404 and no other details.
>
> If you manually start things can you communicate with the CA?
>
> # ipactl --skip-version-check
>
> Does the CA start? If not add --ignore-service-failures
>
> Once everything else is up and settled, if the CA start failed run:
> systemctl restart pki-tomcatd@pki-tomcat
>
> And see if that is successful. I think it should succeed since it
> appears to have done so in the recent past.
>
> If so try a basic cert command:
> # openssl x509 -serial -noout -in /etc/ipa/ca.crt
> # ipa cert-show <that serial number>
>
> Does it give you data or a connection error?
>
> rob
>
> >
> >
> > On Wed, Feb 5, 2025 at 2:33 AM Alexander Bokovoy
> <abokovoy(a)redhat.com <mailto:abokovoy@redhat.com>
> > <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com>>> wrote:
> >
> > On Пан, 03 лют 2025, Russell Long via FreeIPA-users wrote:
> > >Here is the log, sorry for the delay. Logs are redacted, but the
> > only thing
> > >changed was the domain names and DNs.
> >
> > The upgrade log chokes on the CA application not being
> registered in
> > tomcat container (the corresponding /ca/rest/... path is
> giving 404
> > error).
> >
> > So we get back to the same point as before. An upgrade has been in
> > progress but somehow was interrupted. Directory server was
> having some
> > of listeners disabled to avoid external communication during
> the upgrade
> > and those listeners weren't recovered due to an interruption. You
> > recovered some of them but it looks like there is still
> something that
> > messes up.
> >
> > If you are saying all services are working fine, just the
> upgrade kicks
> > in every time 'ipactl restart' is run (which is part of
> ipa.service
> > machinery), it means the logged IPA data version is older than
> what IPA
> > sees in the RPM database. Temporarily, this can be fixed by
> looking at
> > /var/lib/ipa/sysupgrade/sysupgrade.state and changing
> ipa.data_version
> > value to be exact same as the RPM package version-release values.
> >
> > However, it would help to understand why an upgrade causes CA
> apps to
> > fail to register with the tomcat container. It looks like we
> have at
> > least three such cases on this list over past week or so, all
> on CentOS
> > 9 Stream, so there might be something?
> >
> > May be you can install sos report tool and collect a larger
> amount of
> > data altogether so that we can see a greater picture?
> >
> > # dnf install sos
> > # sos report
> --profile={identity,security,system,services,network}
> > --clean -a
> >
> > This should produce logs with consistently obfuscated
> hostnames and
> > domains across all files. You can add more domains to
> obfuscate with
> > `--domains={domain1,domain2,..}` to `sos report` tool.
> >
> > >
> > >
> > >
> > >On Wed, Jan 29, 2025 at 4:51 PM Rob Crittenden
> <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>> wrote:
> > >
> > >> Russ Long via FreeIPA-users wrote:
> > >> > Things are functional, however IPA still thinks it needs an
> > upgrade, so
> > >> any time the service restarts, it breaks again.
> > >> >
> > >>
> > >> If you have time to run the upgrade again and send us a
> compressed
> > >> /var/log/ipaupgrade.log we can see if we can identify the
> root cause.
> > >>
> > >> rob
> > >>
> > >>
> >
> >
> >
> >
> > --
> > / Alexander Bokovoy
> > Sr. Principal Software Engineer
> > Security / Identity Management Engineering
> > Red Hat Limited, Finland
> >
>