On Tue, 7 Feb 2023 10:35:35 +0100
Florence Blanc-Renaud <flo(a)redhat.com> wrote:
Hi,
On Tue, Feb 7, 2023 at 1:28 AM Jernej Jakob via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org> wrote:
> Hi David. I had the same issue here and found your writeup to be very
> helpful.
>
> I used more or less the same ldap actions to delete the certificates
> and requests (~3.6k) from LDAP. This did make 'ipa cert-find' display
> just the one "used"/"correct" certificate for the host, but the
main
> issue is not fixed. The webUI still displays all the old certificates
> that I have deleted from LDAP. Opening the "Hosts" tab or a host page
> takes very long, around 1-2 minutes.
>
> So I want to know what else needs to be done to make the webUI "forget"
> about the wrongly issued certificates?
>
> Where does the webUI get its list of certificates?
> I did some searching through the code and could only find the JS
> code that invokes a RPC call. But I could not find the code that
> handles that call.
>
> The webui is making a call equivalent to "ipa cert-find" which is handled
by the following code:
https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/cert.py#...
The call looks for certificates in multiple locations:
- in the subtree "ou=certificateRepository,ou=ca,o=ipaca"
- in the suffix "dc=example,dc=com", in the users/hosts/services entries
You cleaned the certificates from the cert repository but there may be many
entries (users/hosts/services) containing a userCertificate attribute. To
avoid seeing those certs you would have to delete the corresponding
userCertificate values.
HTH,
flo
Thanks flo, this was exactly what I needed, I managed to delete them
from userCertificate under
"fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com"
and now the webui is fast again!
In retrospect I think making the webui work fast even with such a large
number of certificates would be better, since this cleanup is not easy
and it's possible to delete too much. That would probably mean adding a
pager and filtering to the certificate list which is displayed on the
host page. Right now they are all displayed at once on one page and
there is no way to filter out e.g. all revoked certificates. Also find
and fix why loading the hosts list takes so long even though it doesn't
display any certificates on that page.
Here are the commands to delete the certificates from userCertificate
of the host. Add them to those in the previous message, after o=ipaca
cleaning but before changelog purging.
First save a list of all current userCertificate entries to a file
ldapsearch -LLL -x -D "cn=directory manager" -W -o ldif-wrap=no
-b "fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com"
userCertificate > userCertificate_badhost.ldif
Copy the certificate you want to keep from where it's stored on the
host to where you are working. Put it in badhost.crt in PEM format.
Create a new ldif without that certificate.
sed -e '/^dn: .*$/d'
-e "s#userCertificate:: $(cat badhost.crt|grep -v -e '-----'|tr -d
'\n')##"
userCertificate_badhost.ldif > userCertificate_badhost_to_delete.ldif
You can delete other certificates that you also want to keep from this
ldif. If you know the serial numbers of them, you can construct a shell
one-liner script with a loop that iterates over each certificate in each
line of the file, uses openssl x509 to retrieve its serial from
base64-encoded certificate and deletes the certificate from the file if
it matches.
I didn't need that so I don't have the script.
Format the ldif so it can be consumed by ldapmodify. Note that you have
to remove the newlines that the mailing list adds so that everything is
on one line, especially printf's format strings!
{ printf 'dn: fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com\n
changetype: modify\ndelete: userCertificate\n';
while read -r; do if [ -z "$REPLY" ]; then continue; fi;
printf '%s\n' "$REPLY"; done <
userCertificate_badhost_to_delete.ldif;
} > userCertificate_badhost_to_delete_ldapmodify.ldif
Test if the ldif is correct
ldapmodify -nv -x -D "cn=directory manager" -W
-f userCertificate_badhost_to_delete_ldapmodify.ldif
echo $? should say 0, if not, there's an error in the ldif you need to
correct.
Then remove the -nv and run the ldapmodify again to delete the entries
for real.
>
> IIRC my issue that caused certmonger to request a certificate over and
> over was caused by a bug after upgrading a client from Ubuntu 14.04 to
> 16.04. The path to ca_external_helper changed but it was not changed in
> /var/lib/certmonger/cas/* which caused certmonger to fail running
> ca_external_helper. To fix it I did:
>
> sed -i -e 's#x86_64-linux-gnu/##g; s#certmonger/certmonger#certmonger#g'
> /var/lib/certmonger/cas/*
>
>
> Below is the exact procedure I used to delete the certificates from
> LDAP.
>
> First fix the issue that caused the issuing of too many certificates.
> Make sure it successfully issued and saved the cert on the client and
> that it's in status "MONITORING", "stuck: no".
> Find the serial number of the cert currently present on the client.
> 'sudo getcert list', look at "certificate:" in my case it was in
> "/etc/ssl/private/hostname-ipa-cert.crt"
> openssl x509 -in /etc/ssl/private/hostname-ipa-cert.crt -noout -text
> In my case it was 268369940.
>
> Used the following shell script to revoke all the certificates with
> serial not matching. I did this before I knew howo to get the cert
> serials from ldap so it uses ipa cert-find. It's a slow process.
> for s in $(ipa cert-find --hosts=badhost --pkey-only --sizelimit=0|awk
> '/Serial number/{print $3 ;}'); do if [ "$s" =
"268369940" ] || [ -z "$s"
> ]; then continue; fi; echo "revoking $s"; ipa cert-revoke "$s";
done
>
> You can view all the revoked cert cn's with this command before
> deleting them.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(certStatus=REVOKED))' dn
certStatus|less
>
> Make a list of all cert cn's not matching the used cert, save output
> into a file, ready to be read by ldapdelete later.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(!(cn=268369940)))' | grep -o
'cn=.*' >
> cert_to_delete_not_used_badhost
>
> Make a list of all the requestId for all the certs to be deleted.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(!(cn=268369940)))' metaInfo|grep
-oP
> 'requestId:\K.*' > cert_request_to_delete_not_used_from_metaInfo_badhost
>
> In my case there were a couple more requests than issued certs, I'm not
> sure why. I made a list of all requests for this host excluding the
> requestId of the correct cert. First find the correct/used cert
> requestId. In my case it was 9990026.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(cn=268369940))' metaInfo|grep -oP
> 'requestId:\K.*'
>
> Then get a list of all requests for that host, excluding that one
> requestId.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=ca,ou=requests,o=ipaca"
>
'(&(extdata-req--005fsubject--005fname--002ecn=badhost)(!(cn=9990026)))'
> dn|grep -o 'cn=.*' > cert_request_to_delete_not_used_badhost
>
> Count the number of certs/requests from the previous operations. The
> first two must match, the 3rd shows how many extra requests there are.
> wc -l cert_to_delete_not_used_badhost
> cert_request_to_delete_not_used_from_metaInfo_badhost
> cert_request_to_delete_not_used_badhost
> 3982 cert_to_delete_not_used_badhost
> 3982 cert_request_to_delete_not_used_from_metaInfo_badhost
> 3990 cert_request_to_delete_not_used_badhost
>
> So there are 8 extra requests without corresponding certs. I chose to
> ignore them for now. They're probably fine to delete in the future.
>
> Before deleting the requests, make a file in a format ldapdelete
> expects.
> while read -r; do printf 'cn=%s,ou=ca,ou=requests,o=ipaca\n'
"$REPLY";
> done < cert_request_to_delete_not_used_from_metaInfo_badhost >
> cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
>
> Now the actual deletion steps.
> Delete the certs.
> ldapdelete -x -D "cn=directory manager" -W -f
> cert_to_delete_not_used_badhost
>
> Delete the requests.
> ldapdelete -x -D "cn=directory manager" -W -f
> cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
>
> (you can add '-nv' to test ldapdelete)
>
>
> After this, I decided to trim the changelog and tombstones from ldap.
> Roughly followed the info from
>
https://www.port389.org/docs/389ds/FAQ/changelog-trimming.html
> Do the steps on all servers.
>
>
> cat > changelog_short.ldif <<-'EOF'
> dn: cn=changelog5,cn=config
> changetype: modify
> replace: nsslapd-changelogmaxage
> nsslapd-changelogmaxage: 300
> -
> replace: nsslapd-changelogcompactdb-interval
> nsslapd-changelogcompactdb-interval: 300
> -
> replace: nsslapd-changelogtrim-interval
> nsslapd-changelogtrim-interval: 30
> -
>
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5ReplicaPurgeDelay
> nsds5ReplicaPurgeDelay: 300
> -
> replace: nsds5ReplicaTombstonePurgeInterval
> nsds5ReplicaTombstonePurgeInterval: 300
> -
> 'EOF'
>
>
> cat > changelog_normal.ldif <<-'EOF'
> dn: cn=changelog5,cn=config
> changetype: modify
> replace: nsslapd-changelogmaxage
> nsslapd-changelogmaxage: 7d
> -
> delete: nsslapd-changelogcompactdb-interval
> -
> delete: nsslapd-changelogtrim-interval
> -
>
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5ReplicaPurgeDelay
> nsds5ReplicaPurgeDelay: 604800
> -
> replace: nsds5ReplicaTombstonePurgeInterval
> nsds5ReplicaTombstonePurgeInterval: 86400
> -
> 'EOF'
>
>
> ldapmodify -x -D "cn=directory manager" -W -f changelog_short.ldif
> systemctl restart dirsrv(a)EXAMPLE-COM.service
>
> Make a change in the webUI or ipa cli. There must be a write/change
> operation in LDAP.
> Wait >5 minutes. Monitor /var/log/dirsrv/slapd-EXAMPLE-COM/errors on all
> servers, there should be no errors.
>
> ldapmodify -x -D "cn=directory manager" -W -f changelog_normal.ldif
> systemctl restart dirsrv(a)EXAMPLE-COM.service
>
> Before the purge /var/lib/dirsrv/slapd-EXAMPLE-COM was 360M, after the
> purge 295M. This was only on the server I executed the deletions on,
> not on any replicas. So not a drastic difference.
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
>
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
> Do not reply to spam, report it:
>
https://pagure.io/fedora-infrastructure/new_issue
>