Dear ipa-users,
I've recently observed a pattern where adding a host certificate to a host only shows the association in the GUI for the server which issues the cert. I'm running FreeIPA 4.4.4.
I request a certificate from the host(s) in question with something like:
ipa-getcert request -f /path -k /path -r
All IPA servers show the cert as being issued and valid on the certificates page. Visiting the "https://myserver/ipa/ui/#/e/host/details/hostame.fqdn shows a host certificate from the machine that issued the cert Visiting the same host page from other ipa servers does not show the host cert associated. Users and hosts continue to synchronise, as do other cert details!
I can manually associate the host to cert on other servers using the "add" button in the Host certifcate section of the host page, but this feels wrong. Any ideas on how to troubleshoot this? It feels like the CAs don't quite get which one is in charge, and could be a result of me tearing down the original ubuntu based ones to replace with fedora, or a mistake I have made whilst doing so.
Any advice appreciated,
David
David Harvey via FreeIPA-users wrote:
Dear ipa-users,
I've recently observed a pattern where adding a host certificate to a host only shows the association in the GUI for the server which issues the cert. I'm running FreeIPA 4.4.4.
I request a certificate from the host(s) in question with something like:
ipa-getcert request -f /path -k /path -r
All IPA servers show the cert as being issued and valid on the certificates page. Visiting the "https://myserver/ipa/ui/#/e/host/details/hostame.fqdn shows a host certificate from the machine that issued the cert Visiting the same host page from other ipa servers does not show the host cert associated. Users and hosts continue to synchronise, as do other cert details!
I can manually associate the host to cert on other servers using the "add" button in the Host certifcate section of the host page, but this feels wrong. Any ideas on how to troubleshoot this? It feels like the CAs don't quite get which one is in charge, and could be a result of me tearing down the original ubuntu based ones to replace with fedora, or a mistake I have made whilst doing so.
I'd still check for replication issues.
Are you sure the host entries in LDAP are the same between the different masters?
Can you look in /var/log/httpd/error_log to see if anything is being logged when the certificate is not showing?
rob
Thanks for your swift response Rob,
My apologies, it looks like my superficial replication check was insufficient.
ipa-replica-manage -v list ipa2.mydom ipa3.mydom: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2018-02-01 11:47:10+00:00 ipa1.mydom: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (18) Replication error acquiring replica: Incremental update transient error. Backing off, will retry update later. (transient error) last update ended: 1970-01-01 00:00:00+00:00
Which led me to check on the snowflake where I'm seeing
Feb 1 11:48:49 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:49.866140639 +0000] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meToipa1.mydom" (ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized. Feb 1 11:48:52 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:52.916537089 +0000] - ERR - agmt="cn=meToipa1.mydom" (ipa1:389) - clcache_load_buffer - Can't locate CSN 5a687250000500100000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. Feb 1 11:48:52 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:52.919314318 +0000] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meToipa1.mydom" (ipa1:389): CSN 5a687250000500100000 not found, we aren't as up to date, or we purged Feb 1 11:48:52 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:52.922208937 +0000] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meToipa1.mydom" (ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized. Feb 1 11:48:55 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:55.956362678 +0000] - ERR - agmt="cn=meToipa1.mydom" (ipa1:389) - clcache_load_buffer - Can't locate CSN 5a687250000500100000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. Feb 1 11:48:55 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:55.959110311 +0000] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meToipa1.mydom" (ipa1:389): CSN 5a687250000500100000 not found, we aren't as up to date, or we purged Feb 1 11:48:55 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:55.961578933 +0000] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meToipa1.mydom" (ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
The only obvious error (which I suspect is unrelated) I could spot in http land was:
[Thu Feb 01 10:40:39.686959 2018] [wsgi:error] [pid 7302:tid 140268792428288] [remote 10.70.64.26:57792] ipa: ERROR: plugin index generation failed: Supplied plugin directory path is not a directory
I'll aim to reinitialise the problem box based on this. Without wanting to make excuses for my ineptitude, are there any plans to increase visibility for replication issues to surface them more obviously?
Thank you so much for your guidance, hugely appreciated.
David
On 31 January 2018 at 21:48, Rob Crittenden rcritten@redhat.com wrote:
David Harvey via FreeIPA-users wrote:
Dear ipa-users,
I've recently observed a pattern where adding a host certificate to a host only shows the association in the GUI for the server which issues the cert. I'm running FreeIPA 4.4.4.
I request a certificate from the host(s) in question with something like:
ipa-getcert request -f /path -k /path -r
All IPA servers show the cert as being issued and valid on the certificates page. Visiting the "https://myserver/ipa/ui/#/e/host/details/hostame.fqdn shows a host certificate from the machine that issued the cert Visiting the same host page from other ipa servers does not show the host cert associated. Users and hosts continue to synchronise, as do other cert details!
I can manually associate the host to cert on other servers using the "add" button in the Host certifcate section of the host page, but this feels wrong. Any ideas on how to troubleshoot this? It feels like the CAs don't quite get which one is in charge, and could be a result of me tearing down the original ubuntu based ones to replace with fedora, or a mistake I have made whilst doing so.
I'd still check for replication issues.
Are you sure the host entries in LDAP are the same between the different masters?
Can you look in /var/log/httpd/error_log to see if anything is being logged when the certificate is not showing?
rob
Initial impression having re-initialised looks encouraging. I didn't have a guarantee reproducible steps, so will keep an eye on it, but the errors are no more, and associating a cert on one master was reflected on another. \o/ Thanks again,
David
On 1 February 2018 at 11:57, David Harvey davidcharvey@googlemail.com wrote:
Thanks for your swift response Rob,
My apologies, it looks like my superficial replication check was insufficient.
ipa-replica-manage -v list ipa2.mydom ipa3.mydom: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2018-02-01 11:47:10+00:00 ipa1.mydom: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (18) Replication error acquiring replica: Incremental update transient error. Backing off, will retry update later. (transient error) last update ended: 1970-01-01 00:00:00+00:00
Which led me to check on the snowflake where I'm seeing
Feb 1 11:48:49 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:49.866140639 +0000] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meToipa1. mydom" (ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized. Feb 1 11:48:52 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:52.916537089 +0000] - ERR - agmt="cn=meToipa1.mydom" (ipa1:389) - clcache_load_buffer - Can't locate CSN 5a687250000500100000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. Feb 1 11:48:52 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:52.919314318 +0000] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meToipa1.mydom" (ipa1:389): CSN 5a687250000500100000 not found, we aren't as up to date, or we purged Feb 1 11:48:52 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:52.922208937 +0000] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meToipa1. mydom" (ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized. Feb 1 11:48:55 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:55.956362678 +0000] - ERR - agmt="cn=meToipa1.mydom" (ipa1:389) - clcache_load_buffer
- Can't locate CSN 5a687250000500100000 in the changelog (DB rc=-30988). If
replication stops, the consumer may need to be reinitialized. Feb 1 11:48:55 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:55.959110311 +0000] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meToipa1.mydom" (ipa1:389): CSN 5a687250000500100000 not found, we aren't as up to date, or we purged Feb 1 11:48:55 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:55.961578933 +0000] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meToipa1. mydom" (ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
The only obvious error (which I suspect is unrelated) I could spot in http land was:
[Thu Feb 01 10:40:39.686959 2018] [wsgi:error] [pid 7302:tid 140268792428288] [remote 10.70.64.26:57792] ipa: ERROR: plugin index generation failed: Supplied plugin directory path is not a directory
I'll aim to reinitialise the problem box based on this. Without wanting to make excuses for my ineptitude, are there any plans to increase visibility for replication issues to surface them more obviously?
Thank you so much for your guidance, hugely appreciated.
David
On 31 January 2018 at 21:48, Rob Crittenden rcritten@redhat.com wrote:
David Harvey via FreeIPA-users wrote:
Dear ipa-users,
I've recently observed a pattern where adding a host certificate to a host only shows the association in the GUI for the server which issues the cert. I'm running FreeIPA 4.4.4.
I request a certificate from the host(s) in question with something
like:
ipa-getcert request -f /path -k /path -r
All IPA servers show the cert as being issued and valid on the certificates page. Visiting the "https://myserver/ipa/ui/#/e/host/details/hostame.fqdn shows a host certificate from the machine that issued the cert Visiting the same host page from other ipa servers does not show the host cert associated. Users and hosts continue to synchronise, as do other cert details!
I can manually associate the host to cert on other servers using the "add" button in the Host certifcate section of the host page, but this feels wrong. Any ideas on how to troubleshoot this? It feels like the CAs don't quite get which one is in charge, and could be a result of me tearing down the original ubuntu based ones to replace with fedora, or a mistake I have made whilst doing so.
I'd still check for replication issues.
Are you sure the host entries in LDAP are the same between the different masters?
Can you look in /var/log/httpd/error_log to see if anything is being logged when the certificate is not showing?
rob
freeipa-users@lists.fedorahosted.org
-
David Harvey -
Rob Crittenden