On 04/01/2021 17:46, Alexander Bokovoy wrote:

On ma, 04 tammi 2021, lejeczek via FreeIPA-users wrote:

...

Hi guys.

I'm trying to setup a first master during which I get: ...   [7/7]: configuring ipa-dnskeysyncd to start on boot Done configuring DNS key synchronization service (ipa-dnskeysyncd). Restarting ipa-dnskeysyncd Restarting named Named service failed to start (CalledProcessError(Command ['/bin/systemctl', 'restart', 'named-pkcs11.service'] returned non-zero exit status 1: 'Job for named-pkcs11.service failed because a timeout was exceeded.\nSee "systemctl status named-pkcs11.service" and "journalctl -xe" for details.\n')) ...

and that is the only error from the setup which seemingly continues and completes successfully:

... Using existing certificate '/etc/ipa/ca.crt'. Client hostname: c8kubermaster1.private.openshift.c8 Realm: PRIVATE.OPENSHIFT.C8 DNS Domain: private.openshift.c8 IPA Server: c8kubermaster1.private.openshift.c8 BaseDN: dc=private,dc=openshift,dc=c8

Configured sudoers in /etc/authselect/user-nsswitch.conf Configured /etc/sssd/sssd.conf Systemwide CA database updated. Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub Could not update DNS SSHFP records. SSSD enabled Configured /etc/openldap/ldap.conf Configured /etc/ssh/ssh_config Configured /etc/ssh/sshd_config Configuring private.openshift.c8 as NIS domain. Client configuration complete. The ipa-client-install command was successful

DNS query for c8kubermaster1.private.openshift.c8. 1 failed: The DNS operation timed out after 30.000322580337524 seconds unable to resolve host name c8kubermaster1.private.openshift.c8. to IP address, ipa-ca DNS record will be incomplete ==============================================================================

Setup complete

Next steps:     1. You must make sure these network ports are open:         TCP Ports:           * 80, 443: HTTP/HTTPS           * 389, 636: LDAP/LDAPS           * 88, 464: kerberos           * 53: bind         UDP Ports:           * 88, 464: kerberos           * 53: bind           * 123: ntp

    2. You can now obtain a kerberos ticket using the command: 'kinit admin'        This ticket will allow you to use the IPA tools (e.g., ipa user-add)        and the web user interface.

Be sure to back up the CA certificates stored in /root/cacert.p12 These files are required to create replicas. The password for these files is the Directory Manager password The ipa-server-install command was successful

Yet, very first reboot and ipa.service fails to start, but before that reboot if I -> $ systemctl restart named-pkcs11.service I takes rather long 10 or so secons and journal shows ... LDAP configuration synchronization failed: socket is not connected ... but socket is there: /var/run/slapd-PRIVATE-OPENSHIFT-C8.socket More from named's journal: ... esolver priming query complete LDAP error: Can't contact LDAP server: ldap_sync_poll() failed ldap_syncrepl will reconnect in 60 seconds GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 2 successfully reconnected to LDAP server LDAP configuration for instance 'ipa' synchronized GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 2 LDAP data for instance 'ipa' are being synchronized, please ignore message 'all zones loaded'

Is it named-pkcs11 looking for wrong bits or something not good with dirsrv or .. maybe something else... would you anybody know?

It looks like 389-ds LDAP server start takes quite some time before it starts listening on the LDAPI socket, so first attempt to connect by bind-dyndb-ldap driver fails.

Not sure it is a problem with the amount of resources available on this system (a VM? a container? How many CPUs available? Disk is slow? etc). Eventually bind-dyndb-ldap succeeds on re-try so it might only be a too fast parallel startup for some components and not the other.

It's a VM but I think resource are sufficient - 6GB and dual 2.5GHz Haswell vcpu - and there is nothing else running on the system except for 'named'. It's been a while since last tries in that KVM and now with many updates I cannot say if these below are new or I saw them before: ...   [22/28]: enabling CA instance   [23/28]: migrating certificate profiles to LDAP Profile 'acmeServerCert' is already in LDAP and enabled; skipping Profile 'caCMCserverCert' is already in LDAP and enabled; skipping Profile 'caCMCECserverCert' is already in LDAP and enabled; skipping ... Profile 'caTokenUserDelegateSigningKeyEnrollment' is already in LDAP and enabled; skipping   [24/28]: importing IPA certificate profiles

I shall mention it's Centos Stream with: ipa-server-4.9.0-1.module_el8.4.0+639+a88aab78.x86_64 389-ds-base-1.4.3.16-8.module_el8.4.0+644+ed25d39e.x86_64

But again, installation continues till the finish with: The ipa-server-install command was successful

If this happens later, when 389-ds already started, there might be another issue -- if it is really slow to respond on LDAPI connection, may be LDAP server was already swapped out by something more memory hungry? In general, it would be good to look at the overall picture with the workload you have on that system.

On 04/01/2021 17:46, Alexander Bokovoy wrote:

On ma, 04 tammi 2021, lejeczek via FreeIPA-users wrote:

...

Hi guys.

I'm trying to setup a first master during which I get: ...   [7/7]: configuring ipa-dnskeysyncd to start on boot Done configuring DNS key synchronization service (ipa-dnskeysyncd). Restarting ipa-dnskeysyncd Restarting named Named service failed to start (CalledProcessError(Command ['/bin/systemctl', 'restart', 'named-pkcs11.service'] returned non-zero exit status 1: 'Job for named-pkcs11.service failed because a timeout was exceeded.\nSee "systemctl status named-pkcs11.service" and "journalctl -xe" for details.\n')) ...

and that is the only error from the setup which seemingly continues and completes successfully:

... Using existing certificate '/etc/ipa/ca.crt'. Client hostname: c8kubermaster1.private.openshift.c8 Realm: PRIVATE.OPENSHIFT.C8 DNS Domain: private.openshift.c8 IPA Server: c8kubermaster1.private.openshift.c8 BaseDN: dc=private,dc=openshift,dc=c8

Configured sudoers in /etc/authselect/user-nsswitch.conf Configured /etc/sssd/sssd.conf Systemwide CA database updated. Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub Could not update DNS SSHFP records. SSSD enabled Configured /etc/openldap/ldap.conf Configured /etc/ssh/ssh_config Configured /etc/ssh/sshd_config Configuring private.openshift.c8 as NIS domain. Client configuration complete. The ipa-client-install command was successful

DNS query for c8kubermaster1.private.openshift.c8. 1 failed: The DNS operation timed out after 30.000322580337524 seconds unable to resolve host name c8kubermaster1.private.openshift.c8. to IP address, ipa-ca DNS record will be incomplete ==============================================================================

Setup complete

Next steps:     1. You must make sure these network ports are open:         TCP Ports:           * 80, 443: HTTP/HTTPS           * 389, 636: LDAP/LDAPS           * 88, 464: kerberos           * 53: bind         UDP Ports:           * 88, 464: kerberos           * 53: bind           * 123: ntp

    2. You can now obtain a kerberos ticket using the command: 'kinit admin'        This ticket will allow you to use the IPA tools (e.g., ipa user-add)        and the web user interface.

Be sure to back up the CA certificates stored in /root/cacert.p12 These files are required to create replicas. The password for these files is the Directory Manager password The ipa-server-install command was successful

Yet, very first reboot and ipa.service fails to start, but before that reboot if I -> $ systemctl restart named-pkcs11.service I takes rather long 10 or so secons and journal shows ... LDAP configuration synchronization failed: socket is not connected ... but socket is there: /var/run/slapd-PRIVATE-OPENSHIFT-C8.socket More from named's journal: ... esolver priming query complete LDAP error: Can't contact LDAP server: ldap_sync_poll() failed ldap_syncrepl will reconnect in 60 seconds GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 2 successfully reconnected to LDAP server LDAP configuration for instance 'ipa' synchronized GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 2 LDAP data for instance 'ipa' are being synchronized, please ignore message 'all zones loaded'

Is it named-pkcs11 looking for wrong bits or something not good with dirsrv or .. maybe something else... would you anybody know?

It looks like 389-ds LDAP server start takes quite some time before it starts listening on the LDAPI socket, so first attempt to connect by bind-dyndb-ldap driver fails.

Not sure it is a problem with the amount of resources available on this system (a VM? a container? How many CPUs available? Disk is slow? etc). Eventually bind-dyndb-ldap succeeds on re-try so it might only be a too fast parallel startup for some components and not the other.

If this happens later, when 389-ds already started, there might be another issue -- if it is really slow to respond on LDAPI connection, may be LDAP server was already swapped out by something more memory hungry? In general, it would be good to look at the overall picture with the workload you have on that system.

I think the culprit here is ipa-server-4.9.0-1.module_el8.4.0+639+a88aab78.x86_64 ipa-server-4.8.7-13.module_el8.3.0+606+1e8766d7.x86_64 seems to be free from that problem. So guys - if you are on Centos Stream be aware - ipa-server-4.9.0 comes with centos-stream-repos-8-2.el8.noarch - before you add those repos to your setups.

many thanks, L.

On 15/02/2021 17:11, Alexander Bokovoy wrote:

On ma, 15 helmi 2021, lejeczek via FreeIPA-users wrote:

...

On 04/01/2021 17:46, Alexander Bokovoy wrote:

...

On ma, 04 tammi 2021, lejeczek via FreeIPA-users wrote:

...

Hi guys.

I'm trying to setup a first master during which I get: ...   [7/7]: configuring ipa-dnskeysyncd to start on boot Done configuring DNS key synchronization service (ipa-dnskeysyncd). Restarting ipa-dnskeysyncd Restarting named Named service failed to start (CalledProcessError(Command ['/bin/systemctl', 'restart', 'named-pkcs11.service'] returned non-zero exit status 1: 'Job for named-pkcs11.service failed because a timeout was exceeded.\nSee "systemctl status named-pkcs11.service" and "journalctl -xe" for details.\n')) ...

and that is the only error from the setup which seemingly continues and completes successfully:

... Using existing certificate '/etc/ipa/ca.crt'. Client hostname: c8kubermaster1.private.openshift.c8 Realm: PRIVATE.OPENSHIFT.C8 DNS Domain: private.openshift.c8 IPA Server: c8kubermaster1.private.openshift.c8 BaseDN: dc=private,dc=openshift,dc=c8

Configured sudoers in /etc/authselect/user-nsswitch.conf Configured /etc/sssd/sssd.conf Systemwide CA database updated. Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub Could not update DNS SSHFP records. SSSD enabled Configured /etc/openldap/ldap.conf Configured /etc/ssh/ssh_config Configured /etc/ssh/sshd_config Configuring private.openshift.c8 as NIS domain. Client configuration complete. The ipa-client-install command was successful

DNS query for c8kubermaster1.private.openshift.c8. 1 failed: The DNS operation timed out after 30.000322580337524 seconds unable to resolve host name c8kubermaster1.private.openshift.c8. to IP address, ipa-ca DNS record will be incomplete ==============================================================================

Setup complete

Next steps:     1. You must make sure these network ports are open:         TCP Ports:           * 80, 443: HTTP/HTTPS           * 389, 636: LDAP/LDAPS           * 88, 464: kerberos           * 53: bind         UDP Ports:           * 88, 464: kerberos           * 53: bind           * 123: ntp

    2. You can now obtain a kerberos ticket using the command: 'kinit admin'        This ticket will allow you to use the IPA tools (e.g., ipa user-add)        and the web user interface.

Be sure to back up the CA certificates stored in /root/cacert.p12 These files are required to create replicas. The password for these files is the Directory Manager password The ipa-server-install command was successful

Yet, very first reboot and ipa.service fails to start, but before that reboot if I -> $ systemctl restart named-pkcs11.service I takes rather long 10 or so secons and journal shows ... LDAP configuration synchronization failed: socket is not connected ... but socket is there: /var/run/slapd-PRIVATE-OPENSHIFT-C8.socket More from named's journal: ... esolver priming query complete LDAP error: Can't contact LDAP server: ldap_sync_poll() failed ldap_syncrepl will reconnect in 60 seconds GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 2 successfully reconnected to LDAP server LDAP configuration for instance 'ipa' synchronized GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 1 GSSAPI client step 2 LDAP data for instance 'ipa' are being synchronized, please ignore message 'all zones loaded'

Is it named-pkcs11 looking for wrong bits or something not good with dirsrv or .. maybe something else... would you anybody know?

It looks like 389-ds LDAP server start takes quite some time before it starts listening on the LDAPI socket, so first attempt to connect by bind-dyndb-ldap driver fails.

Not sure it is a problem with the amount of resources available on this system (a VM? a container? How many CPUs available? Disk is slow? etc). Eventually bind-dyndb-ldap succeeds on re-try so it might only be a too fast parallel startup for some components and not the other.

If this happens later, when 389-ds already started, there might be another issue -- if it is really slow to respond on LDAPI connection, may be LDAP server was already swapped out by something more memory hungry? In general, it would be good to look at the overall picture with the workload you have on that system.

I think the culprit here is ipa-server-4.9.0-1.module_el8.4.0+639+a88aab78.x86_64 ipa-server-4.8.7-13.module_el8.3.0+606+1e8766d7.x86_64 seems to be free from that problem. So guys - if you are on Centos Stream be aware - ipa-server-4.9.0 comes with centos-stream-repos-8-2.el8.noarch - before you add those repos to your setups.

I do not see this problem at all, nor on CentOS 8 Stream with 4.9.0-1, nor on RHEL 8.4 nightlies with extensive set of tests.

If you'd show exact instructions how you reached this state, it would be easier to reproduce.

I do all this in a private cloud off openstack and there I hit all the limits where I cannot spin up more instances, for now I'm happy I can use IPA there but will see if I can re-investigate. But really, all I did was replace image of the instance so KVM in terms of resources available to os/IPA - out of the question as virtually identical. Only one thing I did with 4.9.0 before standing it up as its own master was, I tried to plug it in as a replica/master to already existing Centos 7's IPA. That how it started and with problems right away. Still it would boggled my mind how could such a failed attempt to setup replica crippled the system so badly that 'uninstall' of the 'deployment' and rpm packages (a number of time) was not update to clean & make server able to deploy anew as first master. Hmm..