IPA server upgrade fails with KDC error
by Johannes Brandstetter
Hi,
I'm trying to upgrade FreeIPA through ipa-server-upgrade from 4.4 to 4.5. The command fails with an "ACIError: Insufficient access:" . I find in the kdc log that it complains about " Database module does not match KDC version - while initializing database for realm..."
Does anybody know how to fix this?
Some more info:
$ cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
$ tail /var/log/krb5kdc.log
krb5kdc: Server error - while fetching master key K/M for realm XXX
krb5kdc: Database module does not match KDC version - while initializing database for realm XXX
$ sudo less /var/log/ipaupgrade.log
2017-10-16T13:04:13Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-10-16T13:04:13Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-10-16T13:04:13Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2017-10-16T13:04:13Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-10-16T13:04:13Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-10-16T13:04:13Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2017-10-16T13:04:13Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-10-16T13:04:13Z DEBUG duration: 0 seconds
2017-10-16T13:04:13Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2017-10-16T13:04:14Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run
server.upgrade()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1896, in upgrade
data_upgrade.create_instance()
File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 124, in create_instance
runtime=90)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
run_step(full_msg, method)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
method()
File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 96, in __start
api.Backend.ldap2.connect()
File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 66, in connect
conn = self.create_connection(*args, **kw)
File "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py", line 190, in create_connection
client_controls=clientctrls)
File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1111, in external_bind
'', auth_tokens, server_controls, client_controls)
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1007, in error_handler
raise errors.ACIError(info=info)
2017-10-16T13:04:14Z DEBUG The ipa-server-upgrade command failed, exception: ACIError: Insufficient access:
2017-10-16T13:04:14Z ERROR Insufficient access:
2017-10-16T13:04:14Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
$ sudo less /var/log/yum.log
Oct 16 05:36:02 Updated: ipa-common-4.5.0-21.el7.centos.1.2.noarch
Oct 16 05:36:02 Updated: ipa-client-common-4.5.0-21.el7.centos.1.2.noarch
Oct 16 05:36:25 Updated: libipa_hbac-1.15.2-50.el7_4.2.x86_64
Oct 16 05:36:53 Updated: python-libipa_hbac-1.15.2-50.el7_4.2.x86_64
Oct 16 05:36:55 Updated: python2-ipalib-4.5.0-21.el7.centos.1.2.noarch
Oct 16 05:36:55 Updated: python2-ipaclient-4.5.0-21.el7.centos.1.2.noarch
Oct 16 05:37:23 Updated: ipa-python-compat-4.5.0-21.el7.centos.1.2.noarch
Oct 16 05:38:43 Updated: ipa-server-common-4.5.0-21.el7.centos.1.2.noarch
Oct 16 05:38:44 Updated: python2-ipaserver-4.5.0-21.el7.centos.1.2.noarch
Oct 16 05:38:44 Updated: sssd-ipa-1.15.2-50.el7_4.2.x86_64
Oct 16 05:39:01 Installed: ipa-client-4.5.0-21.el7.centos.1.2.x86_64
Oct 16 05:39:28 Updated: ipsilon-tools-ipa-2.0.2-5.el7.centos.noarch
Oct 16 05:39:29 Updated: ipa-server-4.5.0-21.el7.centos.1.2.x86_64
Oct 16 05:40:48 Erased: ipa-admintools-4.4.0-14.el7.centos.7.noarch
Oct 16 05:19:30 Updated: krb5-libs-1.15.1-8.el7.x86_64
Oct 16 05:19:30 Updated: krb5-workstation-1.15.1-8.el7.x86_64
Oct 16 05:19:31 Updated: krb5-server-1.15.1-8.el7.x86_64
Oct 16 05:19:31 Updated: krb5-pkinit-1.15.1-8.el7.x86_64
Oct 16 05:38:22 Updated: sssd-krb5-common-1.15.2-50.el7_4.2.x86_64
Oct 16 05:38:57 Updated: sssd-krb5-1.15.2-50.el7_4.2.x86_64
Cheers,
Johannes
5 years, 5 months
using freeipa with an AWS elastic load balancer
by ridha.zorgui＠infor.com
I set up a FreeIPA master and replica behind an elastic load balancer in AWS cloud. FreeIPA Clients will be contacting the replica and the master sever through the load balancer so the dns name used when configurting the clients is the ELB CNAME. The problem is when retreiving ldap data and during the authentication, the SSL handshake fails as the certificate sent back from the master or replica has a hostname different than the one used in the sssd ( the ELB CNAME). so the connection is terminated. There is a workaround which is the use reqcert=allow but this bring a security issue with a MITM attack. another solution i found is the use SAN. I was able to add the ELB DNS as a SAN in freeipa servers certificate. i made sure it is there by downloading the certificate and checking that the elb san exist but when testing it the same problem remain. Please help.
5 years, 6 months
Can't install CA from replica file - Failed to import EncryptedPrivateKeyInfo to token
by H. Frenzel
Hi,
I tried to install a CA to the 2nd master a replicafile which was
created on the 1st master (with self-signed CA), with fails with:
ipa : DEBUG stderr=TokenException: Failed to import
EncryptedPrivateKeyInfo to token: (-8152) The key does not support the
requested operation.
What could be wrong here? - Please find the detailed debug log of
ipa-ca-install as attachment.
Thx & b/r
H.
5 years, 6 months
Certificates renewing with the wrong Subject
by Roderick Johnstone
Hi
Our freeipa certificates need to be renewed due to passing their expiry
dates.
While some certificates have renewed ok, the ipaCert and
auditSigningCert are renewing but the new certificates have the wrong
Subject.
Environment is:
serverA (CRL, first, master) RHEL 7.3, ipa 4.4
serverB (replica) RHEL 7.3, ipa 4.4
serverC (replica) RHEL 7.4, ipa 4.5
Once there are renewed certificates with the wrong Subject present,
there are various problems with renewing the remaining certificates,
which I think might be related to the bad Subject:
1) When just ipaCert has the wrong subject no further renewals happen
2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
service will not start and no further renewals happen.
I've been round the following loop many times on ServerA, our first master:
1) Restore good certificates from backup
2) Put the clock back to a time when certificates are all valid
3) Resubmit certificates for renewal
Each time the ipaCert renews it has the same wrong Subject. The wrong
Subject includes the host name of one of our ipa client systems.
Each time the auditSigningCert renews it has the same wrong Subject but
a different subject to the ipaCert. The wrong Subject in this case
includes the host name of a system which has never been an ipa client,
but might have been added and removed with ipa host-add and ipa host-del
for testing something, a while ago.
As far as I can see, the "cert_subject" is set correctly in the file
/var/lib/certmonger/<request id> until the point at which the
certificate is actually renewed.
I'd be very grateful for some pointers as to which configuration options
and logs to check through to resolve this problem on our production system.
If its of any relevance we did change which server is the first master
some time ago.
Thanks
Roderick Johnstone
5 years, 8 months
Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
by Jokinen Eemeli
Hello all!
I have very similiar problem as this one:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
ipa-server-upgrade fails as below
--
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
CA did not start in 300.0s
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
--
And the log tells me that CA returns status 500
--
DEBUG Waiting for CA to start...
DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus<http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus>
DEBUG request body ''
DEBUG response status 500
DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 15 Jun 2018 10:05:29 GMT
Connection: close
DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
DEBUG Waiting for CA to start...
ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 48, in run
raise admintool.ScriptError(str(e))
The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
ERROR CA did not start in 300.0s
--
With command "ipactl start --ignore-service-failures" I can start all the services but pki-tomcatd.
--
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
--
Suggested resolution to above problem doesn't help me since the LDAP and NSS DB seem to have same certificates (some difference in wrapping but the string is same if I take out the line breaks) and even the serial number matches.
--
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDjD...
...Prh2G
-----END CERTIFICATE-----
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' |grep Serial
Serial Number: 4 (0x4)
ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDjD...
...Prh2
G
description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem,
O=<<REALM>>
seeAlso: CN=CA Subsystem,O=<<REALM>>
--
And here's where my actual knowledge of things end. I've been trying to figure out all kind of logs (tomcat, Kerberos, directory server, ...) but haven't found a solid reason for it. I'm starting to believe this is a certificate issue, because although "getcert list" tells me that the certificate status is "Monitoring" on all certificates the expiry date is already in the past (current date 20.6.2018, certificate expiry 21.03.2018) on 4 certificates and it won't update even if I resubmit it or delete certificate and manually redo it (it got the same date as the "old ones"). The "main certs" ("caSigningCert cert-pki-ca", "Server-Cert cert-pki-ca" and two directory server certs) are valid for years (until 2020+).
--
Request ID '20160331084234':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=OCSP Subsystem,O=<<REALM>>
expires: 2018-03-21 09:42:04 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160331085008':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>
expires: 2020-03-04 09:58:23 UTC
principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>>
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
--
Has anyone else bumped into same kind of issues? Any ideas where I should continue looking? I'm starting to run out of ideas...
Eemeli Jokinen
5 years, 8 months
/etc/httpd/alias not getting renewed cert
by Thomas Letherby
Hello all,
I had an issue a short while ago with a replica which turned out to be an
expired certificate which I renewed and all seemed good.
Seemed...
It now appears that although the certificate renewed as seen by getcert
-list, it didn't update /etc/httpd/alias and so the httpd and tomcat-pki
services won't start unless I set the date to before the certificate
expired, and even then sometimes the httpd error_log shows:
Unable to verify certificate 'Server-Cert'. Add "NSSEnforceValidCerts off"
to nss.conf so the server can start until the problem can be resolved.
and the service fails to start.
I've tried resubmitting the certificate, and it doesn't seem to throw an
error, but it doesn't update /alias either.
Trying to access the server via the web page shows the old certificate
still in use.
I see the same certificate error with the replica server, which was freshly
rebuilt and added last week.
I've doubtless dug further into the hole trying to troubleshoot this, so I
probably need to start from the beginning again, and a pointer in the right
direction would be a great help!
A getcert list shows all the certificates expiry dates well into the future.
How can I get the certs back in sync? I've found a few guides and most seem
to be for earlier versions, and I'm not sure if they're still current.
I can post whatever logs you think will help, I'm afraid I'm not familiar
enough with them all to tell which are the most relevant. Is there a guide
for the logs?
Thanks for any help you can give,
Thomas
5 years, 8 months
Getting Synology NAS to play nice with FreeIPA
by Kristian Petersen
I have a synology NAS which hosts some SMB shares on my network. I would
like to be able to use FreeIPA as the LDAP provider it checks against for
authenticating these shares. I have a system user that I created in
FreeIPA for this purpose.
I configured the NAS to connect to my FreeIPA server for LDAP, but I get a
message about a failure to access some users NT passwords and how the Samba
service may not work for these users. It also says it could be either a
lack of NT passwords for the users or insufficient privileges to access
them. After chatting with Synology support they wanted me to enable CIFS
plaintext password authentication. However, if I select that option it
given me a warning about the share not being able to be the remote mount
target of CIFS anymore due to SMB being set to v1 only and disabling the
SMB related Bonjour service. If the system user doesn't have the needed
privileges, how can I fix that since I can't enroll the NAS?
--
Kristian Petersen
System Administrator
BYU Dept. of Chemistry and Biochemistry
5 years, 9 months
"No valid Negotiate header in server response" error when trying to install
by greg＠greg-gilbert.com
Hi all,
I've been having an issue recently where my servers can't install
FreeIPA due to this error:
Cannot connect to the server due to generic error: error marshalling
data for XML-RPC transport: message: need a <type 'unicode'>; got 'No
valid Negotiate header in server response' (a <type 'str'>)
Installation failed. Rolling back changes.
Unenrolling client from IPA server
Restarting FreeIPA (ipactl stop, ipactl start) solves the solution, but
not permanently.
I captured the FreeIPA logs during that event, which you can see here:
https://pastebin.com/WL7Cg90V
Any ideas what's going wrong? One solution would be to have a cron
restart FreeIPA nightly, but that's not ideal.
Thanks,
Greg
5 years, 9 months
