2:15 p.m.
Hello everyone,
We were using Freeipa on Fedora 24. And we are in the process to upgrade to
Fedora 28.
We have a cluster of 2 nodes (freeipa-01 and freeipa-02).
I am trying to upgrade one server after the other, from one release to the
next.
Basically:
freeipa-01 Fedora 24 -> Fedora 25
freeipa-02 Fedora 24 -> Fedora 25
freeipa-02 Fedora 25 -> Fedora 26
freeipa-01 Fedora 25 -> Fedora 26
freeipa-01 Fedora 26 -> Fedora 27
freeipa-02 Fedora 26 -> Fedora 27
freeipa-02 Fedora 27 -> Fedora 28
freeipa-01 Fedora 27 -> Fedora 28
Since Fedora doesn’t support to jump from one version to another, except
one release at the time.
My idea is to check that once a server is upgraded, then everything is
stable, before going to the next server, and try to be as near as possible
from a version point of view between the 2 freeipa node cluster.
Today <http://airmail.calendar/2018-06-12%2012:00:00%20CEST>;, I could
upgrade without problems from Fedora 24 -> Fedora 25 on both nodes
(freeipa-01 and freeipa-02).
In trying to upgrade to Fedora 26, I got some problems, the main problem is
that the upgrade of ldap 389 is not successful, and the one from IPA either.
After investigating a long moment, I have found that ns-slapd listen only
to IPv6, on UDP, and NOT on IPv4 and TCP.
Here is what I have:
[root@freeipa-02 lib]# lsof -Pni |grep slap
ns-slapd 21005 dirsrv 9u IPv6 1617283379 <//1617283379> 0t0
UDP *:389
ns-slapd 21005 dirsrv 77u IPv4 1617321218 <//1617321218> 0t0
TCP 10.100.0.102:60646->10.100.0.101:389 (ESTABLISHED)
ns-slapd 21005 dirsrv 81u IPv4 1617317640 <//1617317640> 0t0
TCP 10.100.0.102:60648->10.100.0.101:389 (ESTABLISHED)
So, I decided to look at the file dse.ldif, and found that the entry
"nsslapd-port” was set to “0” and no “nsslapd-listenhost” was not set at
all.
I have then added the line
nsslapd-listenhost: 0.0.0.0
and changed the nsslapd-port to look like:
nsslap-port: 389
And after doing a
systemctl stop dirsrv@DOM-LOCAL ; systemctl start dirsrv@DOM-LOCAL
No changes… all modification on my dse.ldif were gone.
I stopped again the dirsrv, did again my changes on dse.ldif, and run the
following command:
/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-DOM-LOCAL -i
/var/run/dirsrv/slapd-DOM-LOCAL.pid
and now, I have the following:
[root@freeipa-02 updates]# lsof -Pni |grep 389
ns-slapd 78507 dirsrv 10u IPv6 1681165214 <//1681165214> 0t0
UDP *:389
ns-slapd 78507 dirsrv 11u IPv4 1681165216 <//1681165216> 0t0
TCP *:389 (LISTEN)
ns-slapd 78507 dirsrv 114u IPv4 1684131928 <//1684131928> 0t0
TCP 10.100.0.102:389->10.100.0.110:36828 (ESTABLISHED)
So my questions are:
- how to change the dse.ldif file?
- Is there another way to ensure that the port that listen is TCP / 389 on
IPv4?
- Is there something that needs to be done between Fedora 25 and 26?
Knowing that I will go to Fedora 28, is there something that I need to be
aware of?
- Anything that can help me generally with my upgrade path?
Best regards,
Alessandro
2:22 p.m.
On Tue, 2018-06-12 at 12:15 -0700, Alessandro Perucchi via FreeIPA-
users wrote:
Hello everyone,
We were using Freeipa on Fedora 24. And we are in the process to upgrade to
Fedora 28.
We have a cluster of 2 nodes (freeipa-01 and freeipa-02).
I am trying to upgrade one server after the other, from one release to the
next.
Basically:
freeipa-01 Fedora 24 -> Fedora 25
freeipa-02 Fedora 24 -> Fedora 25
freeipa-02 Fedora 25 -> Fedora 26
freeipa-01 Fedora 25 -> Fedora 26
freeipa-01 Fedora 26 -> Fedora 27
freeipa-02 Fedora 26 -> Fedora 27
freeipa-02 Fedora 27 -> Fedora 28
freeipa-01 Fedora 27 -> Fedora 28
Since Fedora doesn’t support to jump from one version to another, except
one release at the time.
My idea is to check that once a server is upgraded, then everything is
stable, before going to the next server, and try to be as near as possible
from a version point of view between the 2 freeipa node cluster.
Today <http://airmail.calendar/2018-06-12%2012:00:00%20CEST>;, I could
upgrade without problems from Fedora 24 -> Fedora 25 on both nodes
(freeipa-01 and freeipa-02).
In trying to upgrade to Fedora 26, I got some problems, the main problem is
that the upgrade of ldap 389 is not successful, and the one from IPA either.
After investigating a long moment, I have found that ns-slapd listen only
to IPv6, on UDP, and NOT on IPv4 and TCP.
Here is what I have:
[root@freeipa-02 lib]# lsof -Pni |grep slap
ns-slapd 21005 dirsrv 9u IPv6 1617283379 <//1617283379> 0t0
UDP *:389
ns-slapd 21005 dirsrv 77u IPv4 1617321218 <//1617321218> 0t0
TCP 10.100.0.102:60646->10.100.0.101:389 (ESTABLISHED)
ns-slapd 21005 dirsrv 81u IPv4 1617317640 <//1617317640> 0t0
TCP 10.100.0.102:60648->10.100.0.101:389 (ESTABLISHED)
So, I decided to look at the file dse.ldif, and found that the entry
"nsslapd-port” was set to “0” and no “nsslapd-listenhost” was not set at
all.
I have then added the line
nsslapd-listenhost: 0.0.0.0
and changed the nsslapd-port to look like:
nsslap-port: 389
And after doing a
systemctl stop dirsrv@DOM-LOCAL ; systemctl start dirsrv@DOM-LOCAL
No changes… all modification on my dse.ldif were gone.
I stopped again the dirsrv, did again my changes on dse.ldif, and run the
following command:
/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-DOM-LOCAL -i
/var/run/dirsrv/slapd-DOM-LOCAL.pid
and now, I have the following:
[root@freeipa-02 updates]# lsof -Pni |grep 389
ns-slapd 78507 dirsrv 10u IPv6 1681165214 <//1681165214> 0t0
UDP *:389
ns-slapd 78507 dirsrv 11u IPv4 1681165216 <//1681165216> 0t0
TCP *:389 (LISTEN)
ns-slapd 78507 dirsrv 114u IPv4 1684131928 <//1684131928> 0t0
TCP 10.100.0.102:389->10.100.0.110:36828 (ESTABLISHED)
So my questions are:
- how to change the dse.ldif file?
You have to stop ns-slapd before changing the file.
- Is there another way to ensure that the port that listen is TCP /
389 on
IPv4?
The port was disabled during some upgrade operations, your situation
meant some upgrade failed and that old version failed to set back the
port in dse.ldif
This is a bug and shouldn't happen in recent versions.
- Is there something that needs to be done between Fedora 25 and 26?
Is this upgrade bug repeatable ? (keep in mind that F26 is practically
EOL)
Knowing that I will go to Fedora 28, is there something that I need
to be
aware of?
Yes, read this list archives before you attempt F28 upgrades, you may
have to use updates-testing as the GA bits where busted wrt replication
for upgrades.
- Anything that can help me generally with my upgrade path?
In general your approach is ok, make backups :-)
Simo.
--
Simo Sorce
Sr. Principal Software Engineer
Red Hat, Inc
2:39 p.m.
On 12 June 2018 at 21:22:56, Simo Sorce (simo(a)redhat.com) wrote:
On Tue, 2018-06-12 at 12:15 -0700, Alessandro Perucchi via FreeIPA-
users wrote:
Hello everyone,
We were using Freeipa on Fedora 24. And we are in the process to upgrade
to
Fedora 28.
We have a cluster of 2 nodes (freeipa-01 and freeipa-02).
I am trying to upgrade one server after the other, from one release to the
next.
Basically:
freeipa-01 Fedora 24 -> Fedora 25
freeipa-02 Fedora 24 -> Fedora 25
freeipa-02 Fedora 25 -> Fedora 26
freeipa-01 Fedora 25 -> Fedora 26
freeipa-01 Fedora 26 -> Fedora 27
freeipa-02 Fedora 26 -> Fedora 27
freeipa-02 Fedora 27 -> Fedora 28
freeipa-01 Fedora 27 -> Fedora 28
Since Fedora doesn’t support to jump from one version to another, except
one release at the time.
My idea is to check that once a server is upgraded, then everything is
stable, before going to the next server, and try to be as near as possible
from a version point of view between the 2 freeipa node cluster.
Today, I could
upgrade without problems from Fedora 24 -> Fedora 25 on both nodes
(freeipa-01 and freeipa-02).
In trying to upgrade to Fedora 26, I got some problems, the main problem
is
that the upgrade of ldap 389 is not successful, and the one from IPA
either.
After investigating a long moment, I have found that ns-slapd listen
only
to IPv6, on UDP, and NOT on IPv4 and TCP.
Here is what I have:
[root@freeipa-02 lib]# lsof -Pni |grep slap
ns-slapd 21005 dirsrv 9u IPv6 1617283379 <//1617283379> 0t0
UDP *:389
ns-slapd 21005 dirsrv 77u IPv4 1617321218 <//1617321218> 0t0
TCP 10.100.0.102:60646->10.100.0.101:389 (ESTABLISHED)
ns-slapd 21005 dirsrv 81u IPv4 1617317640 <//1617317640> 0t0
TCP 10.100.0.102:60648->10.100.0.101:389 (ESTABLISHED)
So, I decided to look at the file dse.ldif, and found that the entry
"nsslapd-port” was set to “0” and no “nsslapd-listenhost” was not set at
all.
I have then added the line
nsslapd-listenhost: 0.0.0.0
and changed the nsslapd-port to look like:
nsslap-port: 389
And after doing a
systemctl stop dirsrv@DOM-LOCAL ; systemctl start dirsrv@DOM-LOCAL
No changes… all modification on my dse.ldif were gone.
I stopped again the dirsrv, did again my changes on dse.ldif, and run the
following command:
/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-DOM-LOCAL -i
/var/run/dirsrv/slapd-DOM-LOCAL.pid
and now, I have the following:
[root@freeipa-02 updates]# lsof -Pni |grep 389
ns-slapd 78507 dirsrv 10u IPv6 1681165214 <//1681165214> 0t0
UDP *:389
ns-slapd 78507 dirsrv 11u IPv4 1681165216 <//1681165216> 0t0
TCP *:389 (LISTEN)
ns-slapd 78507 dirsrv 114u IPv4 1684131928 <//1684131928> 0t0
TCP 10.100.0.102:389->10.100.0.110:36828 (ESTABLISHED)
So my questions are:
- how to change the dse.ldif file?
You have to stop ns-slapd before changing the file.
This is what I have done several times. or have I… maybe not…
I will try again.
- Is there another way to ensure that the port that listen is TCP /
389
on
IPv4?
The port was disabled during some upgrade operations, your situation
meant some upgrade failed and that old version failed to set back the
port in dse.ldif
This is a bug and shouldn't happen in recent versions.
Does it means that I need to upgrade to Fedora 28, and then try to upgrade
FreeIPA?
- Is there something that needs to be done between Fedora 25 and 26?
Is this upgrade bug repeatable ? (keep in mind that F26 is practically
EOL)
Yes, it is repeatable, since I am trying to do it since this 24 hours, and
it drives me crazy… and nothing by googling seems to help!
I know this is EOL, or nearly… That’s also why we wanted to upgrade to the
latest.
Knowing that I will go to Fedora 28, is there something that I need
to be
aware of?
Yes, read this list archives before you attempt F28 upgrades, you may
have to use updates-testing as the GA bits where busted wrt replication
for upgrades.
Ok, guess I have some reading to do :-D
- Anything that can help me generally with my upgrade path?
In general your approach is ok, make backups :-)
Glad that I’m doing it right :-)
If you have any other approach, then I am also open to anything else.
Nevertheless thank you!
4:56 p.m.
I think the problem might be related to that error I got within dirsrv:
[12/Jun/2018:21:28:41.780869662 +0000] - ERR - set_krb5_creds - Could not
get initial credentials for principal
[ldap/freeipa-02.jahia.local(a)JAHIA.LOCAL] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
if I do a klist -kt /etc/dirsrv/ds.keytab
I get the following:
Keytab name: FILE:/etc/dirsrv/ds.keytab
KVNO Timestamp Principal
---- -----------------
--------------------------------------------------------
2 12/20/16 09:39:50 ldap/freeipa-02.jahia.local(a)JAHIA.LOCAL
2 12/20/16 09:39:50 ldap/freeipa-02.jahia.local(a)JAHIA.LOCAL
2 12/20/16 09:39:50 ldap/freeipa-02.jahia.local(a)JAHIA.LOCAL
2 12/20/16 09:39:51 ldap/freeipa-02.jahia.local(a)JAHIA.LOCAL
2 12/20/16 09:39:51 ldap/freeipa-02.jahia.local(a)JAHIA.LOCAL
2 12/20/16 09:39:51 ldap/freeipa-02.jahia.local(a)JAHIA.LOCAL
And after I started the disabled service “krb5kdc”, everything was
“””solved””” for that part of errors.
I also found that during the upgrade from F25 -> F26 the dse.ldif was
changed to have
nsslapd-port: 0
instead of the port 389 as it was in F25.
Since `nsslapd-port: 0` means to use the ldaps.
and since everything behind is using that (ipa-server-upgrade) I cannot
finish the upgrade correctly.
I could go past some steps of the upgrade, but not past the CA verification:
Upgrading IPA:
[1/10]: stopping directory server
[2/10]: saving configuration
[3/10]: disabling listeners
[4/10]: enabling DS global lock
[5/10]: starting directory server
[6/10]: updating schema
[7/10]: upgrading server
[8/10]: stopping directory server
[9/10]: restoring configuration
[10/10]: starting directory server
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
/etc/dirsrv/slapd-JAHIA-LOCAL/certmap.conf is now managed by IPA. It will
be overwritten. A backup of the original will be made.
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
ipa-server-upgrade manually.
CA did not start in 300.0s
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more
information
And the error I get is basically:
0.localhost-startStop-1 - [12/Jun/2018:21:22:29 UTC] [8] [3] In Ldap
(bound) connection pool to host freeipa-02.jahia.local port 636, Cannot
connect to LDAP server. Error: netscape.ldap.LDAPException: Unable to
create socket: java.net.ConnectException: Connection refused (Connection
refused) (-1)
which caused the following error:
12-Jun-2018 21:27:50.393 WARNING
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.core.ContainerBase.backgroundProcess Exception
processing realm com.netscape.cms.tomcat.ProxyRealm@67a
992d background process
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:130)
at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1154)
at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5715)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
at java.lang.Thread.run(Thread.java:748)
and then the following error:
2018-06-12T21:27:21Z DEBUG The CA status is: check interrupted due to
error: Retrieving CA status failed with status 500
2018-06-12T21:27:21Z DEBUG Waiting for CA to start...
2018-06-12T21:27:22Z DEBUG request POST
http://freeipa-02.jahia.local:8080/ca/admin/ca/getStatus
2018-06-12T21:27:22Z DEBUG request body ''
2018-06-12T21:27:22Z DEBUG response status 500
2018-06-12T21:27:22Z DEBUG response headers {'content-length': '2351',
'content-language': 'en', 'server': 'Apache-Coyote/1.1',
'connection':
'close', 'date': 'Tue, 12 Jun 2018 21:27:22 GMT',
'content-type': 'tex
t/html;charset=utf-8'}
2018-06-12T21:27:22Z DEBUG response body '<!DOCTYPE
html><html><head><title>Apache Tomcat/8.0.50 - Error
report</title><style
type="text/css">H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#
525D76;font-size:22px;} H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P {font-family:Tahoma,Arial,sans-serif;background:white
;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line
{height: 1px; background-color: #525D76; border: none;}</style>
</head><body><h1>HTTP Status 500 - Subsystem
unavailable</h1><div class=
"line"></div><p><b>type</b> Exception
report</p><p><b>message</b>
<u>Subsystem unavailable</u></p><p><b>description</b>
<u>The server
encountered an internal error that prevented it from fulfilling this
request.<
/u></p><p><b>exception</b></p><pre>javax.ws.rs.ServiceUnavailableException:
Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:138)\n\torg.apache.catalina.authenti
cator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)\n\torg.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAcces
sLogValve.java:620)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)\n\torg.apache.coyo
te.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)\n\torg.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)\n\torg.apache.tomcat.util.net.NioEndpoint$So
cketProcessor.run(NioEndpoint.java:1495)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tor
g.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/8.0.50 logs.</u></p><hr class="line"><h3>Apache
Tomcat/8.0.50</h3></body></html>'
So the question now would be… What am I missing to have ldaps with port 636?
Thank you in advance again!
Alessandro
On 12 June 2018 at 21:22:56, Simo Sorce (simo(a)redhat.com) wrote:
On Tue, 2018-06-12 at 12:15 -0700, Alessandro Perucchi via FreeIPA-
users wrote:
Hello everyone,
We were using Freeipa on Fedora 24. And we are in the process to upgrade
to
Fedora 28.
We have a cluster of 2 nodes (freeipa-01 and freeipa-02).
I am trying to upgrade one server after the other, from one release to
the
next.
Basically:
freeipa-01 Fedora 24 -> Fedora 25
freeipa-02 Fedora 24 -> Fedora 25
freeipa-02 Fedora 25 -> Fedora 26
freeipa-01 Fedora 25 -> Fedora 26
freeipa-01 Fedora 26 -> Fedora 27
freeipa-02 Fedora 26 -> Fedora 27
freeipa-02 Fedora 27 -> Fedora 28
freeipa-01 Fedora 27 -> Fedora 28
Since Fedora doesn’t support to jump from one version to another, except
one release at the time.
My idea is to check that once a server is upgraded, then everything is
stable, before going to the next server, and try to be as near as
possible
from a version point of view between the 2 freeipa node cluster.
Today <http://airmail.calendar/2018-06-12%2012:00:00%20CEST>;, I could
upgrade without problems from Fedora 24 -> Fedora 25 on both nodes
(freeipa-01 and freeipa-02).
In trying to upgrade to Fedora 26, I got some problems, the main problem
is
that the upgrade of ldap 389 is not successful, and the one from IPA
either.
After investigating a long moment, I have found that ns-slapd listen
only
to IPv6, on UDP, and NOT on IPv4 and TCP.
Here is what I have:
[root@freeipa-02 lib]# lsof -Pni |grep slap
ns-slapd 21005 dirsrv 9u IPv6 1617283379 <//1617283379> 0t0
UDP *:389
ns-slapd 21005 dirsrv 77u IPv4 1617321218 <//1617321218> 0t0
TCP 10.100.0.102:60646->10.100.0.101:389 (ESTABLISHED)
ns-slapd 21005 dirsrv 81u IPv4 1617317640 <//1617317640> 0t0
TCP 10.100.0.102:60648->10.100.0.101:389 (ESTABLISHED)
So, I decided to look at the file dse.ldif, and found that the entry
"nsslapd-port” was set to “0” and no “nsslapd-listenhost” was not set at
all.
I have then added the line
nsslapd-listenhost: 0.0.0.0
and changed the nsslapd-port to look like:
nsslap-port: 389
And after doing a
systemctl stop dirsrv@DOM-LOCAL ; systemctl start dirsrv@DOM-LOCAL
No changes… all modification on my dse.ldif were gone.
I stopped again the dirsrv, did again my changes on dse.ldif, and run the
following command:
/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-DOM-LOCAL -i
/var/run/dirsrv/slapd-DOM-LOCAL.pid
and now, I have the following:
[root@freeipa-02 updates]# lsof -Pni |grep 389
ns-slapd 78507 dirsrv 10u IPv6 1681165214 <//1681165214> 0t0
UDP *:389
ns-slapd 78507 dirsrv 11u IPv4 1681165216 <//1681165216> 0t0
TCP *:389 (LISTEN)
ns-slapd 78507 dirsrv 114u IPv4 1684131928 <//1684131928> 0t0
TCP 10.100.0.102:389->10.100.0.110:36828 (ESTABLISHED)
So my questions are:
- how to change the dse.ldif file?
You have to stop ns-slapd before changing the file.
- Is there another way to ensure that the port that listen is TCP /
389
on
IPv4?
The port was disabled during some upgrade operations, your situation
meant some upgrade failed and that old version failed to set back the
port in dse.ldif
This is a bug and shouldn't happen in recent versions.
- Is there something that needs to be done between Fedora 25 and 26?
Is this upgrade bug repeatable ? (keep in mind that F26 is practically
EOL)
Knowing that I will go to Fedora 28, is there something that I need
to be
aware of?
Yes, read this list archives before you attempt F28 upgrades, you may
have to use updates-testing as the GA bits where busted wrt replication
for upgrades.
- Anything that can help me generally with my upgrade path?
In general your approach is ok, make backups :-)
Simo.
--
Simo Sorce
Sr. Principal Software Engineer
Red Hat, Inc
2154
days inactive
2154
days old
freeipa-users@lists.fedorahosted.org
3 comments
2 participants
participants (2)
-
Alessandro Perucchi -
Simo Sorce