Hi.
Into my environment I have two IPA server, replicating
each other.
They are both 7.6 OS systems, ipa-server RPM version
is 4.6.4-10.0.1.el7_6.2.x86_64.
The first server installed was srv01 (many years ago),
then I installed the replica into srv02 (like a year
later the 1st node).
When I had a single server I did also a trust with my
corporate Active Directory.
VMs are running in 2 different hypervisor clusters.
Now the replication doesn't works. Into log files I
have this error:
[16/Apr/2020:12:25:36.856632697 +0200] - ERR -
csngen_adjust_time - Adjustment limit exceeded;
value - 23221226, limit - 86400
[16/Apr/2020:12:25:36.857909222 +0200] - ERR -
NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com"
(srv01:389): Fatal error - too much time skew
between replicas!
[16/Apr/2020:12:25:36.862233147 +0200] - ERR -
NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com"
(srv01:389): Incremental update failed and requires
administrator action
I tried to force the replica, but the limit exceeded
problem doesn't allow the sync.
I know that the problem is that CSN generator has
become grossly skewed.
Using the external script readNsState.py I found that
there was as offset time for about a month, so ... I
waited for a month and then the issue disappeared.
But now the offset is about 9 months ... I can't wait
so much time :)
[root@srv01 scripts]# ./readNsState.py
/etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif
nsState is
BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA==
Little Endian
For replica
cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping
tree,cn=con
fmtstr=[H6x3QH6x]
size=40
len of nsstate is 40
CSN generator state:
Replica ID : 4
Sampled Time : 1610364802
Gen as csn : 5ffc37822996500040000
Time as str : Mon Jan 11 12:33:22 2021
Local Offset : 320118
Remote Offset : 10244
Seq. num : 29965
System time : Tue Apr 21 15:03:45 2020
Diff in sec. : -22890577
Day:sec diff : -265:5423
nsState is
YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA==
Little Endian
For replica cn=replica,cn=o\3Dipaca,cn=mapping
tree,cn=config
fmtstr=[H6x3QH6x]
size=40
len of nsstate is 40
CSN generator state:
Replica ID : 96
Sampled Time : 1587031299
Gen as csn : 5e982d03001900960000
Time as str : Thu Apr 16 12:01:39 2020
Local Offset : 0
Remote Offset : 10333
Seq. num : 19
System time : Tue Apr 21 15:03:45 2020
Diff in sec. : 442926
Day:sec diff : 5:10926
[root@srv02 scripts]# ./readNsState.py
/etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif
nsState is
AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA==
Little Endian
For replica
cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping
tree,cn=con
fmtstr=[H6x3QH6x]
size=40
len of nsstate is 40
CSN generator state:
Replica ID : 3
Sampled Time : 1587474004
Gen as csn : 5e9eee54000000030000
Time as str : Tue Apr 21 15:00:04 2020
Local Offset : 0
Remote Offset : 23221169
Seq. num : 0
System time : Tue Apr 21 15:02:38 2020
Diff in sec. : 154
Day:sec diff : 0:154
nsState is
YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA==
Little Endian
For replica cn=replica,cn=o\3Dipaca,cn=mapping
tree,cn=config
fmtstr=[H6x3QH6x]
size=40
len of nsstate is 40
CSN generator state:
Replica ID : 97
Sampled Time : 1587031342
Gen as csn : 5e982d2e001800970000
Time as str : Thu Apr 16 12:02:22 2020
Local Offset : 325
Remote Offset : 9965
Seq. num : 18
System time : Tue Apr 21 15:02:38 2020
Diff in sec. : 442816
Day:sec diff : 5:10816
As you can see in the 1st node the Time as str is Jan
11 of 2021.
With timedatectl command I see that both VMs use the
same Time zone and the clock is correct.
I found this old article to fix my issue:
https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html
But ... I had the same issue in the past, always in
the 1st server. So, in my mind I don't want to try to
use that fix.
I have a new hypervisor cluster, so I would prefer to
reinstall the 1st server, using these steps:
1) check if all roles (also the CA) is installed in
srv02
You can find here some data about the VMs:
[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com
Server name: srv01.ipa.mydomain.com
Managed suffixes: domain, ca
Min domain level: 0
Max domain level: 1
Enabled server roles: CA server, IPA master, DNS
server, NTP server, AD trust controller
[root@srv02 ~]# ipa server-show srv02.ipa.mydomain.com
Server name: srv02.ipa.mydomain.com
Managed suffixes: domain, ca
Min domain level: 0
Max domain level: 1
Enabled server roles: CA server, IPA master, DNS
server, NTP server
[root@srv01 ~]# ipa config-show
Maximum username length: 32
Home directory base: /home
Default shell: /bin/bash
Default users group: ipausers
Default e-mail domain: ipa.mydomain.com
Search time limit: 2
Search size limit: 100
User search fields:
uid,givenname,sn,telephonenumber,ou,title
Group search fields: cn,description
Enable migration mode: FALSE
Certificate Subject base: O=IPA.MYDOMAIN.COM
Password Expiration Notification (days): 4
Password plugin features: AllowNThash
SELinux user map order:
guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023
Default SELinux user: unconfined_u:s0-s0:c0.c1023
Default PAC types: MS-PAC, nfs:NONE
IPA masters: srv01.ipa.mydomain.com,
srv02.ipa.mydomain.com
IPA CA servers: srv01.ipa.mydomain.com,
srv02.ipa.mydomain.com
IPA NTP servers: srv01.ipa.mydomain.com,
srv02.ipa.mydomain.com
IPA CA renewal master: srv01.ipa.mydomain.com
[root@srv02 ~]# ipa config-show
Maximum username length: 32
Home directory base: /home
Default shell: /bin/bash
Default users group: ipausers
Default e-mail domain: ipa.mydomain.com
Search time limit: 2
Search size limit: 100
User search fields:
uid,givenname,sn,telephonenumber,ou,title
Group search fields: cn,description
Enable migration mode: FALSE
Certificate Subject base: O=IPA.MYDOMAIN.COM
Password Expiration Notification (days): 4
Password plugin features: AllowNThash
SELinux user map order:
guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023
Default SELinux user: unconfined_u:s0-s0:c0.c1023
Default PAC types: MS-PAC, nfs:NONE
IPA masters: srv01.ipa.mydomain.com,
srv02.ipa.mydomain.com
IPA CA servers: srv01.ipa.mydomain.com,
srv02.ipa.mydomain.com
IPA NTP servers: srv01.ipa.mydomain.com,
srv02.ipa.mydomain.com
IPA CA renewal master: srv01.ipa.mydomain.com
[root@srv01 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
[root@srv02 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
[root@srv01 ~]# certutil -L -d
/etc/pki/pki-tomcat/alias
Certificate Nickname
Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert cert-pki-ca
u,u,u
subsystemCert cert-pki-ca
u,u,u
caSigningCert cert-pki-ca
CTu,Cu,Cu
ocspSigningCert cert-pki-ca
u,u,u
auditSigningCert cert-pki-ca
u,u,Pu
[root@srv02 ~]# certutil -L -d
/etc/pki/pki-tomcat/alias
Certificate Nickname
Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert cert-pki-ca
u,u,u
subsystemCert cert-pki-ca
u,u,u
caSigningCert cert-pki-ca
CTu,u,u
ocspSigningCert cert-pki-ca
u,u,u
auditSigningCert cert-pki-ca
u,u,Pu
It seems that AD trust controller role, IPA CA
renewal master, smb and windbind are only in the 1st
server.
And also caSigningCert cert-pki-ca entry is different
(CTu,Cu,Cu vs CTu,u,u).
I can see only in the 1st server these DNS records:
_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs
SRV 0 100 88 srv01
_kerberos._tcp.dc._msdcs SRV 0 100 88 srv01
_kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88
srv01
_kerberos._udp.dc._msdcs SRV 0 100 88 srv01
_ldap._tcp.Default-First-Site-Name._sites.dc._msdcs
0 100 389 srv01
_ldap._tcp.dc._msdcs 0 100 389 srv01
Srv01 is the first master, I know, but is the server
VM that has clock problems, in both situations.
So I want to keep srv02 and install a new one.
What do I have to do to let the 2nd VM be a single
server?
Could I use these URLs?
https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Master
https://www.freeipa.org/page/V4/Server_Roles#Upgrade
2) uninstall ipa-server from the 1st server (srv01)
and then powering off it, assuming that all data into
the 2nd one are ok (srv02)
3) update freeipa and all other RPM packages into the
VM srv02
4) install a new fresh VM, always with 7 release, and
create a new replica
Could I use the same old hostname (srv01) and IP
address for this new VM? Or is better to use the same
IP but a new name, like srv03?
Do you think this is the right way to solve my issue?
Or do you have any better idea?
Please let me know, thanks.
Bye, Morgan