10:42 a.m.
Hi.
Into my environment I have two IPA server, replicating each other.
They are both 7.6 OS systems, ipa-server RPM version is
4.6.4-10.0.1.el7_6.2.x86_64.
The first server installed was srv01 (many years ago), then I installed the
replica into srv02 (like a year later the 1st node).
When I had a single server I did also a trust with my corporate Active
Directory.
VMs are running in 2 different hypervisor clusters.
Now the replication doesn't works. Into log files I have this error:
*[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time -
Adjustment limit exceeded; value - 23221226, limit -
86400[16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin -
repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com
<http://meTosrv01.ipa.mydomain.com>" (srv01:389): Fatal error - too much
time skew between replicas![16/Apr/2020:12:25:36.862233147 +0200] - ERR -
NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com
<http://meTosrv01.ipa.mydomain.com>" (srv01:389): Incremental update failed
and requires administrator action*
I tried to force the replica, but the limit exceeded problem doesn't allow
the sync.
I know that the problem is that CSN generator has become grossly skewed.
Using the external script readNsState.py I found that there was as offset
time for about a month, so ... I waited for a month and then the issue
disappeared.
But now the offset is about 9 months ... I can't wait so much time :)
*[root@srv01 scripts]# ./readNsState.py
/etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldifnsState is
BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA==Little EndianFor
replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping
tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN
generator state: Replica ID : 4 Sampled Time : 1610364802 Gen
as csn : 5ffc37822996500040000 Time as str : Mon Jan 11 12:33:22
2021 Local Offset : 320118 Remote Offset : 10244 Seq. num :
29965 System time : Tue Apr 21 15:03:45 2020 Diff in sec. :
-22890577 Day:sec diff : -265:5423nsState is
YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA==Little EndianFor
replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state:
Replica ID : 96 Sampled Time : 1587031299 Gen as csn :
5e982d03001900960000 Time as str : Thu Apr 16 12:01:39 2020 Local
Offset : 0 Remote Offset : 10333 Seq. num : 19 System time
: Tue Apr 21 15:03:45 2020 Diff in sec. : 442926 Day:sec diff :
5:10926[root@srv02 scripts]# ./readNsState.py
/etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldifnsState is
AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA==Little EndianFor
replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping
tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN
generator state: Replica ID : 3 Sampled Time : 1587474004 Gen
as csn : 5e9eee54000000030000 Time as str : Tue Apr 21 15:00:04
2020 Local Offset : 0 Remote Offset : 23221169 Seq. num : 0
System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 154
Day:sec diff : 0:154nsState is
YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA==Little EndianFor
replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state:
Replica ID : 97 Sampled Time : 1587031342 Gen as csn :
5e982d2e001800970000 Time as str : Thu Apr 16 12:02:22 2020 Local
Offset : 325 Remote Offset : 9965 Seq. num : 18 System time
: Tue Apr 21 15:02:38 2020 Diff in sec. : 442816 Day:sec diff :
5:10816*
As you can see in the 1st node the Time as str is Jan 11 of 2021.
With timedatectl command I see that both VMs use the same Time zone and the
clock is correct.
I found this old article to fix my issue:
*https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html
<https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html...
But ... I had the same issue in the past, always in the 1st server. So, in
my mind I don't want to try to use that fix.
I have a new hypervisor cluster, so I would prefer to reinstall the 1st
server, using these steps:
1) check if all roles (also the CA) is installed in srv02
You can find here some data about the VMs:
*[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com> Server name: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com> Managed suffixes: domain, ca Min domain
level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master,
DNS server, NTP server, AD trust controller[root@srv02 ~]# ipa server-show
srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> Server name:
srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> Managed suffixes:
domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles:
CA server, IPA master, DNS server, NTP server[root@srv01 ~]# ipa
config-show Maximum username length: 32 Home directory base: /home
Default shell: /bin/bash Default users group: ipausers Default e-mail
domain: ipa.mydomain.com <http://ipa.mydomain.com> Search time limit: 2
Search size limit: 100 User search fields:
uid,givenname,sn,telephonenumber,ou,title Group search fields:
cn,description Enable migration mode: FALSE Certificate Subject base:
O=IPA.MYDOMAIN.COM <http://IPA.MYDOMAIN.COM> Password Expiration
Notification (days): 4 Password plugin features: AllowNThash SELinux user
map order:
guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023
Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types:
MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>;, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA CA servers: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>;, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA NTP servers: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>;, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA CA renewal master:
srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>[root@srv02 ~]# ipa
config-show Maximum username length: 32 Home directory base: /home
Default shell: /bin/bash Default users group: ipausers Default e-mail
domain: ipa.mydomain.com <http://ipa.mydomain.com> Search time limit: 2
Search size limit: 100 User search fields:
uid,givenname,sn,telephonenumber,ou,title Group search fields:
cn,description Enable migration mode: FALSE Certificate Subject base:
O=IPA.MYDOMAIN.COM <http://IPA.MYDOMAIN.COM> Password Expiration
Notification (days): 4 Password plugin features: AllowNThash SELinux user
map order:
guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023
Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types:
MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>;, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA CA servers: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>;, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA NTP servers: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>;, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA CA renewal master:
srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>[root@srv01 ~]#
ipactl statusDirectory Service: RUNNINGkrb5kdc Service: RUNNINGkadmin
Service: RUNNINGnamed Service: RUNNINGhttpd Service: RUNNINGipa-custodia
Service: RUNNINGntpd Service: RUNNINGpki-tomcatd Service: STOPPEDsmb
Service: RUNNINGwinbind Service: RUNNINGipa-otpd Service:
RUNNINGipa-dnskeysyncd Service: RUNNINGipa: INFO: The ipactl command was
successful[root@srv02 ~]# ipactl statusDirectory Service: RUNNINGkrb5kdc
Service: RUNNINGkadmin Service: RUNNINGnamed Service: RUNNINGhttpd Service:
RUNNINGipa-custodia Service: RUNNINGntpd Service: RUNNINGpki-tomcatd
Service: STOPPEDipa-otpd Service: RUNNINGipa-dnskeysyncd Service:
RUNNINGipa: INFO: The ipactl command was successful[root@srv01 ~]# certutil
-L -d /etc/pki/pki-tomcat/aliasCertificate Nickname
Trust Attributes
SSL,S/MIME,JAR/XPIServer-Cert cert-pki-ca
u,u,usubsystemCert cert-pki-ca
u,u,ucaSigningCert cert-pki-ca
CTu,Cu,CuocspSigningCert cert-pki-ca
u,u,uauditSigningCert cert-pki-ca
u,u,Pu[root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/aliasCertificate
Nickname Trust Attributes
SSL,S/MIME,JAR/XPIServer-Cert cert-pki-ca
u,u,usubsystemCert cert-pki-ca
u,u,ucaSigningCert cert-pki-ca
CTu,u,uocspSigningCert cert-pki-ca
u,u,uauditSigningCert cert-pki-ca u,u,Pu*
It seems that AD trust controller role, IPA CA renewal master, smb and
windbind are only in the 1st server.
And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs
CTu,u,u).
I can see only in the 1st server these DNS records:
*_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88
srv01_kerberos._tcp.dc._msdcs SRV 0 100 88
srv01_kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88
srv01_kerberos._udp.dc._msdcs SRV 0 100 88
srv01_ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389
srv01_ldap._tcp.dc._msdcs 0 100 389 srv01*
Srv01 is the first master, I know, but is the server VM that has clock
problems, in both situations.
So I want to keep srv02 and install a new one.
What do I have to do to let the 2nd VM be a single server?
Could I use these URLs?
*https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Master
<https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_M...
<https://www.freeipa.org/page/V4/Server_Roles#Upgrade>*
2) uninstall ipa-server from the 1st server (srv01) and then powering off
it, assuming that all data into the 2nd one are ok (srv02)
3) update freeipa and all other RPM packages into the VM srv02
4) install a new fresh VM, always with 7 release, and create a new replica
Could I use the same old hostname (srv01) and IP address for this new VM?
Or is better to use the same IP but a new name, like srv03?
Do you think this is the right way to solve my issue?
Or do you have any better idea?
Please let me know, thanks.
Bye, Morgan