Hi,

CSN generator time skew is a pending issue still under investigation.

At the moment the way your csn generator is messed up looks not fatal. You can allow replication to continue with the setting of nsslapd-ignore-time-skew on all servers. (https://access.redhat.com/solutions/1162703)

If it does not allow replication to continue there is a recovery procedure but I would recommend to first try ignore-time-skew (https://access.redhat.com/solutions/3543811)

NTP tuning or specific VMs are suspected to contribute to time skew. What type of VMs are you using (local or cloud (AWS)) ?

best regards
thierry

On 4/21/20 5:42 PM, Morgan Marodin via FreeIPA-users wrote:

Hi.

Into my environment I have two IPA server, replicating each other.
They are both 7.6 OS systems, ipa-server RPM version is 4.6.4-10.0.1.el7_6.2.x86_64.

The first server installed was srv01 (many years ago), then I installed the replica into srv02 (like a year later the 1st node).
When I had a single server I did also a trust with my corporate Active Directory.
VMs are running in 2 different hypervisor clusters.

Now the replication doesn't works. Into log files I have this error:
[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time - Adjustment limit exceeded; value - 23221226, limit - 86400
[16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com" (srv01:389): Fatal error - too much time skew between replicas!
[16/Apr/2020:12:25:36.862233147 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com" (srv01:389): Incremental update failed and requires administrator action

I tried to force the replica, but the limit exceeded problem doesn't allow the sync.
I know that the problem is that CSN generator has become grossly skewed.
Using the external script readNsState.py I found that there was as offset time for about a month, so ... I waited for a month and then the issue disappeared.
But now the offset is about 9 months ... I can't wait so much time :)

[root@srv01 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif
nsState is BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA==
Little Endian
For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con
  fmtstr=[H6x3QH6x]
  size=40
  len of nsstate is 40
  CSN generator state:
    Replica ID    : 4
    Sampled Time  : 1610364802
    Gen as csn    : 5ffc37822996500040000
    Time as str   : Mon Jan 11 12:33:22 2021
    Local Offset  : 320118
    Remote Offset : 10244
    Seq. num      : 29965
    System time   : Tue Apr 21 15:03:45 2020
    Diff in sec.  : -22890577
    Day:sec diff  : -265:5423

nsState is YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA==
Little Endian
For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
  fmtstr=[H6x3QH6x]
  size=40
  len of nsstate is 40
  CSN generator state:
    Replica ID    : 96
    Sampled Time  : 1587031299
    Gen as csn    : 5e982d03001900960000
    Time as str   : Thu Apr 16 12:01:39 2020
    Local Offset  : 0
    Remote Offset : 10333
    Seq. num      : 19
    System time   : Tue Apr 21 15:03:45 2020
    Diff in sec.  : 442926
    Day:sec diff  : 5:10926

[root@srv02 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif
nsState is AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA==
Little Endian
For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con
  fmtstr=[H6x3QH6x]
  size=40
  len of nsstate is 40
  CSN generator state:
    Replica ID    : 3
    Sampled Time  : 1587474004
    Gen as csn    : 5e9eee54000000030000
    Time as str   : Tue Apr 21 15:00:04 2020
    Local Offset  : 0
    Remote Offset : 23221169
    Seq. num      : 0
    System time   : Tue Apr 21 15:02:38 2020
    Diff in sec.  : 154
    Day:sec diff  : 0:154

nsState is YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA==
Little Endian
For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
  fmtstr=[H6x3QH6x]
  size=40
  len of nsstate is 40
  CSN generator state:
    Replica ID    : 97
    Sampled Time  : 1587031342
    Gen as csn    : 5e982d2e001800970000
    Time as str   : Thu Apr 16 12:02:22 2020
    Local Offset  : 325
    Remote Offset : 9965
    Seq. num      : 18
    System time   : Tue Apr 21 15:02:38 2020
    Diff in sec.  : 442816
    Day:sec diff  : 5:10816

As you can see in the 1st node the Time as str is Jan 11 of 2021.
With timedatectl command I see that both VMs use the same Time zone and the clock is correct.

I found this old article to fix my issue:
https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html

But ... I had the same issue in the past, always in the 1st server. So, in my mind I don't want to try to use that fix.
I have a new hypervisor cluster, so I would prefer to reinstall the 1st server, using these steps:

1) check if all roles (also the CA) is installed in srv02
You can find here some data about the VMs:

[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com
  Server name: srv01.ipa.mydomain.com
  Managed suffixes: domain, ca
  Min domain level: 0
  Max domain level: 1
  Enabled server roles: CA server, IPA master, DNS server, NTP server, AD trust controller

[root@srv02 ~]# ipa server-show srv02.ipa.mydomain.com
  Server name: srv02.ipa.mydomain.com
  Managed suffixes: domain, ca
  Min domain level: 0
  Max domain level: 1
  Enabled server roles: CA server, IPA master, DNS server, NTP server


[root@srv01 ~]# ipa config-show
  Maximum username length: 32
  Home directory base: /home
  Default shell: /bin/bash
  Default users group: ipausers
  Default e-mail domain: ipa.mydomain.com
  Search time limit: 2
  Search size limit: 100
  User search fields: uid,givenname,sn,telephonenumber,ou,title
  Group search fields: cn,description
  Enable migration mode: FALSE
  Certificate Subject base: O=IPA.MYDOMAIN.COM
  Password Expiration Notification (days): 4
  Password plugin features: AllowNThash
  SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023
  Default SELinux user: unconfined_u:s0-s0:c0.c1023
  Default PAC types: MS-PAC, nfs:NONE
  IPA masters: srv01.ipa.mydomain.com, srv02.ipa.mydomain.com
  IPA CA servers: srv01.ipa.mydomain.com, srv02.ipa.mydomain.com
  IPA NTP servers: srv01.ipa.mydomain.com, srv02.ipa.mydomain.com
  IPA CA renewal master: srv01.ipa.mydomain.com

[root@srv02 ~]# ipa config-show
  Maximum username length: 32
  Home directory base: /home
  Default shell: /bin/bash
  Default users group: ipausers
  Default e-mail domain: ipa.mydomain.com
  Search time limit: 2
  Search size limit: 100
  User search fields: uid,givenname,sn,telephonenumber,ou,title
  Group search fields: cn,description
  Enable migration mode: FALSE
  Certificate Subject base: O=IPA.MYDOMAIN.COM
  Password Expiration Notification (days): 4
  Password plugin features: AllowNThash
  SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023
  Default SELinux user: unconfined_u:s0-s0:c0.c1023
  Default PAC types: MS-PAC, nfs:NONE
  IPA masters: srv01.ipa.mydomain.com, srv02.ipa.mydomain.com
  IPA CA servers: srv01.ipa.mydomain.com, srv02.ipa.mydomain.com
  IPA NTP servers: srv01.ipa.mydomain.com, srv02.ipa.mydomain.com
  IPA CA renewal master: srv01.ipa.mydomain.com


[root@srv01 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

[root@srv02 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful


[root@srv01 ~]# certutil -L -d /etc/pki/pki-tomcat/alias
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI
Server-Cert cert-pki-ca                                      u,u,u
subsystemCert cert-pki-ca                                    u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
ocspSigningCert cert-pki-ca                                  u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu


[root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/alias
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI
Server-Cert cert-pki-ca                                      u,u,u
subsystemCert cert-pki-ca                                    u,u,u
caSigningCert cert-pki-ca                                    CTu,u,u
ocspSigningCert cert-pki-ca                                  u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu

It seems that AD trust controller role, IPA CA renewal master, smb and windbind are only in the 1st server.

And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs CTu,u,u).

I can see only in the 1st server these DNS records:
_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01
_kerberos._tcp.dc._msdcs SRV 0 100 88 srv01
_kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01
_kerberos._udp.dc._msdcs SRV 0 100 88 srv01
_ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389 srv01
_ldap._tcp.dc._msdcs 0 100 389 srv01

Srv01 is the first master, I know, but is the server VM that has clock problems, in both situations.
So I want to keep srv02 and install a new one.

What do I have to do to let the 2nd VM be a single server?
Could I use these URLs?
https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Master
https://www.freeipa.org/page/V4/Server_Roles#Upgrade


2) uninstall ipa-server from the 1st server (srv01) and then powering off it, assuming that all data into the 2nd one are ok (srv02)

3) update freeipa and all other RPM packages into the VM srv02

4) install a new fresh VM, always with 7 release, and create a new replica
Could I use the same old hostname (srv01) and IP address for this new VM? Or is better to use the same IP but a new name, like srv03?


Do you think this is the right way to solve my issue?
Or do you have any better idea?

Please let me know, thanks.
Bye, Morgan

_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org

_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org