[Freeipa-users] Re: Replication issue with CSN generator

On 8/10/20 11:51 AM, Morgan Marodin via FreeIPA-users wrote: > My issue got worse, the certificate has expired on the replica server, > and it can't be renewed because it cannot communicate with the master > server. > I can start the server using the /--ignore-service-failure/ parameter, > but the /httpd/ and the /pki-tomcatd/ failed to start ... > > Do I have to start up with a fresh install? :( > It's often possible to repair a broken replica but it will be quicker to completely reinstall the replica. On the master, remove the replica: # kinit admin # ipa server-del <replicafqdn> --force On the replica, uninstall ipa: ipa-server-install --uninstall -U kdestroy -A On the replica, re-install: ipa-replica-install [options] HTH, flo > Do you know any company that can give me support about FreeIPA? > > Please let me know, thanks. > Bye, Morgan > > Il giorno lun 27 apr 2020 alle ore 12:16 Morgan Marodin > <morgan(a)marodin.it <mailto:morgan@marodin.it>> ha scritto: > > Sorry Thierry, any news for my issue? > Did you see new logs? > > Could I use the command /ipa-replica-manage re-initialize --from > srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com>/ in the First > Master server? > Or is it better to solve the issue in another way? > > Please let me know, thanks. > Bye > > Il giorno gio 23 apr 2020 alle ore 09:54 Morgan Marodin > <morgan(a)marodin.it <mailto:morgan@marodin.it>> ha scritto: > > Hi Theirry. > > To tell the truth my configuration was already set to on, on > both VMs: > /[root@srv01 ~]# ldapsearch -D "cn=Directory Manager" -h > localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skew > nsslapd-ignore-time-skew: on/ > > /[root@srv02 ~]# ldapsearch -D "cn=Directory Manager" -h > localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skew > nsslapd-ignore-time-skew: on/ > > Anyway, I tried to set up it to /off/ and then to /on/ again, > but now I have a new issue into logs of the 2nd server :( > > Srv01 logs are similar as before: > /[23/Apr/2020:09:28:45.958922636 +0200] - WARN - csngen_new_csn > - Too much time skew (-22777212 secs). Current seqnum=5ca0/ > > Srv02 logs now are like these: > /[23/Apr/2020:09:32:14.919328803 +0200] - ERR - > agmt="cn=meTosrv01.ipa.mydomain.com > <http://meTosrv01.ipa.mydomain.com>" (srv01:389) - > clcache_load_buffer - Can't locate CSN 5e7cfe03000400030000 in > the changelog (DB rc=-30988). If replication stops, the consumer > may need to be reinitialized. > [23/Apr/2020:09:32:14.920873489 +0200] - ERR - > NSMMReplicationPlugin - changelog program - repl_plugin_name_cl > - agmt="cn=meTosrv01.ipa.mydomain.com > <http://meTosrv01.ipa.mydomain.com>" (srv01:389): CSN > 5e7cfe03000400030000 not found, we aren't as up to date, or we > purged > [23/Apr/2020:09:32:14.922161821 +0200] - ERR - > NSMMReplicationPlugin - send_updates - > agmt="cn=meTosrv01.ipa.mydomain.com > <http://meTosrv01.ipa.mydomain.com>" (srv01:389): Data required > to update replica has been purged from the changelog. If the > error persists the replica must be reinitialized./ > > I have just tried to force a replica on both sides, without success: > /[root@srv01 ~]# ipa-replica-manage force-sync --from > srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> > No status yet > No status yet > No status yet > > [root@srv02 ~]# ipa-replica-manage force-sync --from > srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> > No status yet > No status yet > No status yet/ > > What could I do now? > Thanks, bye > > Il giorno mer 22 apr 2020 alle ore 17:08 thierry bordaz via > FreeIPA-users <freeipa-users(a)lists.fedorahosted.org > <mailto:freeipa-users@lists.fedorahosted.org>> ha scritto: > > Hi Morgan, > > Sure. The most immediate and safest action is to do > > |dn: cn=config changetype: modify replace: > nsslapd-ignore-time-skew nsslapd-ignore-time-skew: on | > > On all servers in the topology (no need to restart). Then > monitor if replication is catching up. > Okay NTP issues is likely the RC of your time skew but there > is not easy way to prove it if any. > > best regards > theirry > > > > On 4/22/20 3:16 PM, Morgan Marodin via FreeIPA-users wrote: >> Hi. >> >> I don't have access to RedHat portal :( >> There are similar articles in a public forum? >> >> Anyway ... could I stop ipa-server, change the value of >> /nsslapd-ignore-time-skew/ into >> //etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif/ and start >> again the server? >> Or is more complicated to change the configuration? >> >> VMs are local, but the cluster where the 1st server is >> running is affected by NTP problems ... >> For this reason I want to remove the First Master and >> install another replica in the new cluster. >> >> Thanks, bye. >> Morgan >> >> Il giorno mer 22 apr 2020 alle ore 11:33 thierry bordaz >> via FreeIPA-users <freeipa-users(a)lists.fedorahosted.org >> <mailto:freeipa-users@lists.fedorahosted.org>> ha scritto: >> >> Hi, >> >> CSN generator time skew is a pending issue still under >> investigation. >> >> At the moment the way your csn generator is messed up >> looks not fatal. You can allow replication to continue >> with the setting of nsslapd-ignore-time-skew on all >> servers. (https://access.redhat.com/solutions/1162703) >> >> If it does not allow replication to continue there is >> a recovery procedure but I would recommend to first >> try ignore-time-skew >> (https://access.redhat.com/solutions/3543811) >> >> NTP tuning or specific VMs are suspected to contribute >> to time skew. What type of VMs are you using (local or >> cloud (AWS)) ? >> >> best regards >> thierry >> >> On 4/21/20 5:42 PM, Morgan Marodin via FreeIPA-users >> wrote: >>> Hi. >>> >>> Into my environment I have two IPA server, >>> replicating each other. >>> They are both 7.6 OS systems, ipa-server RPM version >>> is 4.6.4-10.0.1.el7_6.2.x86_64. >>> >>> The first server installed was srv01 (many years >>> ago), then I installed the replica into srv02 (like a >>> year later the 1st node). >>> When I had a single server I did also a trust with my >>> corporate Active Directory. >>> VMs are running in 2 different hypervisor clusters. >>> >>> Now the replication doesn't works. Into log files I >>> have this error: >>> /[16/Apr/2020:12:25:36.856632697 +0200] - ERR - >>> csngen_adjust_time - Adjustment limit exceeded; value >>> - 23221226, limit - 86400 >>> [16/Apr/2020:12:25:36.857909222 +0200] - ERR - >>> NSMMReplicationPlugin - repl5_inc_run - >>> agmt="cn=meTosrv01.ipa.mydomain.com >>> <http://meTosrv01.ipa.mydomain.com>" (srv01:389): >>> Fatal error - too much time skew between replicas! >>> [16/Apr/2020:12:25:36.862233147 +0200] - ERR - >>> NSMMReplicationPlugin - repl5_inc_run - >>> agmt="cn=meTosrv01.ipa.mydomain.com >>> <http://meTosrv01.ipa.mydomain.com>" (srv01:389): >>> Incremental update failed and requires administrator >>> action/ >>> >>> I tried to force the replica, but the limit exceeded >>> problem doesn't allow the sync. >>> I know that the problem is that CSN generator has >>> become grossly skewed. >>> Using the external script readNsState.py I found that >>> there was as offset time for about a month, so ... I >>> waited for a month and then the issue disappeared. >>> But now the offset is about 9 months ... I can't wait >>> so much time :) >>> >>> /[root@srv01 scripts]# ./readNsState.py >>> /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif >>> nsState is >>> BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA== >>> Little Endian >>> For replica >>> cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping >>> tree,cn=con >>> fmtstr=[H6x3QH6x] >>> size=40 >>> len of nsstate is 40 >>> CSN generator state: >>> Replica ID : 4 >>> Sampled Time : 1610364802 >>> Gen as csn : 5ffc37822996500040000 >>> *Time as str : Mon Jan 11 12:33:22 2021* >>> Local Offset : 320118 >>> Remote Offset : 10244 >>> Seq. num : 29965 >>> System time : Tue Apr 21 15:03:45 2020 >>> Diff in sec. : -22890577 >>> Day:sec diff : -265:5423 >>> >>> nsState is >>> YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA== >>> Little Endian >>> For replica cn=replica,cn=o\3Dipaca,cn=mapping >>> tree,cn=config >>> fmtstr=[H6x3QH6x] >>> size=40 >>> len of nsstate is 40 >>> CSN generator state: >>> Replica ID : 96 >>> Sampled Time : 1587031299 >>> Gen as csn : 5e982d03001900960000 >>> Time as str : Thu Apr 16 12:01:39 2020 >>> Local Offset : 0 >>> Remote Offset : 10333 >>> Seq. num : 19 >>> System time : Tue Apr 21 15:03:45 2020 >>> Diff in sec. : 442926 >>> Day:sec diff : 5:10926 >>> >>> [root@srv02 scripts]# ./readNsState.py >>> /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif >>> nsState is >>> AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA== >>> Little Endian >>> For replica >>> cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping >>> tree,cn=con >>> fmtstr=[H6x3QH6x] >>> size=40 >>> len of nsstate is 40 >>> CSN generator state: >>> Replica ID : 3 >>> Sampled Time : 1587474004 >>> Gen as csn : 5e9eee54000000030000 >>> Time as str : Tue Apr 21 15:00:04 2020 >>> Local Offset : 0 >>> Remote Offset : 23221169 >>> Seq. num : 0 >>> System time : Tue Apr 21 15:02:38 2020 >>> Diff in sec. : 154 >>> Day:sec diff : 0:154 >>> >>> nsState is >>> YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA== >>> Little Endian >>> For replica cn=replica,cn=o\3Dipaca,cn=mapping >>> tree,cn=config >>> fmtstr=[H6x3QH6x] >>> size=40 >>> len of nsstate is 40 >>> CSN generator state: >>> Replica ID : 97 >>> Sampled Time : 1587031342 >>> Gen as csn : 5e982d2e001800970000 >>> Time as str : Thu Apr 16 12:02:22 2020 >>> Local Offset : 325 >>> Remote Offset : 9965 >>> Seq. num : 18 >>> System time : Tue Apr 21 15:02:38 2020 >>> Diff in sec. : 442816 >>> Day:sec diff : 5:10816/ >>> >>> As you can see in the 1st node the Time as str is Jan >>> 11 of 2021. >>> With timedatectl command I see that both VMs use the >>> same Time zone and the clock is correct. >>> >>> I found this old article to fix my issue: >>> / https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html/ >>> >>> But ... I had the same issue in the past, always in >>> the 1st server. So, in my mind I don't want to try to >>> use that fix. >>> I have a new hypervisor cluster, so I would prefer to >>> reinstall the 1st server, using these steps: >>> >>> 1) check if all roles (also the CA) is installed in srv02 >>> You can find here some data about the VMs: >>> >>> /[root@srv01 ~]# ipa server-show >>> srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> >>> Server name: srv01.ipa.mydomain.com >>> <http://srv01.ipa.mydomain.com> >>> Managed suffixes: domain, ca >>> Min domain level: 0 >>> Max domain level: 1 >>> Enabled server roles: CA server, IPA master, DNS >>> server, NTP server, *AD trust controller* >>> >>> [root@srv02 ~]# ipa server-show >>> srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> >>> Server name: srv02.ipa.mydomain.com >>> <http://srv02.ipa.mydomain.com> >>> Managed suffixes: domain, ca >>> Min domain level: 0 >>> Max domain level: 1 >>> Enabled server roles: CA server, IPA master, DNS >>> server, NTP server >>> >>> >>> [root@srv01 ~]# ipa config-show >>> Maximum username length: 32 >>> Home directory base: /home >>> Default shell: /bin/bash >>> Default users group: ipausers >>> Default e-mail domain: ipa.mydomain.com >>> <http://ipa.mydomain.com> >>> Search time limit: 2 >>> Search size limit: 100 >>> User search fields: >>> uid,givenname,sn,telephonenumber,ou,title >>> Group search fields: cn,description >>> Enable migration mode: FALSE >>> Certificate Subject base: O=IPA.MYDOMAIN.COM >>> <http://IPA.MYDOMAIN.COM> >>> Password Expiration Notification (days): 4 >>> Password plugin features: AllowNThash >>> SELinux user map order: >>> guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 >>> Default SELinux user: unconfined_u:s0-s0:c0.c1023 >>> Default PAC types: MS-PAC, nfs:NONE >>> IPA masters: srv01.ipa.mydomain.com >>> <http://srv01.ipa.mydomain.com>;, >>> srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> >>> IPA CA servers: srv01.ipa.mydomain.com >>> <http://srv01.ipa.mydomain.com>;, >>> srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> >>> IPA NTP servers: srv01.ipa.mydomain.com >>> <http://srv01.ipa.mydomain.com>;, >>> srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> >>> IPA CA renewal master: srv01.ipa.mydomain.com >>> <http://srv01.ipa.mydomain.com> >>> >>> [root@srv02 ~]# ipa config-show >>> Maximum username length: 32 >>> Home directory base: /home >>> Default shell: /bin/bash >>> Default users group: ipausers >>> Default e-mail domain: ipa.mydomain.com >>> <http://ipa.mydomain.com> >>> Search time limit: 2 >>> Search size limit: 100 >>> User search fields: >>> uid,givenname,sn,telephonenumber,ou,title >>> Group search fields: cn,description >>> Enable migration mode: FALSE >>> Certificate Subject base: O=IPA.MYDOMAIN.COM >>> <http://IPA.MYDOMAIN.COM> >>> Password Expiration Notification (days): 4 >>> Password plugin features: AllowNThash >>> SELinux user map order: >>> guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 >>> Default SELinux user: unconfined_u:s0-s0:c0.c1023 >>> Default PAC types: MS-PAC, nfs:NONE >>> IPA masters: srv01.ipa.mydomain.com >>> <http://srv01.ipa.mydomain.com>;, >>> srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> >>> IPA CA servers: srv01.ipa.mydomain.com >>> <http://srv01.ipa.mydomain.com>;, >>> srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> >>> IPA NTP servers: srv01.ipa.mydomain.com >>> <http://srv01.ipa.mydomain.com>;, >>> srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> >>> *IPA CA renewal master: srv01.ipa.mydomain.com >>> <http://srv01.ipa.mydomain.com>* >>> >>> >>> [root@srv01 ~]# ipactl status >>> Directory Service: RUNNING >>> krb5kdc Service: RUNNING >>> kadmin Service: RUNNING >>> named Service: RUNNING >>> httpd Service: RUNNING >>> ipa-custodia Service: RUNNING >>> ntpd Service: RUNNING >>> pki-tomcatd Service: STOPPED >>> *smb Service: RUNNING >>> winbind Service: RUNNING* >>> ipa-otpd Service: RUNNING >>> ipa-dnskeysyncd Service: RUNNING >>> ipa: INFO: The ipactl command was successful >>> >>> [root@srv02 ~]# ipactl status >>> Directory Service: RUNNING >>> krb5kdc Service: RUNNING >>> kadmin Service: RUNNING >>> named Service: RUNNING >>> httpd Service: RUNNING >>> ipa-custodia Service: RUNNING >>> ntpd Service: RUNNING >>> pki-tomcatd Service: STOPPED >>> ipa-otpd Service: RUNNING >>> ipa-dnskeysyncd Service: RUNNING >>> ipa: INFO: The ipactl command was successful >>> >>> >>> [root@srv01 ~]# certutil -L -d /etc/pki/pki-tomcat/alias >>> Certificate Nickname Trust Attributes >>> SSL,S/MIME,JAR/XPI >>> Server-Cert cert-pki-ca u,u,u >>> subsystemCert cert-pki-ca u,u,u >>> caSigningCert cert-pki-ca *CTu,Cu,Cu* >>> ocspSigningCert cert-pki-ca u,u,u >>> auditSigningCert cert-pki-ca u,u,Pu >>> >>> >>> [root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/alias >>> Certificate Nickname Trust Attributes >>> SSL,S/MIME,JAR/XPI >>> Server-Cert cert-pki-ca u,u,u >>> subsystemCert cert-pki-ca u,u,u >>> caSigningCert cert-pki-ca *CTu,u,u* >>> ocspSigningCert cert-pki-ca u,u,u >>> auditSigningCert cert-pki-ca u,u,Pu/ >>> >>> >>> It seems that AD trust controller role, IPA CA >>> renewal master, smb and windbind are only in the 1st >>> server. >>> And also caSigningCert cert-pki-ca entry is different >>> (CTu,Cu,Cu vs CTu,u,u). >>> >>> I can see only in the 1st server these DNS records: >>> /_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs >>> SRV 0 100 88 srv01 >>> _kerberos._tcp.dc._msdcs SRV 0 100 88 srv01 >>> _kerberos._udp.Default-First-Site-Name._sites.dc._msdcs >>> SRV 0 100 88 srv01 >>> _kerberos._udp.dc._msdcs SRV 0 100 88 srv01 >>> _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 >>> 100 389 srv01 >>> _ldap._tcp.dc._msdcs 0 100 389 srv01/ >>> >>> Srv01 is the first master, I know, but is the server >>> VM that has clock problems, in both situations. >>> So I want to keep srv02 and install a new one. >>> >>> What do I have to do to let the 2nd VM be a single >>> server? >>> Could I use these URLs? >>> / https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_M... >>> https://www.freeipa.org/page/V4/Server_Roles#Upgrade/ >>> >>> >>> 2) uninstall ipa-server from the 1st server (srv01) >>> and then powering off it, assuming that all data into >>> the 2nd one are ok (srv02) >>> >>> 3) update freeipa and all other RPM packages into the >>> VM srv02 >>> >>> 4) install a new fresh VM, always with 7 release, and >>> create a new replica >>> Could I use the same old hostname (srv01) and IP >>> address for this new VM? Or is better to use the same >>> IP but a new name, like srv03? >>> >>> >>> Do you think this is the right way to solve my issue? >>> Or do you have any better idea? >>> >>> Please let me know, thanks. >>> Bye, Morgan >>> >>> _______________________________________________ >>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org <mailto: freeipa-users(a)lists.fedorahosted.org&gt; >>> To unsubscribe send an email tofreeipa-users-leave(a)lists.fedorahosted.org <mailto: freeipa-users-leave(a)lists.fedorahosted.org&gt; >>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... >> >> _______________________________________________ >> FreeIPA-users mailing list -- >> freeipa-users(a)lists.fedorahosted.org >> <mailto:freeipa-users@lists.fedorahosted.org> >> To unsubscribe send an email to >> freeipa-users-leave(a)lists.fedorahosted.org >> <mailto:freeipa-users-leave@lists.fedorahosted.org> >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... >> >> >> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org <mailto: freeipa-users(a)lists.fedorahosted.org&gt; >> To unsubscribe send an email tofreeipa-users-leave(a)lists.fedorahosted.org <mailto: freeipa-users-leave(a)lists.fedorahosted.org&gt; >> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > _______________________________________________ > FreeIPA-users mailing list -- > freeipa-users(a)lists.fedorahosted.org > <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to > freeipa-users-leave(a)lists.fedorahosted.org > <mailto:freeipa-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho... > _______________________________________________ FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...