I have 4 IPA servers, all masters, that were previously configured in a "full mesh" replication. 2 in "prod", 2 in "preprod". While trying to fix a replication issue, I accidentally did a: ipa-replica-manage del on one of the prod servers for BOTH preprod servers.
Now, the prod servers don't "see" either of the preprod servers, so I effectively created a "split-brain" between the 2 environments. Preprod still "knows about" the prod ipa servers, but I can't figure out how to re-establish the replication agreements.
I was about to just blow away the preprod servers and rebuild them (which i did before on one of them) but noticed one of them has the "KRA" role, and it is the only one in the domain that has it. I don't know what that does, or what the effects would be if it went away. I'm guessing bad.
I have tried "ipa topologysegment-reinitialize domain" on the segments that preprod still has, but those segments did not show up in prod. ipa topologysuffix-verify domain says "in order" everywhere.
At this point I am completely lost on how to proceed.
What details can I provide for any help anyone is willing to provide?
ok, did a little googling, and seems like KRA refers to the "vault" feature? I didn't originally install this myself, so wasn't sure if it is used for anything critical. I ran: # ipa vault-find ---------------- 0 vaults matched ---------------- ---------------------------- Number of entries returned 0 ----------------------------
So, can I assume it is safe to blow away and rebuild the server that has this role?
On Wed, Jan 31, 2018 at 3:56 PM, Rob Brown dtownrobbrown@gmail.com wrote:
I have 4 IPA servers, all masters, that were previously configured in a "full mesh" replication. 2 in "prod", 2 in "preprod". While trying to fix a replication issue, I accidentally did a: ipa-replica-manage del on one of the prod servers for BOTH preprod servers.
Now, the prod servers don't "see" either of the preprod servers, so I effectively created a "split-brain" between the 2 environments. Preprod still "knows about" the prod ipa servers, but I can't figure out how to re-establish the replication agreements.
I was about to just blow away the preprod servers and rebuild them (which i did before on one of them) but noticed one of them has the "KRA" role, and it is the only one in the domain that has it. I don't know what that does, or what the effects would be if it went away. I'm guessing bad.
I have tried "ipa topologysegment-reinitialize domain" on the segments that preprod still has, but those segments did not show up in prod. ipa topologysuffix-verify domain says "in order" everywhere.
At this point I am completely lost on how to proceed.
What details can I provide for any help anyone is willing to provide?
Though you can completely rebuild preprod servers, still it would be interesting how to reconnect prod servers with replicas again.
2018-02-01 8:41 GMT+03:00 Rob Brown via FreeIPA-users < freeipa-users@lists.fedorahosted.org>:
ok, did a little googling, and seems like KRA refers to the "vault" feature? I didn't originally install this myself, so wasn't sure if it is used for anything critical. I ran:
# ipa vault-find
0 vaults matched
Number of entries returned 0
So, can I assume it is safe to blow away and rebuild the server that has this role?
On Wed, Jan 31, 2018 at 3:56 PM, Rob Brown dtownrobbrown@gmail.com wrote:
I have 4 IPA servers, all masters, that were previously configured in a "full mesh" replication. 2 in "prod", 2 in "preprod". While trying to fix a replication issue, I accidentally did a: ipa-replica-manage del on one of the prod servers for BOTH preprod servers.
Now, the prod servers don't "see" either of the preprod servers, so I effectively created a "split-brain" between the 2 environments. Preprod still "knows about" the prod ipa servers, but I can't figure out how to re-establish the replication agreements.
I was about to just blow away the preprod servers and rebuild them (which i did before on one of them) but noticed one of them has the "KRA" role, and it is the only one in the domain that has it. I don't know what that does, or what the effects would be if it went away. I'm guessing bad.
I have tried "ipa topologysegment-reinitialize domain" on the segments that preprod still has, but those segments did not show up in prod. ipa topologysuffix-verify domain says "in order" everywhere.
At this point I am completely lost on how to proceed.
What details can I provide for any help anyone is willing to provide?
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Agreed! I would love to know if that is possible... seems like it should be. As mentioned previously, preprod still has the agreements, but prod does not. Not really sure how I should proceed. I'm a bit stuck, not wanting to further break anything. For now, auth is still working in both envs. --- [root@ipa-preprod-1201]# ipa topologysegment-find domain ------------------ 5 segments matched ------------------ Segment name: ipa-preprod-1201-to-ipa-preprod-1202 Left node: ipa-preprod-1201 Right node: ipa-preprod-1202 Connectivity: both
Segment name: ipa-preprod-1201-to-ipa-prod-1201 Left node: ipa-preprod-1201 Right node: ipa-prod-1201 Connectivity: both
Segment name: ipa-preprod-1202-to-ipa-prod-1201 Left node: ipa-preprod-1202 Right node: ipa-prod-1201 Connectivity: both
Segment name: ipa-prod-1201-to-ipa-prod-1202 Left node: ipa-prod-1201 Right node: ipa-prod-1202 Connectivity: both
Segment name: ipa-prod-1202-to-ipa-preprod-1201 Left node: ipa-prod-1202 Right node: ipa-preprod-1201 Connectivity: both
[root@ipa-prod-1201]# ipa topologysegment-find domain ------------------ 2 segments matched ------------------ Segment name: ipa-preprod-1201-to-ipa-preprod-1202 Left node: ipa-preprod-1201 Right node: ipa-preprod-1202 Connectivity: both
Segment name: ipa-prod-1201-to-ipa-prod-1202 Left node: ipa-prod-1201 Right node: ipa-prod-1202 Connectivity: both ---------------------------- Number of entries returned 2 ----------------------------
I think part of the problem is that when I did the ipa-replica-manage del, it removed the preprod servers:
[root@ipa-prod-1201]# ipa server-find --------------------- 2 IPA servers matched --------------------- Server name: ipa-prod-1201 Min domain level: 0 Max domain level: 1
Server name: ipa-prod-1202 Min domain level: 0 Max domain level: 1 ---------------------------- Number of entries returned 2 ----------------------------
but they still exist on the preprod side:
[root@ipa-preprod-1201]# ipa server-find --------------------- 4 IPA servers matched --------------------- Server name: ipa-preprod-1201 Min domain level: 0 Max domain level: 1
Server name: ipa-preprod-1202 Min domain level: 0 Max domain level: 1
Server name: ipa-prod-1201 Min domain level: 0 Max domain level: 1
Server name: ipa-prod-1202 Min domain level: 0 Max domain level: 1 ---------------------------- Number of entries returned 4 ----------------------------
On Wed, Jan 31, 2018 at 10:52 PM, Andrew Radygin randrewg@gmail.com wrote:
Though you can completely rebuild preprod servers, still it would be interesting how to reconnect prod servers with replicas again.
2018-02-01 8:41 GMT+03:00 Rob Brown via FreeIPA-users < freeipa-users@lists.fedorahosted.org>:
ok, did a little googling, and seems like KRA refers to the "vault" feature? I didn't originally install this myself, so wasn't sure if it is used for anything critical. I ran:
# ipa vault-find
0 vaults matched
Number of entries returned 0
So, can I assume it is safe to blow away and rebuild the server that has this role?
On Wed, Jan 31, 2018 at 3:56 PM, Rob Brown dtownrobbrown@gmail.com wrote:
I have 4 IPA servers, all masters, that were previously configured in a "full mesh" replication. 2 in "prod", 2 in "preprod". While trying to fix a replication issue, I accidentally did a: ipa-replica-manage del on one of the prod servers for BOTH preprod servers.
Now, the prod servers don't "see" either of the preprod servers, so I effectively created a "split-brain" between the 2 environments. Preprod still "knows about" the prod ipa servers, but I can't figure out how to re-establish the replication agreements.
I was about to just blow away the preprod servers and rebuild them (which i did before on one of them) but noticed one of them has the "KRA" role, and it is the only one in the domain that has it. I don't know what that does, or what the effects would be if it went away. I'm guessing bad.
I have tried "ipa topologysegment-reinitialize domain" on the segments that preprod still has, but those segments did not show up in prod. ipa topologysuffix-verify domain says "in order" everywhere.
At this point I am completely lost on how to proceed.
What details can I provide for any help anyone is willing to provide?
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedo rahosted.org
-- Best regards, Andrew.
BTW: [root@ipa-prod-1201]# cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) [root@ipa-prod-1201]# rpm -qa|grep ipa-server-4 ipa-server-4.4.0-14.el7.centos.6.x86_64
On Thu, Feb 1, 2018 at 10:53 AM, Rob Brown dtownrobbrown@gmail.com wrote:
Agreed! I would love to know if that is possible... seems like it should be. As mentioned previously, preprod still has the agreements, but prod does not. Not really sure how I should proceed. I'm a bit stuck, not wanting to further break anything. For now, auth is still working in both envs.
[root@ipa-preprod-1201]# ipa topologysegment-find domain
5 segments matched
Segment name: ipa-preprod-1201-to-ipa-preprod-1202 Left node: ipa-preprod-1201 Right node: ipa-preprod-1202 Connectivity: both
Segment name: ipa-preprod-1201-to-ipa-prod-1201 Left node: ipa-preprod-1201 Right node: ipa-prod-1201 Connectivity: both
Segment name: ipa-preprod-1202-to-ipa-prod-1201 Left node: ipa-preprod-1202 Right node: ipa-prod-1201 Connectivity: both
Segment name: ipa-prod-1201-to-ipa-prod-1202 Left node: ipa-prod-1201 Right node: ipa-prod-1202 Connectivity: both
Segment name: ipa-prod-1202-to-ipa-preprod-1201 Left node: ipa-prod-1202 Right node: ipa-preprod-1201 Connectivity: both
[root@ipa-prod-1201]# ipa topologysegment-find domain
2 segments matched
Segment name: ipa-preprod-1201-to-ipa-preprod-1202 Left node: ipa-preprod-1201 Right node: ipa-preprod-1202 Connectivity: both
Segment name: ipa-prod-1201-to-ipa-prod-1202 Left node: ipa-prod-1201 Right node: ipa-prod-1202 Connectivity: both
Number of entries returned 2
I think part of the problem is that when I did the ipa-replica-manage del, it removed the preprod servers:
[root@ipa-prod-1201]# ipa server-find
2 IPA servers matched
Server name: ipa-prod-1201 Min domain level: 0 Max domain level: 1
Server name: ipa-prod-1202 Min domain level: 0 Max domain level: 1
Number of entries returned 2
but they still exist on the preprod side:
[root@ipa-preprod-1201]# ipa server-find
4 IPA servers matched
Server name: ipa-preprod-1201 Min domain level: 0 Max domain level: 1
Server name: ipa-preprod-1202 Min domain level: 0 Max domain level: 1
Server name: ipa-prod-1201 Min domain level: 0 Max domain level: 1
Server name: ipa-prod-1202 Min domain level: 0 Max domain level: 1
Number of entries returned 4
On Wed, Jan 31, 2018 at 10:52 PM, Andrew Radygin randrewg@gmail.com wrote:
Though you can completely rebuild preprod servers, still it would be interesting how to reconnect prod servers with replicas again.
2018-02-01 8:41 GMT+03:00 Rob Brown via FreeIPA-users < freeipa-users@lists.fedorahosted.org>:
ok, did a little googling, and seems like KRA refers to the "vault" feature? I didn't originally install this myself, so wasn't sure if it is used for anything critical. I ran:
# ipa vault-find
0 vaults matched
Number of entries returned 0
So, can I assume it is safe to blow away and rebuild the server that has this role?
On Wed, Jan 31, 2018 at 3:56 PM, Rob Brown dtownrobbrown@gmail.com wrote:
I have 4 IPA servers, all masters, that were previously configured in a "full mesh" replication. 2 in "prod", 2 in "preprod". While trying to fix a replication issue, I accidentally did a: ipa-replica-manage del on one of the prod servers for BOTH preprod servers.
Now, the prod servers don't "see" either of the preprod servers, so I effectively created a "split-brain" between the 2 environments. Preprod still "knows about" the prod ipa servers, but I can't figure out how to re-establish the replication agreements.
I was about to just blow away the preprod servers and rebuild them (which i did before on one of them) but noticed one of them has the "KRA" role, and it is the only one in the domain that has it. I don't know what that does, or what the effects would be if it went away. I'm guessing bad.
I have tried "ipa topologysegment-reinitialize domain" on the segments that preprod still has, but those segments did not show up in prod. ipa topologysuffix-verify domain says "in order" everywhere.
At this point I am completely lost on how to proceed.
What details can I provide for any help anyone is willing to provide?
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedo rahosted.org
-- Best regards, Andrew.
freeipa-users@lists.fedorahosted.org
-
Andrew Radygin -
Rob Brown