I located every entry in LDAP that referenced the failed server and removed each of them.  I know that the entries in the etc ipa masters hierarchies wouldn't go until I'd removed several of the others, which know included the custodia entries.  I think there weren't any topology entries by that point.

Sorry not to be more helpful...

On Tue, Jan 8, 2019 at 5:12 PM Rob Crittenden <rcritten@redhat.com> wrote:
K. M. Peterson via FreeIPA-users wrote:
> I'm going to reply to myself, after several more hours of digging, I
> discovered that although it wasn't true at the time I posted the above
> question, eventually, as with the original post from Lachlan Musicman
> <https://lists.fedorahosted.org/archives/users/46343247263810572257541459042951629750/>,
> the WebUI died, and that meant no self-service for the rest of the
> team.  And that made it into an emergency.
>
> So, I fired up my LDAP editor (I've been using JXWorkBench) and went to
> eradicate all the traces of the failed replica.  Which fixed the issue;
> and I'm fairly sure there aren't any lingering effects.  I think.
>
> But this was the first time I've used the editor to actual effect any
> changes to things; and I'm going to post the underlying question that
> raised in a new thread...
>
> This seems to have bitten at least a few of us; I'd be happy to know how
> to file a bug if there's a useful contribution there.  Thanks!

You didn't happen to keep a list of the entries/values you removed did you?

rob

>
> On Sat, Jan 5, 2019 at 4:47 PM K. M. Peterson <kmp.lists@gmail.com
> <mailto:kmp.lists@gmail.com>> wrote:
>
>     Hate _hate_ to open old threads, but...
>
>     I'm also seeing this.  I've been trying to add another replica to
>     our topology (this would be on a different subnet than the current
>     pair); the ipa-replica-install command has been failing for various
>     reasons that I've been fixing or circumventing and I've just been
>     re-spinning the new server between each attempt to keep the
>     environment clean.  The latest death was apparently because of an
>     issue with /etc/openldap/ldap.conf which I was debugging and was
>     about to remove the server from IPA and reset it.
>
>     However, I'm not able to do so.  All attempts are met with "ERROR:
>     invalid 'PKINIT enabled server': all masters must have IPA master
>     role enabled" - in fact, even poking around trying to do an ipa
>     config-show  (on either of the current masters) just generates that
>     error.  I've also tried uninstalling the replica and client on the
>     new host, and it seems to have completed successfully, but I can't
>     re-enroll it either, so it's "dead to the other masters", except...   
>
>     There is nothing I want to do at this point other than another
>     iteration on my problem adding another replica.  There's no data on
>     replica, nothing is relying on it, and I've tried as hard as
>     possible to make the installation entirely vanilla.  I haven't
>     manually enabled PKINIT; ipa-pkinit-manage status on the current
>     masters says it's enabled.  As for the server roles,
>     server-role-find shows the two current servers and the new one; the
>     latter's "role status" for CA Server is "absent".  I've had issues
>     before where I've had to enumerate the RUVs and remove them (done
>     that).  Just want the references to this to go away, so that I can
>     keep working towards the most minimal and concise installation.
>
>     Any ideas on where I can go to get out of this situation?  Many thanks!
>
>     (Everything completely updated to *4.6.4-10.el7.centos, initial
>     installation was about one year ago, domain level 1; tried all the
>     ipa server del and ipa-replica-manage del suggestions which aren't
>     working for me this time, no AD integration...)
>
>     On Tue, Nov 20, 2018 at 1:48 AM Brian Topping via FreeIPA-users
>     <freeipa-users@lists.fedorahosted.org
>     <mailto:freeipa-users@lists.fedorahosted.org>> wrote:
>
>         Oh, forgot to mention, current domain level is `1`...
>         _______________________________________________
>         FreeIPA-users mailing list --
>         freeipa-users@lists.fedorahosted.org
>         <mailto:freeipa-users@lists.fedorahosted.org>
>         To unsubscribe send an email to
>         freeipa-users-leave@lists.fedorahosted.org
>         <mailto:freeipa-users-leave@lists.fedorahosted.org>
>         Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>         List Guidelines:
>         https://fedoraproject.org/wiki/Mailing_list_guidelines
>         List Archives:
>         https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>