I run a 4-node IPA cluster on AWS spanning a few global regions and tied into a particularly complex AD forest -- never had the DNS issues you mention but I've never had to talk to IPA on-prem either. And our setup is strange as we could never get the global AD admins to make DNS entries for us among other issues so we ended up choosing a totally new TLD domain name to run IPA on and bind our servers against; this works fine except we can't leverage kerberos based features because our realm is different from our domain.  All we needed was RBAC login using AD usernames and passwords though so our needs are met. For DNS on AWS though ... - You can override or alter DHCP level nameserver assignments at a VPC-level if you want to adjust DHCP option sets. This  is a big decision as it is VPC-wide in scope but works well to push out "global" DNS and nameserver preferences - If VPC level changes are too much then it's pretty easy on an EC2 instance to override or alter DNS resolution, nameservers and search order in ways that persist beyond reboot. Specific method depends OS you are using and if you want to this on the CLI via ansible or manual operations or if you want to do this at the cloud-init "just booting up" level I did have a nasty DNS issue in the past with an old school client that believed all DNS should be served out of AD and that pre-creating reverse PTR records was "nonsense" and "not needed" which broke a ton of AWS HPC and EMR use cases and other projects where software does a reverse lookup on it's IP to learn more about its self.  In that setting we built our own custom DNS server based on DNSMasq -- DNSMasq is perfect because you can construct a totally bespoke DNS environment combining delegated nameservers, local name servers and you can even have DNSMasq read DNS entries from an /etc/hosts style file so you don't have to maintain a complex bind-format zone file I don't know the exact scope of your situation but this seems like a case for overriding the DNS settings on the EC2 hosts running IPA to talk to your colo nameservers rather than the AWS designated ones that show up via your DHCP Option Set Chris William Muriithi via FreeIPA-users wrote on 5/27/20 10:30 AM:

Replies inline ... Yep! Our oddball IPA setup looks like this, due to a combination of senior IT leadership being frightened of the cloud and global active directory domain admins doing their usual global admin thing and refusing to even take meetings with mortals. So we had to self-support and only got a grudging 1-way trust set with the domain after escalating to very high levels. It works though.    AD Domain (complex multi-child forest)  == company-x.com with child domains at  nafta.company-x.com, apac.company-x.com etc. etc.    Our FreeIPA domain name is == company-x-ipa.com So our REALM is COMPANY-X-IPA and that is where we bind our client machines to when enrolling but we set the local domain name and hostname to "company-x.com" when we enroll the host This works well but from what I've been told we lose the ability to do things like kerberized authentication for filessytem mounts etc. -- since all we really needed was RBAC managed SSH login with a small smattering of custom sudo roles and a bunch of local SSH key storage it works fine for us. Yeah most of this DNS is on-premise; again because of a corporate desire to use AD for DNS. We basically point all our nameservers at the on-prem domain controllers and don't use FreeIPA for DNS records really much at all (even the _SVR records that enable FreeIPA auto-discovery) -- but no network, DNS query or technical issues at all with DNS being outside of AWS -- works fine for us even DNS queries made on the IPA masters.   For rare cases where that did not work (HPC use cases)  we did the DNSMasq thing so we could inject custom reverse DNS responses while still delegating other queries back to the domain controllers. Our VPC is direct connected back to the corporate WAN; no NAT on the private AWS subnets so it would be flat/private.  Lots of security controls and firewalls doing SSL intercept and decrypt at edges that blocked

Hi Rafael, Possibly this is the case. I will try to create a fresh IPA as the realm will change and will update the list on the result. Then figure out how to migrate the data manually. Something different. For example, I setup an IPA client, with the server on the co-location - over VPN. The ipa-client enrol fine, and I can even ssh in. Cool, 4 days later, no change, we can't authenticate. When debugging, all looks like a DNS issue, but oddly, it had worked a couple of day earlier. Was assuming AWS infrastructure are hijacking DNS traffic as the rest - ssh using IP address still works from AWS to co-location over VPN. Thanks again Rafael. William