Hello,
we got to a point when we have Travis CI tests pass both with FreeIPA containers under docker, and under podman:
https://travis-ci.org/freeipa/freeipa-container/branches
I'm now trying to decide which setup we should run in CI going forward.
Each test takes about 30 minutes and while we run 8 of them lately, at least two more are turned off (rawhide due to https://bugzilla.redhat.com/show_bug.cgi?id=1747901 and read-write Fedora 30 due to https://bugzilla.redhat.com/show_bug.cgi?id=1755643), and we will want to add CentOS 8 to the mix once the base container images are out. So we are close to a dozen of tests. We run internal + external CA at least for one stable Fedora release, and I like to keep the read-write setup in the mix if possible, as it allows us to find new stuff that should go to persistent storage with docker diff (which would obviously not work on read-only container).
Multiplying that by two, testing both docker and podman, does not seem practical.
We could save some time by running tests/run-master-and-replica.sh first and only running tests/run-partial-tests.sh when the first one fails. On the other hand, we currently do not run cross version upgrades in CI, we just force the upgrade code path but with the same image. So if we added that, we'd increase the needed runtime again.
Docker seems dead on Fedoras as the moby-engine's maintenance stopped in the middle of July:
https://bugzilla.redhat.com/buglist.cgi?classification=Fedora&component=...
On the other hand, judging by the issues that we've seen reported, people run the FreeIPA server containers on other OSes and there dockers in various versions are still more common than podman.
We could save on the running time by moving some of the tests to other CI like CircleCI but I like having overview of all CI tests on one place, and another CI system would break that. On the other hand, for ARM we will likely need to add Shippable to the mix anyway.
I'd appreciate your opinions and suggestions about testing the FreeIPA container upstream going forward.