On Thu, 23 Apr 2020 at 16:37, Kevin Fenzi <kevin(a)scrye.com> wrote:
On Thu, Apr 23, 2020 at 11:46:25AM +0200, Pierre-Yves Chibon wrote:
> Good Morning Everyone,
>
> Kevin and I had a discussion on IRC the other day about mirroring the
ansible
> repo.
> The options we considered are:
...snip...
>
> For a first step I went with a third approach: a small python service
that
> runs every 3 minutes (configurable): git fetch && git fsck (to ensure
the git
> is in a correct state). It actually doesn't run every 3 minutes, it
calls git,
> then wait for 3 minutes then calls git again and so on. So if git was
ever to
> takes 4 minutes to run, there is no risk of the program stepping on
its own
> toes.
ok.
I'm wondering if that might be overkill.
Really if pagure.io blows up, we want to be able to look at the repo we
have on batcave01 and be able to still operate while we recover
pagure.io. If someone pushed or merged some commits in between the last
time we synced and blowup, someone should have them in their local repo
right? so we could just push them to the repo on batcave and be back
completly in sync, no?
but I guess it doesn't hurt to do every 3minutes, just some BW.
Another option I mentioned the other day when we were talking, but I
didnt really explain well was that we could wrap ansible-playbook into a
wrapper that runs a git pull then ansible-playbook... so it would always
be in sync at first you run it. That could be messy tho if there's
another process doing pulls, etc. So I guess scrap that idea.
If we were to do that I would put in a lock logic like we do with our
rsync's and such.
New workflow, everyone uses rbac-playbook and we add some lock code to
check, make a lockfile, do the git pull, run the playbook, remove the
lockfile.
Check to see if a lockfile exists exit unless a --dontcare was put in place
at which point it could do somehting like:
if lockfile is younger than 10 minutes, assume that the plays are fresh
enough and use that.
if lockfile is older than 10 minutes, exit saying "we seem to have a dead
git pull going, please manually check what the problem is"
And I think that wont' work well for our super long ansible runs... I am
beginning to think we have an unsolvable problem here and everything will
suck somewhere.
--
Stephen J Smoogen.