On Fri, Apr 17, 2020 at 8:07 AM Clement Verna <cverna(a)fedoraproject.org&gt; wrote: > Hi all, > > I wanted to start a discussion and possibly some work around automating > the manual tasks involved in the release engineering work. > > In particular I have in mind the following tasks : > - processing the unretire package tickets. > - processing the requests for a new package or a new branch. > - container base image release. > > Instead of looking at each of these individually I was thinking that it > might be interesting to look at having an automation framework or something > like a bot that could be flexible enough to add more tasks or process in > the future. > > To do that we have different possibilities, one could be to build a bot > that has a similar architecture than the compose-tracker ( > https://pagure.io/releng/compose-tracker) ie a fedora-messaging consumer > processing messages. > Another option is to use loopabull ( > https://github.com/maxamillion/loopabull) to trigger ansible playbook on > fedora-messaging messages. > > Both solutions are quite similar, but one (loopabull) provides already > the boilerplate to trigger a script or a function based on messages > received (a bit like AWS lambda or other serverless framework). We also > have already a few process automated that way ( > https://pagure.io/Fedora-Infra/loopabull-tasks). > So I believe that using loopabull might be the best way forward, but I > would be interested to hear about other ideas :-) > I would lean to use loopabull because it already works in a "reactive way" with mq messages and we don't need to write a full application since we will be using ansible (which still can be "extended" developing modules for complex tasks) - the above script could become a couple of ansible modules to be used in a playbook with loopabull. > > Now if we look at the tasks to automate, I was thinking that we could > implement that automation in different phases : > > *un-retiring tickets*: > > - First step would be to run automatically the check-unretirement > script ( > https://pagure.io/releng/blob/master/f/scripts/check-unretirement.py) > and redirect its output into the ticket comments. That way people > processing the ticket have all the information available in the ticket. > - Second step would be to process automatically the tickets that do > not require a new bz review (retired less than 8 weeks ago) > - Finally see if we can process automatically all the tickets. > > > *creating new dist-git repo or branches:* > > - Idea here would be to run the fedscm-admin ( > https://pagure.io/fedscm-admin) cli automatically when a new ticket > is created here (https://pagure.io/releng/fedora-scm-requests). > > I think it would also need to check for missed tickets from time to time, like every 5 minutes for example, in case it missed one for whatever reason. > *container base image release:* > > - Instead of building the image every night, we could rebuild the > image only when at least 1 package present in the base image has been > updated. > - Push the image weekly to the registry if a build happened during > that week. > > Is there any particular reason for using a weekly push schedule (assuming doing one on every build is way too much)?

> > > Again here are some ideas, and I would very much appreciate feedback or > other ideas :-). Also if you think about another process that could be > automated that way please share it :-). > > Thanks > Clément > > _______________________________________________ > rel-eng mailing list -- rel-eng(a)lists.fedoraproject.org > To unsubscribe send an email to rel-eng-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/rel-eng@lists.fedoraproject... > -- Leonardo Rossetti Senior Software Engineer, Red Hat <https://www.redhat.com> <https://www.redhat.com>

On Fri, Apr 17, 2020 at 09:22:27AM -0300, Leonardo Rossetti wrote: > On Fri, Apr 17, 2020 at 8:07 AM Clement Verna > <[1]cverna(a)fedoraproject.org&gt; wrote: > > Hi all, > I wanted to start a discussion and possibly some work around automating > the manual tasks involved in the release engineering work. > In particular I have in mind the following tasks : > - processing the unretire package tickets. > - processing the requests for a new package or a new branch. > - container base image release. > Instead of looking at each of these individually I was thinking that it > might be interesting to look at having an automation framework or > something like a bot that could be flexible enough to add more tasks or > process in the future. > To do that we have different possibilities, one could be to build a bot > that has a similar architecture than the compose-tracker > ([2]https://pagure.io/releng/compose-tracker) ie a fedora-messaging > consumer processing messages. > Another option is to use loopabull > ([3]https://github.com/maxamillion/loopabull) to trigger ansible > playbook on fedora-messaging messages. > Both solutions are quite similar, but one (loopabull) provides already > the boilerplate to trigger a script or a function based on messages > received (a bit like AWS lambda or other serverless framework). We also > have already a few process automated that way > ([4]https://pagure.io/Fedora-Infra/loopabull-tasks). > So I believe that using loopabull might be the best way forward, but I > would be interested to hear about other ideas :-) > > I would lean to use loopabull because it already works in a "reactive way" > with mq messages and we don't need to write a full application since we > will be using ansible (which still can be "extended" developing modules > for complex tasks) - the above script could become a couple of ansible > modules to be used in a playbook with loopabull. I like loopabull, having been pretty much the only one playing with it since Adam, I think it's a nice and fine tool and we should try leveraging it. There is one angle where it isn't straight forward to use, it's secrets. Currently the API tokens the scripts are using are passed as CLI argument when calling the script. If we end up needing something like kerberos keytab for example, we may have to think how to do this and evaluate if loopabull is still a good fit. Pierre _______________________________________________ rel-eng mailing list -- rel-eng(a)lists.fedoraproject.org To unsubscribe send an email to rel-eng-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/rel-eng@lists.fedoraproject...

On Fri, Apr 17, 2020 at 03:14:56PM +0200, Pierre-Yves Chibon wrote: > On Fri, Apr 17, 2020 at 09:22:27AM -0300, Leonardo Rossetti wrote: > > On Fri, Apr 17, 2020 at 8:07 AM Clement Verna > > <[1]cverna(a)fedoraproject.org&gt; wrote: > > > > Hi all, > > I wanted to start a discussion and possibly some work around automating > > the manual tasks involved in the release engineering work. > > In particular I have in mind the following tasks : > >  - processing the unretire package tickets. > >  - processing the requests for a new package or a new branch. > >  - container base image release. > > Instead of looking at each of these individually I was thinking that it > > might be interesting to look at having an automation framework or > > something like a bot that could be flexible enough to add more tasks or > > process in the future. > > To do that we have different possibilities, one could be to build a bot > > that has a similar architecture than the compose-tracker > > ([2]https://pagure.io/releng/compose-tracker) ie a fedora-messaging > > consumer processing messages. > > Another option is to use loopabull > > ([3]https://github.com/maxamillion/loopabull) to trigger ansible > > playbook on fedora-messaging messages. > > Both solutions are quite similar, but one (loopabull) provides already > > the boilerplate to trigger a script or a function based on messages > > received (a bit like AWS lambda or other serverless framework). We also > > have already a few process automated that way > > ([4]https://pagure.io/Fedora-Infra/loopabull-tasks). > > So I believe that using loopabull might be the best way forward, but I > > would be interested to hear about other ideas :-) > > > > I would lean to use loopabull because it already works in a "reactive way" > > with mq messages and we don't need to write a full application since we > > will be using ansible (which still can be "extended" developing modules > > for complex tasks) - the above script could become a couple of ansible > > modules to be used in a playbook with loopabull. > > I like loopabull, having been pretty much the only one playing with it since > Adam, I think it's a nice and fine tool and we should try leveraging it. > There is one angle where it isn't straight forward to use, it's secrets. > Currently the API tokens the scripts are using are passed as CLI argument when > calling the script. > If we end up needing something like kerberos keytab for example, we may have to > think how to do this and evaluate if loopabull is still a good fit. So another issue around loopabull is that I think Maxamillion said the other day that he's not really been maintining it upstream, so we might have to help out there if we want to keep using it. >> Would it be possible to use ansible vault files for that (using some kind >> of internal ansible API)? Possibly, but it would mean it would need secrets to unlock the vault and we don't use valut for anything else currently. ;(

I wonder how hard it would be to move loopabull into openshift... that might also help the secrets thing by allowing it to keep things in openshift secrets...

Is loopabull moved to fedora-messaging? Or still fedmsg?

On Sun, 19 Apr 2020 at 20:24, Pierre-Yves Chibon <[1]pingou(a)pingoured.fr&gt; wrote: .... snip > > Possibly, but it would mean it would need secrets to unlock the vault > and we don't use valut for anything else currently. ;( That almost applies to loopabull as well ;-) (Well we do use it, but not much) > I wonder how hard it would be to move loopabull into openshift... that > might also help the secrets thing by allowing it to keep things in > openshift secrets... I wonder if there is a benefit from using loopabull at that point, having a simple fedora-messaging consumer running in openshift would work just as well. Yeah, also fedora-messaging makes it quite easy to setup the consumer and there is not much boilerplate needed. So we could consolidate on 1 "bot" consumer that listen to all the messages we are interested in and then triggers different tasks. I don't have a strong opinion, I was just thinking that maybe loopabull would mean less maintenance on the longer term (ie focusing on witting the tasks and tests for these tasks) but maybe the more we use it the more we will have to touch loopabull upstream.

> Is loopabull moved to fedora-messaging? Or still fedmsg? I have ported it to fedora-messaging, it only needed a couple of tweaks from the existing amqp plugin. I may still need a release. I know we talked about it with Adam on IRC not long ago, but looking at github it looks like there was no recent release made. So the version deployed in the infra is still using fedmsg ?