Ideas for more releng automation
by Clement Verna
Hi all,
I wanted to start a discussion and possibly some work around automating the
manual tasks involved in the release engineering work.
In particular I have in mind the following tasks :
- processing the unretire package tickets.
- processing the requests for a new package or a new branch.
- container base image release.
Instead of looking at each of these individually I was thinking that it
might be interesting to look at having an automation framework or something
like a bot that could be flexible enough to add more tasks or process in
the future.
To do that we have different possibilities, one could be to build a bot
that has a similar architecture than the compose-tracker (
https://pagure.io/releng/compose-tracker) ie a fedora-messaging consumer
processing messages.
Another option is to use loopabull (https://github.com/maxamillion/loopabull)
to trigger ansible playbook on fedora-messaging messages.
Both solutions are quite similar, but one (loopabull) provides already the
boilerplate to trigger a script or a function based on messages received (a
bit like AWS lambda or other serverless framework). We also have already a
few process automated that way (
https://pagure.io/Fedora-Infra/loopabull-tasks).
So I believe that using loopabull might be the best way forward, but I
would be interested to hear about other ideas :-)
Now if we look at the tasks to automate, I was thinking that we could
implement that automation in different phases :
*un-retiring tickets*:
- First step would be to run automatically the check-unretirement script
(https://pagure.io/releng/blob/master/f/scripts/check-unretirement.py)
and redirect its output into the ticket comments. That way people
processing the ticket have all the information available in the ticket.
- Second step would be to process automatically the tickets that do not
require a new bz review (retired less than 8 weeks ago)
- Finally see if we can process automatically all the tickets.
*creating new dist-git repo or branches:*
- Idea here would be to run the fedscm-admin (
https://pagure.io/fedscm-admin) cli automatically when a new ticket is
created here (https://pagure.io/releng/fedora-scm-requests).
*container base image release:*
- Instead of building the image every night, we could rebuild the image
only when at least 1 package present in the base image has been updated.
- Push the image weekly to the registry if a build happened during that
week.
Again here are some ideas, and I would very much appreciate feedback or
other ideas :-). Also if you think about another process that could be
automated that way please share it :-).
Thanks
Clément
3 years, 11 months
FBR: add sysadmin-analysis to bastion
by Stephen John Smoogen
The sysadmin-analysis group is used on data-analysis01 for people to log in
and work there. Because everyone in that group was in an existing group, I
forgot to add it to bastion. However a new person joined and they don't
have access.
[smooge@batcave01 ansible (master)]$ git diff
diff --git a/inventory/group_vars/bastion b/inventory/group_vars/bastion
index 8e63d1a..160f1d1 100644
--- a/inventory/group_vars/bastion
+++ b/inventory/group_vars/bastion
@@ -23,7 +23,7 @@ custom_rules: [
# TODO - remove modularity-wg membership here once it is not longer needed:
# https://fedorahosted.org/fedora-infrastructure/ticket/5363
-fas_client_groups:
sysadmin-ask,sysadmin-atomic,sysadmin-web,sysadmin-main,sysadmin-cvs,sysadmin-noc,sysadmin-releng,sysadmin-dba,sysadmin-hosted,sysadmin-tools,sysadmin-spin,sysadmin-cloud,fi-apprentice,sysadmin-badges,sysadmin-troubleshoot,sysadmin-qa,sysadmin-centos,sysadmin-ppc,sysadmin-koschei,sysadmin-secondary,sysadmin-fedimg,sysadmin-veteran,sysadmin-mbs,modularity-wg,pungi-devel,sysadmin-upstreamfirst,sysadmin-releasemonitoring,sysadmin-gnome,sysadmin-copr,sysadmin-coreos,sysadmin-dbgserver,sysadmin-osbs,sysadmin-odcs
+fas_client_groups: sysadmin-analysis,
sysadmin-ask,sysadmin-atomic,sysadmin-web,sysadmin-main,sysadmin-cvs,sysadmin-noc,sysadmin-releng,sysadmin-dba,sysadmin-hosted,sysadmin-tools,sysadmin-spin,sysadmin-cloud,fi-apprentice,sysadmin-badges,sysadmin-troubleshoot,sysadmin-qa,sysadmin-centos,sysadmin-ppc,sysadmin-koschei,sysadmin-secondary,sysadmin-fedimg,sysadmin-veteran,sysadmin-mbs,modularity-wg,pungi-devel,sysadmin-upstreamfirst,sysadmin-releasemonitoring,sysadmin-gnome,sysadmin-copr,sysadmin-coreos,sysadmin-dbgserver,sysadmin-osbs,sysadmin-odcs
#
# This is a postfix gateway. This will pick up gateway postfix config in
base
--
Stephen J Smoogen.
3 years, 11 months
FBR: To update ostree on bodhi-backend01, branched-composer, rawhide-composer
by Mohan Boddu
Hi,
Randomly we hit an issue in "ostree pull-local" command that it wont
sync and throws out a traceback. The new ostree build adds more
logging and should help us with concurrently pulling.
Both rawhide-composer.phx2.fp.o and branched-composer.phx2.fp.o are on
F31 and there is a build for it and submitted as an update
https://bodhi.fedoraproject.org/updates/FEDORA-2020-f265f98566
For bodhi-backend01.phx2.fp.o is still on F30, for which there isn't a
build for f30. So, I backported the patches to F30 and built an infra
build for F30 https://koji.fedoraproject.org/koji/taskinfo?taskID=43574522.
Its not tagged because I dont have infra permissions, but I tagged it
manually and it is signed now and tagged into f30-infra-stg. I will
move it to f30-infra once this FBR is approved and will update the
box.
If you think we need to be careful, then I can apply the patch to
rawhide-composer.phx2.fp.o and based on how that goes, we can update
the others.
Thanks.
3 years, 11 months
[PATCH] fas_client: fix template to correctly apply on pkgs02 and add people02
by Kevin Fenzi
The ansible_hostname variable is actually the short name of the host,
not the fqdn, so this conditional didn't match before. Switch it to use
startswith and also add people02 as thats the other host people try and
login to often after changing ssh keys.
With this, pkgs02 and people02 should hopefully update ssh keys from fas
every 15min and avoid manual sync requests to the team.
Signed-off-by: Kevin Fenzi <kevin(a)scrye.com>
---
roles/fas_client/templates/fas-client.cron.j2 | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/roles/fas_client/templates/fas-client.cron.j2 b/roles/fas_client/templates/fas-client.cron.j2
index c0de939..8dd0a78 100644
--- a/roles/fas_client/templates/fas-client.cron.j2
+++ b/roles/fas_client/templates/fas-client.cron.j2
@@ -1,4 +1,4 @@
-{% if ansible_hostname == 'pkgs02.phx2.fedoraproject.org' %}
+{% if ansible_hostname.startswith(('pkgs02', 'people02')) %}
*/15 * * * * root /usr/local/bin/lock-wrapper fasClient "/usr/bin/fasClient -i |& grep -vi deprecation | /usr/local/bin/nag-once fassync 1d 2>&1"
{% else %}
00 20 * * * root /usr/local/bin/lock-wrapper fasClient "/bin/sleep $(($RANDOM \% 3600)); /usr/bin/fasClient -i |& grep -vi deprecation | /usr/local/bin/nag-once fassync 1d 2>&1"
--
1.8.3.1
3 years, 11 months
FBR: Update all systems to nrpe-4.0.4
by Stephen John Smoogen
NRPE for Fedora was updated to 4.0.4 which got auto-updated on many of the
infrastructure systems. However, the noc servers are still running a much
older version of nrpe which is causing some issues with monitoring.
Plan: run on batcave01
sudo ansible -i noc:batcave:bastion:people -m shell -a 'yum
--enablerepo=epel-testing update nagios* nrpe*'
sudo ansible -i noc:batcave:bastion:people -m shell -a 'rkhunter --propupd'
--
Stephen J Smoogen.
3 years, 11 months
FBR: To enable signing on eln builds
by Mohan Boddu
Hi,
The following patch will enable signing on eln builds and for now it
will be used only in stg.
For more info: https://pagure.io/releng/issue/9154
diff --git a/roles/robosignatory/templates/robosignatory.toml.j2
b/roles/robosignatory/templates/robosignatory.toml.j2
index 884f21c..27a12c8 100644
--- a/roles/robosignatory/templates/robosignatory.toml.j2
+++ b/roles/robosignatory/templates/robosignatory.toml.j2
@@ -321,6 +321,14 @@ handlers = ["console"]
key = "{{ (env == 'production')|ternary('epel-6', 'testkey') }}"
keyid = "{{ (env == 'production')|ternary('0608b895',
'd300e724') }}"
+ # ELN signing
+
+ [[consumer_config.koji_instances.primary.tags]]
+ from = "eln-pending"
+ to = "eln-candidate"
+ key = "{{ (env == 'production')|ternary('fedora-33', 'testkey') }}"
+ keyid = "{{ (env == 'production')|ternary('9570ff31',
'd300e724') }}"
+
[consumer_config.ostree_refs]
[consumer_config.ostree_refs."fedora/rawhide/x86_64/iot"]
3 years, 11 months
Meeting Agenda Item: Introduction nasirhm
by Nasir Hussain
Hey Guys, I would like to apply to be a part of the Infrastructure team as an Apprentice, I've gone through the Dev-Guide for it too. I'm pretty interested in Working on Python & can also perform some sysadmin tasks too. I'm a member of the Join SiG and the Fedora Advocates Group to promote fedora in our local hackerspace. What would be the ways for me to get started with contributing to the Infrastructure and would love to have a mentor.
3 years, 11 months
CPE Weekly: 2020-04-18
by Aoife Moloney
---
title: CPE Weekly status email
tags: CPE Weekly, email
---
# CPE Weekly 2020-04-18
Background:
The Community Platform Engineering group is the Red Hat team combining
IT and release engineering from Fedora and CentOS. Check out our
team's info here https://docs.fedoraproject.org/en-US/cpe/
## GitForge Updates
We are still actively engaging with the Fedora Council and have not
made any further progress with this project. We are focusing on
ensuring we have captured the most granular of requirements before
further engaging with GitLab by rereviewing what we have gathered so
far and requesting more specific technical clarifications on some.
We are also as a team discussing the best way to track the updates on
this project publicly and hope to have something published early next
week. We will send an email to the devel list as soon as we have
agreed and created this.
Thank you for your patience and engagement with us thus far.
## Fedora Updates
* Fedora 32 release: RC 1.3 was tested this week
* Go/No-Go meeting happened on Thursday 16th April - decision:No Go
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedora...
### Data Centre Move
* Please note Communishift is now down and is en route to the
datacentre in RDU-CC.
* We have a list of affected services that will be in effect from May
25th - July 1st (est) as part of the full datacentre move. You can
view that list here https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA
* As always, please view our public schedule here for more a more
detailed overview
https://hackmd.io/@fedorainfra2020/rJpsA4FLL#First-draft-of-schedule-for-...
### AAA Replacement
* The team officially kicked off phase two of the project development today.
* They are going to look at applications that currently use the FAS
client to change their codebase to the FASJSON client post production.
* The team are also working on adding a python library to the API so
that this maps to python objects easily.
* They are also working on API endpoints, specifically to be able to
retrieve the open-api specification in JSON format & provide a health
endpoint for monitoring tools.
* Our work is publicly tracked here if you want to find out more
https://github.com/orgs/fedora-infra/projects/6
### CI/CD
* rpmautospec is available for testing in staging, find more info here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedora...
### Sustaining Team
* The team are looking at creating a ticket dashboard for tickets and
have reused most the fedora-gather-easyfix code base to work on this
* https://fedorapeople.org/~humaton/cpe/
* Some cool stats about tickets from the infra repo here
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
* Mbbox:
* Review koji-hub CRD PR
https://github.com/fedora-infra/mbbox/pull/19
Starting to play around with koji-ansible collections for releng work (here).
Automation of the openh264 packages.
## CentOS Updates
### CentOS
* CentOS CI - stable with 0 downtime
* 7.8 QA tree is being tested by CentOS QA folks
### CentOS Stream
* The team are working on having Stream be publically consumable in
the coming months, with SIG enablement so watch the devel lists for
more technical updates as and when they are announced!
As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.
Have a great weekend!
Aoife
Source: https://hackmd.io/gSci385uRoeNVEuoNUS1pg?view
--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
3 years, 11 months
FBR: remove elections from inventory and run noc.yml playbook
by Stephen John Smoogen
The following patch should do a minimal removal of the elections app from
ansible inventory so that a run of noc.yml will remove the systems from
being monitored. Then we can turn off the elections systems.
--
Stephen J Smoogen.
3 years, 11 months