infrastructure November 2020

infrastructure@lists.fedoraproject.org

23 participants
19 discussions

Request for Feedback/Resources for Packager Dashboard/Oraculum
by Frantisek Zatloukal 16 Nov '20

16 Nov '20

Hello, many of you already heard about Fedora Packager Dashboard [0]. In short, for those who have not, it's a web application and backend aiming to make the life of Fedora packagers easier. It combines data from multiple sources (Pagure, Bugzilla, Bodhi, Koschei,...) relevant to the maintainers of Fedora packages. Tracking all these sites can be time consuming, especially if you maintain dozens of different packages, so the Dashboard provides everything a packager might need (or at least what we've thought of) - condensed, cached, searchable and filterable on one page. You can check it out/play with it even if you don't maintain any Fedora packages, there is no authentication needed, just enter any packager’s username. Feature-wise, it's pretty robust already, and there are more things to come (like allowing users to authenticate and see private bugs and more), but the original planned feature set is complete. Currently, it's processing only publicly available data. When we'll start implementing the ability to authenticate and process any private data, we'll work closely with the Infra team to make sure there are no open holes. Since the announcement of the testing phase, it has been running on a temporary server which is barely keeping up, so I'd like to open up a conversation about migration to the Infra OpenShift cluster. Here is a brief overview of it's internals (I can elaborate more if anybody needs me to): Backend is a Flask/Python application leveraging Celery for planning and executing the cache refreshes. The API is striving to be as non-blocking as possible, using asynchronous-inspired behaviour. The client is given the data currently available in cache, and advised about the completeness (complete/partial cache misses) via HTTP status code. The parameters for cache refreshes can be customized, depending on the resources we have/can get. Currently, it's retrieving data for most of the items every 2 hours (with exceptions like package versions which run daily and are terribly slow). Backend is caching data for PRs, bugs, and pre-calculated updates/overrides data for users visiting the app at least once in two weeks. The main storage is a PostgreSQL database, and optionally, if RAM is not an issue, we have a local memory-cached layer that can be enabled (we find it not necessary ATM). Apart from storing the pre-crunched information from public sources, we keep timestamps of the last visit for each user. Frontend is a React app fetching and displaying data from the backend (really nothing to add here :) ). Based on the OpenShift testing, I've come to the following schema of pods: - Redis pod (Celery backend) - PostgreSQL pod (cache and watchdog data storage) - Beat pod (scheduling tasks for data refresh) - Flower pod (backend monitoring; nice to have, not absolutely necessary) - Gunicorn/NGINX pod (Oraculum backend) - NGINX pod (Packager Dashboard front-end). On top of that, we need a number of worker pods completing the scheduled Celery tasks. I am not sure how much RAM each pod in Infra OpenShift has, currently the workers seem to be performing best with about 512 MB memory limit. Ideally we’d like to have at least 12-16 Celery workers (more workers can, of course, run on a single pod). Resource-wise it's not a small application (at least from my perspective :D ), but we believe it's a great value application which saves time for Red Hat and community package maintainers. I'd like to hear your feedback, questions, requests for changes if there is anything preventing it from deployment in Infra OpenShift (architecture/code wise). And of course opinions on the feasibility of moving Oraculum and Packager Dashboard into the Infra OpenShift cluster. Thanks! [0] https://packager.fedorainfracloud.org/

4 11

DKIM & fedora lists
by Dominique Martinet 15 Nov '20

15 Nov '20

Hi, I just enabled dkim/dmarc on my domain last week, so sent my first email with that setup to devel@ just earlier today and got a few "invalid dkim signature" return emails... So I was wondering how do people deal with that? Normally lists have two ways of handling dkim: - either they don't mangle the subject/signed headers (for me: h=Date:From:To:Cc:Subject:References:In-Reply-To:From). In that case, just leave the original dkim header and things should just work™. That's what e.g. kernel lists do and worked well for me. - either they DO mangle headers, often adding a [tag] to the subject line; in which case the From is also updated to be the list address with the original sender name (e.g. Bob <whateverlist@somewhere>) and the original mail is eventually appended to the Reply-To addresses, with the original dkim header stripped off. As far as I can see devel(a)lists.fedoraproject.org doesn't mangle anything so should fall into the first category of "doing nothing just works" -- but it stripped my original dkim header, hence the failures. I'm pretty sure mailman can deal with this, is that on purpose? Or is it just a mishap? my dmarc policy says to ignore dkim failures (for now) so I could just ignore this but it's a bit annoying that I had setup dmarc/dkim because my mails often get treated as spam for some reason and such errors won't be helping... Thanks! PS: I subscribed to the infra list just for this, but did look through the archives quickly and didn't find much, sorry if this had been brought up before. I'll likely unsubscribe back once that's answered one way or another. -- Dominique

4 5

best place to run an email-forwarding bot account?
by Matthew Miller 12 Nov '20

12 Nov '20

The Fedora Council is moving from our mailing list to discussion.fedoraproject.org. However, I want to have a digest of discussion go to the old list, for people who prefer to follow a summary that way. Discourse has a digest feature, but no "automatically send a digest to this address" ability separate from individual user accounts. So, I was thinking I could create a "fedora-council-discuss-bot" FAS account, and have that email go to somewhere with a procmail filter which forwards the digests (and only the digests, obviously) to the council list. Then I'd set the list to only allow posts from that bot address. 1. Does this plan seem reasonable? 2. Is there a place in Fedora Infrastructure that can receive and bounce mail like that? -- Matthew Miller <mattdm(a)fedoraproject.org> Fedora Project Leader

3 4

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 11 Nov '20

11 Nov '20

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2020-11-12 from 08:00:00 to 09:00:00 America/Los_Angeles At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

1 0

Planned Outage: 2020-11-11 21:00 UTC Fedora Services
by Stephen John Smoogen 09 Nov '20

09 Nov '20

There will be an outage starting at 2020-11-11 21:00UTC, which will last approximately 6 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2017-05-03 21:00UTC' Reason for outage: Apply updates and new kernels for all systems running RHEL7/RHEL8 and Fedora 32 Affected Services: Most services in IAD2 will see downtime as we run updates and reboot systems. Ticket Link: https://pagure.io/fedora-infrastructure/issue/NNNN Please join #fedora-admin or #fedora-noc on irc.freenode.net or add comments to the ticket for this outage above. -- Stephen J Smoogen.

1 1

CPE Weekly: 2020-11-08
by Aoife Moloney 08 Nov '20

08 Nov '20

Hi Everyone, Below is this week's CPE weekly for week ending 2020-10-25 for both Fedora & CentOS, and if you want to visit the hackmd link https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the header bar on your left to skip to Fedora or CentOS updates that interest you. ## General Project Updates We've updated our How to work with us section on our wiki https://docs.fedoraproject.org/en-US/cpe/initiatives/ Updated initative timetable can be viewed here https://docs.fedoraproject.org/en-US/cpe/time_tables/ And we have a new initiative repo live on Pagure https://pagure.io/cpe/initiatives-proposal Below are the scheduled projects the CPE team are currently working on for the months of October, November & December: * CentOS Stream Phase 4 - Build system services * Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing * OSBS for aarch64 - this will begin in November * Fedora Messaging Schemas - this work is continuing from Q3 and is being worked on part-time ### Misc #### GitLab New GitLab topic sent to devel-announce(a)lists.fedoraproject.org & centos-devel(a)centos.org on Message Bus is out. See email in hackmd here https://hackmd.io/tfOqCXNEQtqsGNLAEfZ2zg?view ## Project Updates *The below updates are pulled directly from our CPE team call we have every week.* ### Fedora * F33 is released!!!! * F33 arm images are re-created and pushed as a point release * Testdays app up in openshift stg: https://testdays.stg.fedoraproject.org/events and may even be in production at the time of this email! * Openqa has all hardware up and running except one broken power9 box - working on resolving this ### Staging Environment * Build system nearly done - waiting on a firewall change ### Noggin/AAA * If you maintain an application that uses FAS, please test your app authentication in staging and report any issues you may find to the team in #fedora-aaa. * The team are particularly interested in if you cannot login or your user does not have the right groups/permissions/attributes, eg [cla_done] * The teams kanban board where they track their work can be found here https://github.com/orgs/fedora-infra/projects/6 * And we have a project tracker available to be viewed here https://github.com/fedora-infra/aaa-tracker ### OSBS for aarch64 * The team are tracking their work here https://pagure.io/cpe/osbs-project/issues * This project started on November 4th and the team are investigating technical approaches and requirements such as permissions & access to images. ### Fedora Messaging Schemas * This project is worked on on a part time basis as we are prioritizing completing Noggin first before fully committing to its completion * There is a list of applications that require messaging schemas can be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit * There is a readme which contains documentation on messaging schemas, a cookie-cutter template to create the schema and a definition of Done for writing a schemas https://github.com/fedora-infra/fedora-messaging-schemas-issues * The board they are working from can be viewed here https://github.com/orgs/fedora-infra/projects/7 ## CentOS Updates ### CentOS * Update CentOS CI Openshift clusters to latest 4.5.16 * Storage resync completed this week. ### CentOS Stream * The team are working on developing ODCS service in the Stream build infra * Developing scripts to help automate adding artifacts to module builds * Check out the CentOS Stream page on centos.org/centos-stream for information on the latest download information and how to convert from CentOS Linux to Stream - especially if you are running on CentOS Linux 6 which is EOL on November 30th :) ## Team Info ### CPE Product Owner Office Hours IRC office hours are now once per month.Below are the logs from the most recent meetings and dates for the next ones. #### #fedora-meeting-1 * Next Meeting: 2020-11-12 @ 1300 UTC on #fedora-meeting-1 #### #centos-meeting * Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting ## Background: The Community Platform Engineering group, or CPE for short, is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give. See our wiki page here for more information:https://docs.fedoraproject.org/en-US/cpe/ As always, feedback is welcome, and we will continue to look at ways to improve the delivery and readability of this weekly report. Have a great week! Aoife Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view -- Aoife Moloney Product Owner Community Platform Engineering Team Red Hat EMEA Communications House Cork Road Waterford

1 0

Possible outdated Jenkins jobs
by Leonardo Rossetti 05 Nov '20

05 Nov '20

4 12

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 04 Nov '20

04 Nov '20

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2020-11-05 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

1 0

Another Rust MirrorManager experiment
by Adrian Reber 03 Nov '20

03 Nov '20

Our MirrorManager setup exports the current state of all mirrors every hour at :30 to a protobuf based file which is then used by the mirrorlist servers to answer the requests from yum and dnf. The Python script requires up to 10GB of memory and takes between 35 and 50 minutes. The script does a lot of SQL queries and also some really big SQL queries joining up to 6 large MirrorManager tables. I have rewritten this Python script in Rust and now it only needs around 1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB. I think the biggest difference is that I am almost not doing any joins in my SQL request. I download all the tables once and then I do a lot of loops over the downloaded tables and this seems to be massively faster. As the mirrorlist-server in Rust has proven to be extremely stable over the last months we have been using it I would also like to replace the mirrorlist protbuf input generation with my new Rust based code. I am planing to try out the new protobuf file in staging in the next days and would then try to get my new protobuf generation program into Fedora. Once it is packaged I would discuss here how and if we want to deploy in Fedora's infrastructure. Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies. Adrian

4 17

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure November 2020