Request for Feedback/Resources for Packager Dashboard/Oraculum
by Frantisek Zatloukal
Hello,
many of you already heard about Fedora Packager Dashboard [0]. In short,
for those who have not, it's a web application and backend aiming to make
the life of Fedora packagers easier. It combines data from multiple sources
(Pagure, Bugzilla, Bodhi, Koschei,...) relevant to the maintainers of
Fedora packages.
Tracking all these sites can be time consuming, especially if you maintain
dozens of different packages, so the Dashboard provides everything a
packager might need (or at least what we've thought of) - condensed,
cached, searchable and filterable on one page.
You can check it out/play with it even if you don't maintain any Fedora
packages, there is no authentication needed, just enter any packager’s
username. Feature-wise, it's pretty robust already, and there are more
things to come (like allowing users to authenticate and see private bugs
and more), but the original planned feature set is complete.
Currently, it's processing only publicly available data. When we'll start
implementing the ability to authenticate and process any private data,
we'll work closely with the Infra team to make sure there are no open holes.
Since the announcement of the testing phase, it has been running on a
temporary server which is barely keeping up, so I'd like to open up a
conversation about migration to the Infra OpenShift cluster.
Here is a brief overview of it's internals (I can elaborate more if anybody
needs me to):
Backend is a Flask/Python application leveraging Celery for planning and
executing the cache refreshes. The API is striving to be as non-blocking as
possible, using asynchronous-inspired behaviour. The client is given the
data currently available in cache, and advised about the completeness
(complete/partial cache misses) via HTTP status code.
The parameters for cache refreshes can be customized, depending on the
resources we have/can get. Currently, it's retrieving data for most of the
items every 2 hours (with exceptions like package versions which run daily
and are terribly slow). Backend is caching data for PRs, bugs, and
pre-calculated updates/overrides data for users visiting the app at least
once in two weeks. The main storage is a PostgreSQL database, and
optionally, if RAM is not an issue, we have a local memory-cached layer
that can be enabled (we find it not necessary ATM).
Apart from storing the pre-crunched information from public sources, we
keep timestamps of the last visit for each user.
Frontend is a React app fetching and displaying data from the backend
(really nothing to add here :) ).
Based on the OpenShift testing, I've come to the following schema of pods:
-
Redis pod (Celery backend)
-
PostgreSQL pod (cache and watchdog data storage)
-
Beat pod (scheduling tasks for data refresh)
-
Flower pod (backend monitoring; nice to have, not absolutely necessary)
-
Gunicorn/NGINX pod (Oraculum backend)
-
NGINX pod (Packager Dashboard front-end).
On top of that, we need a number of worker pods completing the scheduled
Celery tasks.
I am not sure how much RAM each pod in Infra OpenShift has, currently the
workers seem to be performing best with about 512 MB memory limit. Ideally
we’d like to have at least 12-16 Celery workers (more workers can, of
course, run on a single pod).
Resource-wise it's not a small application (at least from my perspective :D
), but we believe it's a great value application which saves time for Red
Hat and community package maintainers.
I'd like to hear your feedback, questions, requests for changes if there is
anything preventing it from deployment in Infra OpenShift
(architecture/code wise). And of course opinions on the feasibility of
moving Oraculum and Packager Dashboard into the Infra OpenShift cluster.
Thanks!
[0] https://packager.fedorainfracloud.org/
3 years, 5 months
DKIM & fedora lists
by Dominique Martinet
Hi,
I just enabled dkim/dmarc on my domain last week, so sent my first email
with that setup to devel@ just earlier today and got a few "invalid dkim
signature" return emails...
So I was wondering how do people deal with that?
Normally lists have two ways of handling dkim:
- either they don't mangle the subject/signed headers (for me:
h=Date:From:To:Cc:Subject:References:In-Reply-To:From).
In that case, just leave the original dkim header and things should just
work™.
That's what e.g. kernel lists do and worked well for me.
- either they DO mangle headers, often adding a [tag] to the subject
line; in which case the From is also updated to be the list address with
the original sender name (e.g. Bob <whateverlist@somewhere>) and the
original mail is eventually appended to the Reply-To addresses, with the
original dkim header stripped off.
As far as I can see devel(a)lists.fedoraproject.org doesn't mangle
anything so should fall into the first category of "doing nothing just
works" -- but it stripped my original dkim header, hence the failures.
I'm pretty sure mailman can deal with this, is that on purpose? Or is it
just a mishap?
my dmarc policy says to ignore dkim failures (for now) so I could just
ignore this but it's a bit annoying that I had setup dmarc/dkim because
my mails often get treated as spam for some reason and such errors won't
be helping...
Thanks!
PS: I subscribed to the infra list just for this, but did look through
the archives quickly and didn't find much, sorry if this had been
brought up before. I'll likely unsubscribe back once that's answered one
way or another.
--
Dominique
3 years, 5 months
best place to run an email-forwarding bot account?
by Matthew Miller
The Fedora Council is moving from our mailing list to
discussion.fedoraproject.org. However, I want to have a digest of discussion
go to the old list, for people who prefer to follow a summary that way.
Discourse has a digest feature, but no "automatically send a digest to this
address" ability separate from individual user accounts. So, I was thinking
I could create a "fedora-council-discuss-bot" FAS account, and have that
email go to somewhere with a procmail filter which forwards the digests (and
only the digests, obviously) to the council list. Then I'd set the list to
only allow posts from that bot address.
1. Does this plan seem reasonable?
2. Is there a place in Fedora Infrastructure that can receive and bounce
mail like that?
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
3 years, 5 months
CPE Weekly: 2020-11-08
by Aoife Moloney
Hi Everyone,
Below is this week's CPE weekly for week ending 2020-10-25 for both
Fedora & CentOS, and if you want to visit the hackmd link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the
header bar on your left to skip to Fedora or CentOS updates that
interest you.
## General Project Updates
We've updated our How to work with us section on our wiki
https://docs.fedoraproject.org/en-US/cpe/initiatives/
Updated initative timetable can be viewed here
https://docs.fedoraproject.org/en-US/cpe/time_tables/
And we have a new initiative repo live on Pagure
https://pagure.io/cpe/initiatives-proposal
Below are the scheduled projects the CPE team are currently working on
for the months of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time
### Misc
#### GitLab
New GitLab topic sent to devel-announce(a)lists.fedoraproject.org &
centos-devel(a)centos.org on Message Bus is out. See email in hackmd
here https://hackmd.io/tfOqCXNEQtqsGNLAEfZ2zg?view
## Project Updates
*The below updates are pulled directly from our CPE team call we have
every week.*
### Fedora
* F33 is released!!!!
* F33 arm images are re-created and pushed as a point release
* Testdays app up in openshift stg:
https://testdays.stg.fedoraproject.org/events and may even be in
production at the time of this email!
* Openqa has all hardware up and running except one broken power9 box
- working on resolving this
### Staging Environment
* Build system nearly done - waiting on a firewall change
### Noggin/AAA
* If you maintain an application that uses FAS, please test your app
authentication in staging and report any issues you may find to the
team in #fedora-aaa.
* The team are particularly interested in if you cannot login or your
user does not have the right groups/permissions/attributes, eg
[cla_done]
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6
* And we have a project tracker available to be viewed here
https://github.com/fedora-infra/aaa-tracker
### OSBS for aarch64
* The team are tracking their work here
https://pagure.io/cpe/osbs-project/issues
* This project started on November 4th and the team are investigating
technical approaches and requirements such as permissions & access to
images.
### Fedora Messaging Schemas
* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7
## CentOS Updates
### CentOS
* Update CentOS CI Openshift clusters to latest 4.5.16
* Storage resync completed this week.
### CentOS Stream
* The team are working on developing ODCS service in the Stream build infra
* Developing scripts to help automate adding artifacts to module builds
* Check out the CentOS Stream page on centos.org/centos-stream for
information on the latest download information and how to convert from
CentOS Linux to Stream - especially if you are running on CentOS Linux
6 which is EOL on November 30th :)
## Team Info
### CPE Product Owner Office Hours
IRC office hours are now once per month.Below are the logs from the
most recent meetings and dates for the next ones.
#### #fedora-meeting-1
* Next Meeting: 2020-11-12 @ 1300 UTC on #fedora-meeting-1
#### #centos-meeting
* Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting
## Background:
The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.
See our wiki page here for more
information:https://docs.fedoraproject.org/en-US/cpe/
As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.
Have a great week!
Aoife
Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view
--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
3 years, 5 months
Another Rust MirrorManager experiment
by Adrian Reber
Our MirrorManager setup exports the current state of all mirrors every
hour at :30 to a protobuf based file which is then used by the
mirrorlist servers to answer the requests from yum and dnf.
The Python script requires up to 10GB of memory and takes between 35 and
50 minutes. The script does a lot of SQL queries and also some really
big SQL queries joining up to 6 large MirrorManager tables.
I have rewritten this Python script in Rust and now it only needs around
1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB.
I think the biggest difference is that I am almost not doing any joins
in my SQL request. I download all the tables once and then I do a lot of
loops over the downloaded tables and this seems to be massively faster.
As the mirrorlist-server in Rust has proven to be extremely stable over
the last months we have been using it I would also like to replace the
mirrorlist protbuf input generation with my new Rust based code.
I am planing to try out the new protobuf file in staging in the next
days and would then try to get my new protobuf generation program into
Fedora. Once it is packaged I would discuss here how and if we want to
deploy in Fedora's infrastructure.
Having the possibility to generate the mirrorlist input data in about a
minute would significantly reduce the load on the database server and
enable us to react much faster if broken protobuf data has been synced
to the mirrorlist servers on the proxies.
Adrian
3 years, 5 months
