Meeting agenda for 2019-11-21
by Stephen John Smoogen
The infrastructure team will be having its weekly meeting tomorrow,
2019-11-21 at 15:00 UTC in #fedora-meeting-1 on the freenode network.
We have a document at https://board.net/p/fedora-infra
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
## Introduction
We will use it over the week before the meeting to gather status and
info and discussion items and so forth, then use it in the irc meeting
to transfer information to the meetbot logs.
### Meeting start stuff
```
#startmeeting Infrastructure (2019-11-14)
#meetingname infrastructure
#topic aloha
#chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny
abompard bowlofeggs
```
### Determine who the next chair is
#topic Next chair
#info magic eight ball says:
2019-11-21: pingou
2019-11-28 - holiday, NO MEETING!
2019-12-05 -smooge
2019-12-12 -??
2019-12-19 -??
2019-12-26 - NO MEETING
2020-01-02 - NO MEETING
2020-01-09 - ??
### Let new people say hello
```
#topic New folks introductions
#info This is a place where people who are interested in Fedora
Infrastructure can introduce themselves
#info Getting Started Guide:
https://fedoraproject.org/wiki/Infrastructure/GettingStarted
```
### Status / Information / Trivia / Announcements
(We put things here we want others on the team to know, but don't need
to discuss)
(Please use ```#info (the thing - your name)```
```
#topic announcements and information
#info
```
### Things we should discuss
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus /suor decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use ```#topic your discussion topic - your username)```
```
#topic Oncall
#info https://fedoraproject.org/wiki/Infrastructure/Oncall
#info pingou is on call 2019-11-14->2019-11-21
#info kevin is on call 2019-11-21->2019-11-28
#info smooge is on call 2019-11-28->2019-12-05
## .oncalltakeeu .oncalltakeus
#info Summary of last week: (from current oncall )
#info
#topic Monitoring discussion
#info https://nagios.fedoraproject.org/nagios
#info Go over existing out items and fix
#topic Tickets discussion
#info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
#infohttps://pagure.io/fedora-infrastructure/issue/8397
#info Fedora CoreOS Team requests
#link https://hackmd.io/5uB7hOJKSjGUt65iLgPnbA#Existing-requests-for-Fedora-Infra
#topic backlog discussion
#info go over our backlog and discuss and determine priority
#link https://pagure.io/fedora-infrastructure/issue/8178
#info topic: provision new aarch64 builders
#link https://pagure.io/fedora-infrastructure/issue/8157
#info topic: ansible: enable ansible-report as a hook
```
Go thru each ticket one by one
#topic Community Fires Team
### Put all topics for discussion under here
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
### Learn about some application or setup in infrastructure
(This section, each week we get 1 person to talk about an application
or setup that we have. Just going over what it is, how to contribute,
ideas for improvement, etc. Whoever would like to do this, just add
the i/nfo in this section. In the event we don't find someone to teach
about something, we skip this section and just move on to open floor.)
```
#info
```
### Meeting end stuff
```
#topic Open Floor
#endmeeting
--
Stephen J Smoogen.
4 years, 5 months
New mirrorlist server implementation
by Adrian Reber
Fedora's complete MirrorManager setup is still running on Python2. The
code has been ported to Python3 probably over two years ago but we have
not switched yet. One of the reasons is that the backend is running on
RHEL7 which means we are not in a hurry to deploy the Python3 version.
The mirrorlist server which is answering the actual dnf/yum queries for
a mirrorlist/metalink is, however, running in a Fedora 29 container.
This container also still uses Python2 and it actually cannot use the
Python3 version.
One of MirrorManager's design points is that the mirrorlist servers,
which are answering around 27 000 000 requests per day, are not directly
accessing the database. The backend creates a snapshot of the relevant
data (113MB) and the mirrorlist servers are using this snapshot to
answer client requests.
This data exchange is based on Python's pickle format and that does not
seem to work with Python3 if it is generated using Python2.
Having used protobuf before, I added code to also export the data for the
mirrorlist servers based on protobuf.
The good news with protobuf is, that the resulting file is only 66MB
instead of 113MB. The bad news is, that loading it from Python requires
3.5 times the amount of memory during runtime (3.5GB instead of 1GB).
In addition to the data exchange problems between backend and
mirrorlist servers the architecture of the mirrorlist server does not
really make sense today. 12 years ago it made a lot of sense as it could
be easily integrated into httpd and it could be easily reloaded without
stopping the service. Today the mirrorlist server and httpd is all part
of a container which is then behind haproxy. So there is a lot of
infrastructure in the container which is not really useful.
To get rid of the pickle format and to have a simpler architecture I
reimplemented the mirrorlist-server in Rust. This was brought up some
time ago on a ticket and with the protobuf problems I was seeing in
Python it made sense to try it out.
My code currently can be found at https://github.com/adrianreber/mirrorlist-server
and so far the results from the new mirrorlist server are the same as
from the Python based mirrorlist server.
It requires less than 700MB instead of the 1GB in Python with production
based data and seems really fast.
I have set up a test instance with the mirror data from Sunday at:
https://lisas.de/metalink?repo=updates-testing-f31&arch=x86_64
https://lisas.de/mirrorlist?repo=updates-testing-f31&arch=x86_64
The instance is based on the container I pushed to quay.io:
$ podman run quay.io/adrianreber/mirrorlist-server:latest -h
With this change the mirrorlist server would also finally switch to
geoip2. The currently running mirrorlist server still uses the legacy
geoip database.
After the Fedora 31 freeze I would like to introduce this new mirrorlist
server implementation on the proxies. I already verified that I can run
this mirrorlist container rootless. This new container can be a drop-in
replacement for the current container and no infrastructure around it
needs to be changed.
The main changes to get it into production is to change mirrorlist1.service
and mirrorlist2.service to include a line "User=mirrormanager" and
replace the current container name with new container.
Adrian
4 years, 5 months
Problems with vcglib
by J. Scheurich
Hi,
There are problems with the vcglib package
vcglib review request:
https://bugzilla.redhat.com/show_bug.cgi?id=1677989
$ kinit mufti11(a)FEDORAPROJECT.ORG
Password for mufti11(a)FEDORAPROJECT.ORG:
$ fedpkg clone vcglib
Cloning into 'vcglib'...
No such repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Could not execute clone: Failed to execute command.
$
Did i something wrong ?
so long
MUFTI
4 years, 5 months
patches to review to prepare our ansible scripts for fedora-messaging
by karsten＠fedoraproject.org
Hello,
There is work in progress to migrate our applications from the current
message bus 'fedmsg' to the AMPG based 'fedora-messaging'.
Attached are a couple of patches that prepare our ansible scripts for this.
Please review those patches and comment, thanks !
Karsten
4 years, 5 months
Greenwave release
by Giulia Naponiello
Hello,
I'm going to perform a Greenwave release in a bit.
I'll send an email again with the release notes when I'm done with it.
Thank you
Cheers
Giulia
4 years, 5 months
Fwd: How to approve a review request ?
by J. Scheurich
Hi,
>> I want to approve the review request of vimvi-qt, but this is my first
>> offical review 8-(
> You need to set fedora-review+ flag. Click on "Show advanced fields" near
> the top of the bug, and flags should be visible in a drop-down menu on
> the right.
| I still don't find it 8-(
http://wdune.ourproject.org/examples/vimvi.png
| Mufti, it's weird that your screenshot shows that you can set the
| fedora-review flag to ? but not to +. Maybe file a ticket with
| infra---possibly Bugzilla didn't sync with FAS properly or something?
Can you help me ?
so long
MUFTI
4 years, 5 months
CPE Weekly: 2019-11-16
by Aoife Moloney
Hi Everyone!
Travelling yesterday saw a delay in sending this weeks CPE email
blast, but nevertheless, here it is!
Hope everyone has been having a great weekend and if you have any
comments or feedback, please let me know- it is always welcome :)
Background:
The Community Platform Engineering group is the Red Hat team combining
IT and release engineering from Fedora and CentOS. Our goal is to keep
core servers and services running and maintained, build releases, and
other strategic tasks that need more dedicated time than volunteers
can give.
For better communication, we will be giving weekly reports to the
CentOS and Fedora communities about the general tasks and work being
done. Also for better communication between our groups we have
created #redhat-cpe on Freenode IRC! Please feel free to catch us
there, a mail has landed on both the CentOS and Fedora devel lists
with context here.
Note:
This document is currently built from individual reports rolled into a
google document which we edit and copy into a final document. We are
aware that this causes problems with some email readers, and are
working on a method to make this less problematic.
High Level Project Updates:
Fedora:
Rawhide Gating:
Bodhi 5.0 has been deployed in production
We received some feedback about the changes in the UI and are looking
at how we can address them.
We are working on documenting on how packagers will use Bodhi for
multi-builds updates in rawhide:
https://github.com/fedora-infra/bodhi/issues/2322
https://fedoraproject.org/w/index.php?title=User:Nphilipp/Package_update_...
A 5.1 release is scheduled, likely some time next week, to address
some of the issues encountered and fixed in the 5.0 release.
repoSpanner
Our team came across some test reliability issues last week & these
are impeding progress on the 83x patch.
Some more debugging/discussions around RCMs pushes that seem to be
either very slow or hanging but the team is looking into some
solutions
Some progress was also made on the 83x patch however and we managed to
fix one unreliable test.
Application Retirements
Elections
Blocking issue was fixed (https://pagure.io/fedora-infrastructure/issue/8253)
Fedocal
jlanda hitting permission error in communishift
https://pagure.io/fedora-infrastructure/issue/8274
Nuancier
Benson Muite is now working on OIDC authentication
Pastebin
An article will be posted to Magazine instead - keep an eye out!
GDPR policies are still being looked into so stay tuned for updates
soon. This unfortunately takes some time given the legal perspective
needed!
CentOS:
The team is busy working on 8.1
Comps were produced and in git repo for next compose (pungi)
Transmodrify tool rewritten to match the mbs/rpm needs
Analyzing what RH hasn’t pushed and so that is needed
Builds started and are being working on for x86_64, ppc64le, aarch64
and armhfp architectures
https://koji.mbox.centos.org/koji/index
https://koji.armhfp-mbox.centos.org/koji/index
Infrastructure :
Dealing with hardware issues on storage box , so opening tickets with
vendor and internal IT to play MITM and try to get this solved
Dealing with RCM having still issues to push to git.centos.org for
some identified packages (reproducible) so now waiting on Repospanner
maintainer to come with a fix and an ETA for new release that would
fix it
Trying to provision some EC2 instances and working with Fedora to get
proper access to be able to have proper AWS access for CentOS (also
for S3)
Working on various PR for cbs-content-control and/or issues with mash
on https://cbs.centos.org (now resolved)
Misc
EPEL 8 modularity
EPEL8 module builds are working in staging & composes should be closed
There are now tags added to staging in koji
Bodhi configs have been updated
Pungi configs for epel8-playground are updated
Bug about zchunk update for EPEL-7 was fixed
https://github.com/fedora-infra/bodhi/pull/3692
Fedora Docs
Multiple small PRs were merged where people were being sent to Fedora
docs after F31 came out & below are all fixed:
https://pagure.io/fedora-docs/system-administrators-guide/issue/40
https://pagure.io/fedora-docs/system-administrators-guide/issue/39
https://pagure.io/fedora-docs/system-administrators-guide/issue/35
https://pagure.io/fedora-docs/install-guide/issue/33
https://pagure.io/fedora-docs/system-administrators-guide/issue/25
https://pagure.io/fedora-docs/system-administrators-guide/issue/23
https://pagure.io/fedora-docs/quick-docs/issue/155
Kindest regards,
Aoife
--
Aoife Moloney
Feature Driver
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
4 years, 5 months
