Plan for tomorrow's Fedora Infrastructure meeting - 2016-05-26
by Kevin Fenzi
The infrastructure team will be having it's weekly meeting tomorrow,
2016-05-26 at 18:00 UTC in #fedora-meeting on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
kevin
--
= Introduction =
This shared document is for the next fedora infrastructure meeting.
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2016-05-26)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson
#topic New folks introductions / Apprentice feedback
= Status / information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info Mass update/reboot cycle complete. Machines are ready for freeze - kevin/smooge/patrick/tflink
#info added +50GB space to bodhi-backend01, should last a bit - kevin
#info Fedora Message Notifications (FMN) finally caught back up
#info root passwords changed on all machines - kevin
#info ansible 2.1 on batcave01 now - kevin
#info F24 final freeze coming up next week - everyone
#info
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus or decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic
= Apprentice office hours =
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
#topic Apprentice office hours
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for improvement,
etc. Whoever would like to do this, just add the info in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#topic Learn about: mailman3 - kevin
= Meeting end stuff =
#topic Open Floor
#endmeeting
7 years, 11 months
RFC: Fedora Scale-Out Docker Registry Proposal
by Adam Miller
Hello all,
I have been working on a proposal on how to handle Fedora's Docker
Registry requirements to deliver our future "docker-ized" content to users in
a way that won't leave us with a bottleneck to the user community hitting a
single download location. Below you will find that proposal and I would
greatly appreciate feedback. The good news is I did a small PoC of the plan
I would like to propose along with the workflow (simulating some steps in the
process for brevity), and it all worked! The potentially bad news is that
this introduces a reasonable amount of net-new services inside the Fedora
Infrastructure.
I originally sent this to the Fedora rel-eng list[0] for general high level
solution sanity checking before attempting to burden the Infrastructure team
with any of this so most of this email is verbatim of the original one.
However, towards the end I am going to go more in-depth as to the net-new
requirements they will impose on the Fedora Infrastructure Team and I want to
make sure those things are taken into consideration because I absolutely do
not want to ask for the Infra Team to just take on a bunch of new stuff and
then ride off into the sunset while y'all deal with it all but instead make
sure that the things I am requesting are as reasonable as possible and are
realistic expectations for everyone involved.
Without further adieu, the pitch is below.
Quick vocab background for Docker stuff:
registry: a collection of docker image repositories
repository: named after an image and is a collection of multiple tags of
an that image
tag: an arbitrary string assigned to a specific docker image
(identified by the image's sha256 checksum)
NOTE: The "latest" tag is special and is assumed if no tag is
provided. This is true also for a 'docker pull' operation and
an image tagged "latest" will be the default image pulled by
users.
Proposal:
Pulp[1] + Crane[2] + MirrorManager[3] + Docker Distribution[4]
Pulp is a platform for managing repositories of content, such as software
packages, and making it available to a large numbers of consumers. It is also
capable of managing docker content.
Crane is a stand-alone python flask wsgi application written by the Pulp team
to serve as a API entry point for the docker client and will answer to an
user's 'docker pull'. It does not however create content manifests or provide
hosting for docker image content, instead it depends on someone creating the
manifest metadata themselves or having pulp publish it and serves 302
redirects to the docker client so they can find where the docker images
actually live.
I'll assume everyone here knows their way around MirrorManager.
Docker Distribution is the defacto standard open source implementation of the
Docker Registry V2 API spec[5]. It provides many features but the ability to
have it's back-end storage be provided by a "mirror network" much like the one
Fedora has at it's disposal is not one of them. The reason we need this in
place is because the mechanism by which you could push a docker image directly
to Pulp in Docker Registry v1 no longer exists in v2 so we must instead
perform a "sync" operation between the two. (This is a common problem for
all known "third party" v2 registry implementations).
Workflow:
OSBS will perform Builds, as these builds complete they will be pushed
to the docker-distribution (v2) registry, these will be considered
"candidate images". Pulp will sync and publish the candidate repository.
Testing will occur using the "candidate images" (details of how we want
to handle that are outside the scope of this proposal).
A "candidate image" will be marked stable once it's criteria have been
satisfied to do so. (This is vague because this is a topic of ongoing
discussion and work to decide what criteria an image will need to abide
by before being considered "stable" and promoted as such)
Once stable, pulp will publish that repository's content to a directory,
we will split that content and sync the image layers along with their
metadata to Mirror Manager master mirror. We will also sync the repo
metadata published by Pulp to somewhere Crane can pick it up. (This could
and will likely be something that Bodhi triggers via the Pulp REST API)
Mirror Manager will Pulp distribute to the mirrors the image layers and
their metadata.
Crane will get the new repository metadata and will serve redirects to
the new content relative to download.fedoraproject.org which will
perform another redirect (via MirrorManager) where the docker client
upon a "docker pull" will find it's content.
I have put together an ascii diagram in hopes that it will assist in seeing
the full picture.
https://maxamillion.fedorapeople.org/FedoraPulpDocker.txt
(I couldn't get it to paste cleanly into my email client)
Some more in depth technical items around this solution that I think the
Fedora Infrastructure Team are likely interested in:
Pulp Requirements:
- An AMPQ message queue, currently qpid and rabbitmq are supported
upstream. However, the requirement appears to stem from the use
of Celery[6] and Celery upstream supports redis[7] as a broker
backend so I have requested that it be made available as supported
option Pulp[8]. This will obviously take some amount of dev time,
but we can plan for that if adding a message queue to Fedora Infra
is a show stopper.
- MongoDB, this is currently a hard requirement but postgresql is
planned replace MongoDB in the future[9] (probably a year-ish
timeline on that). The question is, can we wait that long from a
Fedora Project standpoint for the new feature before having a
solution in place? I imagine some of this will need to be
planned/scoped as time goes on and we learn more but it's worth
keeping in mind
- Storage. I've been told Pulp likes a lot of storage, I don't know
hard numbers for what we'd need since we're getting into uncharted
territory but I've heard that a few hundred GB is not uncommon in
pulp deployments when combining the MongoDB storage needs with all
the artifacts in the repos.
Crane Requirements:
- Crane is just a small python wsgi app written in flask
A couple of things to note about maintenance and uptime considerations:
The Intermediate docker-distribution registry is needed for builds in
koji+OSBS
Pulp will be required for "promotion" of builds from candidate to testing
or stable
Crane will be required for end users out in the world to access in order
to actually pull down Docker images from us.
The only service here that needs to be public end-user facing (i.e. wide
open to the internet and not have access locked to a FAS group) is Crane.
All other components should be able to be locked down similar to the
"Fedora internal" components koji (builders, etc), bodhi (signing, etc)
and similar.
If there are any questions, comments, or feedback please let me know.
Thank you,
-AdamM
[0] - https://lists.fedoraproject.org/archives/list/rel-eng@lists.fedoraproject...
[1] - http://www.pulpproject.org/
[2] - https://github.com/pulp/crane
[3] - https://github.com/fedora-infra/mirrormanager2/
[4] - https://github.com/docker/distribution/
[5] - https://docs.docker.com/registry/spec/api/
[6] - http://www.celeryproject.org/
[7] - http://redis.io/
[8] - https://pulp.plan.io/issues/1900
[9] - https://pulp.plan.io/issues/1803
7 years, 11 months
Mailman 3 migration status
by Aurelien Bompard
Hey there,
Quick update on the Mailman3 migration. I have now migrated all lists to
the new mailman01 server (including fedorahosted lists). I've also migrated
the SpamAssassin configuration and database and it seems to be working fine.
I had to make a small tweak to the postfix configuration though, and even
if I don't think it opens a vulnerability in any way I'm pointing you to
it: I've opened the submission port (587/tcp) on the loopback address only,
with a bypass of the SpamAssassin check. The reason for that is that
Mailman 3 now sends emails to Postfix as an SMTP client, and with the
previous configuration all outgoing emails from Mailman were scanned, which
made no sense and caused a huge load on the server. I had two options:
1. Set Postfix to listen on localhost:smtp without the spam checking and on
its external IP with the spam checking, but this means hardcoding the
server's external IP in Postfix' master.cfg file
2. Add another port for Postfix's SMTP daemon witout the SpamAssassin
content filter, but keep it on localhost only for security
I chose the latter because I felt that harcoding the IP was a worse
solution, but since we're using Ansible it may not matter. If you prefer
the first solution I can totally make the change.
My next step is to download the old HTML archives and make them available
on mailman01 through Apache to preserve existing URLs in the wild.
Then I think we'll finally be able to take the former servers out of the
loop.
Aurélien
7 years, 11 months
Plan for tomorrow's Fedora Infrastructure meeting - 2016-05-19
by Kevin Fenzi
The infrastructure team will be having it's weekly meeting tomorrow,
2016-05-12 at 18:00 UTC in #fedora-meeting on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
kevin
--
= Introduction =
This shared document is for the next fedora infrastructure meeting.
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2016-05-19)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson
#topic New folks introductions / Apprentice feedback
= Status / information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info comps and spin-kickstarts moved to pagure.io - kevin
#info removed old docs-backend01 el6 instance, no longer needed - kevin
#info added a number of jenkins projects and packages to jenkins builders - kevin
#info helped work on https://pagure.io/quick-fedora-mirror with tibbs - kevin
#info removed download-ib02 (01 is ready to take it's place) - kevin
#info Fixed up several hosts whos playbooks wouldn't complete - kevin
#info Finally got all lists moved and collab03/hosted-lists01 can be retired - kevin/abompard
#info
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus or decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic Fedorahosted migration / EOL - kevin
#topic
= Apprentice office hours =
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
#topic Apprentice office hours
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for improvement,
etc. Whoever would like to do this, just add the info in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#topic Learn about: mailman3 - kevin
= Meeting end stuff =
#topic Open Floor
#endmeeting
7 years, 11 months
[release] pagure: 2.1.1
by Pierre-Yves Chibon
Good Morning Everyone,
I just cut a new release of pagure: 2.1.1
Together with Clément Verna we worked on fixing the bug that affected the update
of this morning and led to old notifications being re-send.
Here is the changelog
* Fri May 13 2016 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.1.1-1
- Update to 2.1.1
- Do not render the comment as markdown when importing tickets via the ticket
git repo
- Revert get_revs_between changes made in
https://pagure.io/pagure/pull-request/941 (Clement Verna)
It is currently running in staging where Clément is testing it and we should
push to prod shortly.
Happy hacking,
Pierre
7 years, 11 months
[release] pagure: 2.1
by Pierre-Yves Chibon
Good Morning everyone,
I just cut a new release of pagure: 2.1
Here is its changelog:
* Fri May 13 2016 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.1-1
- Update to 2.1
- Fix the milter to get it working (hotfixed in prod)
- Fix the fedmsg hook so that it works fine (hotfixed in prod)
- Fix the path of one of the internal API endpoint
- Pass client_encoding utf8 when connecting to the DB (Richard Marko)
- Do not use client_encoding if using sqlite (Ryan Lerch)
- Allow project names up to 255 characters (Richard Marko)
- Add a spinner showing we're working on retrieve the PR status on the PR page
(farhaanbukhsh)
- Rework installing and removing git hooks (Clement Verna)
- Rework the summary of the changes on the PR page (Ryan Lerch)
- Improve the description of the priority system (Lubomír Sedlář)
- Fix commit url in the pagure hook (Mike McLean)
- Improve the regex when fixing/relating a commit to a ticket or a PR (Mike
McLean)
- Improve the description of the pagure hook (Mike McLean)
- Fix the priority system to support tickets without priority
- Fix the ordering of the priority in the drop-down list of priorities
- Ensure the drop-down list of priorities defaults to the current priority
- Adjust the runserver.py script to setup PAGURE_CONFIG before importing pagure
- Remove flashed message when creating a new project
- Add markdown support for making of PR# a link to the corresponding PR
- Include the priority in the JSON representation of a ticket
- Include the priorities in the JSON representation of a project
- Do not update the assignee if the person who commented isn't an admin
- When adding a comment fails, include the comment text in the form if there was
one
- Add support to remove a group from a project
- Add a roadmap feature with corresponding documentation
- Allow 'kbd' and 'var' html tags to render properly
- Fix deleting a project on disk as well as in the DB
- Allow setting the date_created field when importing ticket from git (Clement
Verna)
- Strip GPG signature from the release message on the release page (Jan Pokorný)
- Make comment on PR diffs fit the parent, and not overflow horiz (Ryan Lerch)
As you see, quite a few fixes in there, I'd like to highlight a newly added
feature: roadmap.
When pushing the release commit I think we triggered a bug in the logic used to
detect the new commits pushed, so I think quite a few users got spammed, I'll
look into reverting the change.
Happy hacking,
Pierre
7 years, 11 months
[release] mdapi: 2.6
by Pierre-Yves Chibon
Good Morning everyone,
I just cut a new release of mdapi: 2.6
Here is the corresponding changelog:
* Thu May 12 2016 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.6-1
- Update to 2.6
- Fix get_repo_md to handle gzip correctly (Patrick Uiterwijk)
- Let the mdapi web app to return JSON with the correct mimetype (Patrick
Uiterwijk)
- Adjust get_repo_md for the new URL structure on the mirrors
As you see, mostly bug fixes.
It is currently happily running in stg, I'm waiting for the next cron run to
make sure the f24 branches show up correctly and then I will push to prod.
Happy querying,
Pierre
7 years, 11 months
[release] pkgdb2: 2.3
by Pierre-Yves Chibon
Good Morning everyone,
I just cut a new pkgdb2 release: 2.3
Here is the changelog:
* Thu May 12 2016 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.3-1
- Update to 2.3
- Optimize the pkgdb2_branch script used when branching
- Add the namespace in the JSON representation of a package
- Use more https in the links (Till Maas)
- Remove no longer needed set_monitoring_status.py scrip (Till Maas)
- Use %%license in the spec file (Till Maas)
- Fix the update_package_info script
- Check the review_url upon asking to un-retire a package
- Enforce a namespace policy for branches (Ralph Bean)
- Fix checking who can un-retire a package via pkgdb2's API
- Fix the pagination when browsing the requests to include the status info
- Use the default value for koschei and monitoring_status if None is provided
- Keep the status change of the packae/branch info when doing mass branching
- Improve the mass-branching script
As you can see, mostly bug fixes and a few improvements to backend tools and the
namespace mechanism (so that the `modules` has a different branch management
than the `rpms` namespace).
This namespace policy has a bug in it but since it's not something we're using
in prod I've pushed this new release to prod as well.
Happy packaging,
Pierre
7 years, 11 months
