infrastructure January 2016

infrastructure@lists.fedoraproject.org

23 participants
31 discussions

Support levels and RFR adjustments
by Kevin Fenzi 07 Apr '16

07 Apr '16

So, I sent an email a while back about this to get people thinking, but I didn't get too much feedback from my questions, so this time I am going to actually outline a proposal for people to look at. ;) Currently users expect pretty much any public service we have is fully supported. This means things like updating status when it's down, working anytime something is down to fix it as quickly as we can. New applications/services currently all pass through the (somewhat long) RFR process which we setup to make sure we could support the service moving forward. This is great and all, but some services just aren't as sustainable, or don't really fit into our RFR process very well. Also, our RFR process makes us pretty slow to bring a new service online properly. In order to have support levels, we need a way to communicate that to our users easily and the only/best way I can think of to do that easily is via domain name. If we try and have a table or something it could get pretty confusing for people. Tying it to domain names would make it much easier. So: fedoraproject.org - Anything with this domain is something that has passed though our RFR process and we support fully. This means we update status, we alert on them anytime they have issues, we work on them anytime they are down, etc. fedorainfracloud.org - This comes with a lesser level of support, simply because our cloud doesn't have any kind of HA setup, so it will be down when doing maint or when there's problems. Services in this domain may be down when there is scheduled cloud maint. We monitor, but don't page off hours, we may work on issues only during business hours, etc. Services here may not have passed through our RFR process (perhaps we should have a parallel cloud process) stg.fedoraproject.org - These can be down anytime and we monitor on them, but may not work on them off hours, etc. someother domain that sounds fedora related (fedorarelated.org? fedoralinks.org? ?) - These are things that are fedora related, but not fully controlled by fedora infrastructure. Things like the fedora bootstrap site or the porting python3 in fedora site, or possibly cloud instances that aren't managed by us. These we don't monitor or have status on, and direct people to contact the managers directly. Any other types of sites / domains people can think of? Any general thoughts on the idea? kevin

7 13

ansible 2.0 on batcave01
by Kevin Fenzi 15 Feb '16

15 Feb '16

With Toshio porting our callback plugins last night I have gone ahead and moved batcave01 over to ansible 2.0. I'm sure we will hit some minor issues, but for the most part I think we should be good to go. If you do run into something, please fix it, or report it so we can fix it up. Playbooks should all run fine as normal, some scripts may still need porting. The only other thing related to ansible 2.0 I can think of is that copr may need to adjust to the new API if it's using that directly, but it can do that on it's own timeframe. kevin

2 5

a new MirrorManager-related tool for stripping old metadata and pushing critical updates in a defined time
by Kamil Paral 15 Feb '16

15 Feb '16

Hello, before Christmas QA started discussing changes in the release blocker process for bugs that are not media-related (i.e. don't need to be fixed in the generated .iso/.img files). The conversation is available here: https://lists.fedoraproject.org/archives/list/test%40lists.fedoraproject.or… I took interest in one specific type of such bugs, which are release blockers in one of the existing stable releases (i.e. Branched-1 or Branched-2). These are usually related to the upgrade process. We later decided to mark them with AcceptedPreviousRelease flag in Bugzilla, so I'll call them like this. This email is concerned just with these bugs, i.e. just one type of the non-media-related blockers. (I want to separate the overall topic into several separate discussions, so that it's easier to talk about it and we don't muddy the discussion with too many things at once). For these AcceptedPreviousRelease blockers, we want to make sure that all of this happens before we announce Branched release: a) their updates are pushed to Branched-1 or Branched-2 updates repo b) the contents of those repos are available to all our users, also considering all caches and refresh intervals Ensuring a) is easy and will be tracked by QA and Bodhi as usual, but for b) we will need help from releng/infra. Since many users upgrade immediately on release day, we really need to make sure the updates are available for everybody by that time, otherwise a large portion of our user base will get affected by those blocker bugs. I have tried to investigate how best to ensure the latest repo contents are available to everyone, and I described it here: https://lists.fedoraproject.org/archives/list/test%40lists.fedoraproject.or… As the easiest way to achieve this, I suggested we create a new MirrorManager-related tool which will strip all old metadata from the repo metalink. Dennis from RelEng agreed he could run such a tool in these circumstances to make sure only latest metadata are distributed. I suppose the tool could work something like this: 1. A blocker update was pushed to f23-updates on 2015-01-07. 2. Releng will run the tool like this: $ ./mm-strip-old-metadata --release 23 --repo updates --date 2015-01-07 (This is assuming there is only one push per day, if not, maybe we can have --timestamp instead of --date). 3. The tool will go through all metalinks for f23-updates (all primary architectures), and drop each <mm0:alternate> section which has <mm0:timestamp> older than the provided date (let's say midnight UTC). 4. As a result, end-user machines will only connect to repositories which already serve the blocker update (have that or newer repo tree). And now finally the important question - does infrastructure team thinks this is easily doable, or is there something not accounted for? Is here someone willing to create such a tool? In December, I talked to Adrian Reber and I got the impression he's a MirrorManager developer, so that's the only name I know of, but I don't want to bother anyone directly. Are there more of MM developers? Is anyone willing to help out with this? Thanks a lot, Kamil

3 5

pagure tickets brainstorming
by Kevin Fenzi 04 Feb '16

04 Feb '16

So, a number of our applications/projects have moved from github or fedorahosted to pagure, which is coming along nicely. :) We do have a number of "projects" on fedorahosted that aren't really software projects, but use fedorahosted almost completely for trac. trac is... not enjoyed by many, so it would be nice to look at what we would need to add to pagure tickets handling in order to cover those cases so we could look at moving some of these 'projects' over to pagure. ;) So, first many projects use the trac "wiki" for a page with docs on how to file tickets, etc. I think pagure docs already should handle this case very nicely, although perhaps there should be an option to make the 'docs' page the default for a project instead of 'overview' ? Pagure also has blocking/deps and private tickets. Now, on to the things I think might be desired: * Custom ticket statuses (some projects use this to make statuses that are more descriptive for their project, like "upstream" or "accepted" or whatever. This might require splitting the status of tickets to open or closed 'status' and have a seperate 'resolution' or something. * Tagging of issues. Tons of projects use a 'meeting' keyword to mark tickets they want to discuss in meetings. A way to display only tagged tickets would be good and a bonus would be a irc friendly meeting output to copy and paste. I see a "Tags:" field, but not how to populate it. Is this in progress? * A way to cc or bcc a group of people on all tickets in a project. Do we already have this? * Milestones (but I am not sure how much these are used). Some projects have "Fedora 24 Alpha" "Fedora 24 Beta" type milestones for things to be finished before some event. Perhaps we just want to drop this idea in favor of some kind of deadline listing and emiting a message when the deadline is reached? "This ticket was supposed to be done by now!" * Templates. We use these a lot in infrastructure. Basically when filing a new issue there's a list of templates and when someone selects one it sets the initial contents and assigned and such. These are handy for making sure users give the needed info for a type of request. * Theres a batch modify plugin that lets you modify a bunch of tickets at once. I don't think this is critical, but might be nice to have. * Is there a way to completely delete a ticket? Sometimes we have done that on trac for spam tickets. Thats all I can think of off hand... can other folks think of things we use trac tickets for in the projects that are primarily using trac only? kevin

3 7

Meeting Agenda Item: Introduction - Anshu Prateek
by Anshu Prateek 04 Feb '16

04 Feb '16

Hi all, I am Anshu Prateek (@anshprat), a Fedora Infrastructure Apprentice Alumni from 2013 [1][2]. I have been a regular user of Fedora since FC3, and I love to contribute back whenever possible[3]. My day job is that of a devops and tools/platforms developer. I ve been a passive member after I left, though regularly reading most of the emails on the list here. I can now afford some time (and in part motivated by https://www.hackerearth.com/sprints/open-source-india-hacks-2016/ ) to become an active member again. To start off, I would take a jab at https://github.com/fedora-infra/pkgdb2/issues/299 and https://fedorahosted.org/fedora-infrastructure/ticket/5084 I would need to be added to fi-apprentice to look at the logs for the above. Looking forward to working with the community again :) cheers Anshu Prateek [1] https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr… [2] https://fedorahosted.org/fedora-infrastructure/query?status=closed&owner=an… [3] https://badges.fedoraproject.org/user/anshprat -- regards Anshu Prateek +91.991.610.2967

2 2

Infra YIR draft
by Zach Villers 02 Feb '16

02 Feb '16

Here is a draft of a Year In Review post for the Community Ops Blog. I left some things blank (and omitted ALOT - wow - lost of stuff happened in 2015). I didn't take a stab at the conclusion or 2016 goals section. JFlory7 had asked Infra if we would submit a YIR post. Kevin and Ralph (Nirik/Threebean) also had some ideas. I left out ticket and outage numbers, but can add in something if we want. This is formatted as markdown. Introduction ------------ The Infrastructure Team consists of dedicated volunteers and professionals managing the servers, building the tools and utilities, and creating new applications to make Fedora development a smoother process. We're located all over the globe and communicate primarily by IRC and e-mail. Top 3 Highlights ---------- * Ansible Migration - We believe Ansible is the best new technology for systems deployment and management. This year, Infrastructure team moved all remaining Puppet recipes (78 at start of FY2016) in the infrastructure to Ansible playbooks. The automation provided by Ansible allows us to quickly fix/rebuild/scale our existing services and deploy new services. * HyperKitty Deployment - HyperKitty is a web front end to the new Mailman version 3 which allows users to browse topics in a more familiar, forum-like interface. We will complete development of this application and deploy for use with Fedora mailing lists. * Bodhi 2 - Pronounced as bo-dee is a buddhist term for the wisdom by which one attains enlightenment. Bodhi is a modular web-based system that facilitates the process of publishing package updates for Fedora. It maintains a single stage of repositories by adding/updating/removing packages. * MirrorManager 2 - * Koschei launch - Koschei is a continuous integration service for Fedora packages. Koschei is aimed at helping Fedora developers by detecting problems as soon as they appear in rawhide - it tries to detect package FTBFS in rawhide by scratch-building them in Koji. * RHEL 6 to 7 conversion - Top Goal(s) for 2016 -------------------- 1. 1. 1. Conclusion ---------- Links ----- * https://bodhi.fedoraproject.org/ * https://apps.fedoraproject.org/koschei * https://apps.fedoraproject.org/

4 4

isn't mirrormanager smarter than this?
by Matthew Miller 29 Jan '16

29 Jan '16

I'm getting 404 errors from several places to which mirrormanager wants to send me to get https://download.fedoraproject.org/pub/alt/atomic/stable/Cloud-Images/x86_6… (Specifically, mirrors.rit.edu and mirrors.kernel.org.) Presumbably, these just haven't synced yet. But doesn't mirrormanager get feedback about when mirrors are fresh? -- Matthew Miller <mattdm(a)fedoraproject.org> Fedora Project Leader

4 6

[release] pagure: 1.0
by Pierre-Yves Chibon 28 Jan '16

28 Jan '16

Good Morning everyone, A few hours ago I cut a new release of pagure: 1.0 The changelog is pretty large: * Wed Jan 27 2016 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 1.0-1 - Update to 1.0 - Entirely new UI thanks to the hard work on Ryan Lerch - Add the possibility to edit comments on PR/Tickets (and the option to disable this) (farhaanbukhsh) - Add the number of open Tickets/PR on the project's menu - Also allow PRs to be closed via a git commit message (Patrick Uiterwijk) - Disable issues and PR on forks by default (Vivek Anand) - Fix deleting the temporary folders we create - Un-bundle flask_fas_openid (requires python-fedora 0.7.0 or higher - Add support for an openid backend (ie same thing as FAS but w/o the FPCA enforcing) - Add support to view rst/markdown files as html directly inline (default) or as text (Yves Martin) - Change the encryption system when using pagure with local auth to not be time-sensitive and be stronger in general (farhaanbukhsh) - Change the size of the varchar from 256 to 255 for a better MySQL support - Add support for pagure to work behind a reverse proxy - Rename the cla_required decorator to a more appropriate login_required - Show the in the front page and the page listing all the pull-requests the branch for which a PR can be opened - Rework the avatar to not rely on the ones associated with id.fedoraproject.org - Add support to high-light a section of code in a PR and show the diff automatically if there is such selection This is currently running in prod, with one hotfix and I've been working on trying to fix another issue I've seen. So do expect a 1.0.1 soonish :) The new UI is the work of Ryan Lerch, so don't hesitate to give him cookies on irc: ryanlerch++ Happy hacking! Pierre

2 1

EasyFix 4507
by Zach Villers 27 Jan '16

27 Jan '16

All - I've assigned Easyfix Ticket 4507 to myself (aikidouke) - If you are working on it, please let me know/find me on IRC. Thanks, Zach

1 0

SPF and email forwarding
by Miroslav Suchý 27 Jan '16

27 Jan '16

Today I sent email to packager-sponsors(a)fedoraproject.org and several email returned back due SPF protection. Can someone either implement this: http://www.openspf.org/Best_Practices/Forwarding or turn those email aliases to mailing list? Mirek -------- Přeposlaná zpráva -------- Předmět: Undelivered Mail Returned to Sender Datum: Mon, 25 Jan 2016 10:18:13 +0000 (UTC) Od: Mail Delivery System <MAILER-DAEMON(a)fedoraproject.org> Komu: msuchy(a)redhat.com This is the mail system at host bastion02.phx2.fedoraproject.org. I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below. For further assistance, please send mail to postmaster. If you do so, please include this problem report. You can delete your own text from the attached returned message. The mail system <fedora(a)leemhuis.info> (expanded from <packager-sponsors(a)fedoraproject.org>): host mail.leemhuis.info[195.137.212.29] said: 550 5.7.1 <fedora(a)leemhuis.info>: Recipient address rejected: Please see http://www.openspf.org/Why?s=mfrom;id=msuchy%40redhat.com;ip=209.132.181.3;… (in reply to RCPT TO command) <zbyszek(a)in.waw.pl> (expanded from <packager-sponsors(a)fedoraproject.org>): host kawka.in.waw.pl[178.62.225.244] said: 550-[SPF] 209.132.181.3 is not allowed to send mail from redhat.com. Please 550 see http://www.openspf.org/Why?scope=mfrom;identity=msuchy@redhat.com;ip=209.13… (in reply to RCPT TO command) <Christian.Iseli(a)unil.ch> (expanded from <packager-sponsors(a)fedoraproject.org>): host mx.unil.ch[130.223.27.62] said: 550 209.132.181.3 is not allowed to send mail from redhat.com (SPF failure) (in reply to RCPT TO command) <nicolas.mailhot(a)laposte.net> (expanded from <packager-sponsors(a)fedoraproject.org>): host smtpz4.laposte.net[194.117.213.1] said: 550 5.5.0 SPF: 209.132.181.3 is not allowed to send mail. LPN007_401 (in reply to MAIL FROM command)

2 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure January 2016