infrastructure September 2018

infrastructure@lists.fedoraproject.org

35 participants
56 discussions

About JS framework
by Pierre-Yves Chibon 17 Oct '22

17 Oct '22

Good Morning Everyone, Our infrastructure is mostly a python store, meaning almost all our apps are written in python and most using wsgi. However in python we are using a number of framework: * flask for most * pyramid for some of the biggest (bodhi, FAS3) * Django (askbot, Hyperkitty) * TurboGears2 (fedora-packages) * aiohttp (python3, async app: mdapi) While this makes sometime things difficult, these are fairly standard framework and most of our developers are able to help on all. However, as I see us starting to look at JS for some of our apps (fedora-hubs, wartaa...), I wonder if we could start the discussion early about the different framework and eventually see if we can unify around one. This would also allow those of us not familiar with any JS framework to look at the recommended one instead of picking one up semi-randomly. So has anyone experience with one or more JS framework? Do you have one that would you recommend? Why? Thanks for your inputs, Pierre

10 12

epel7-infra and epel7 conflicts wrt pagure updates (or, can we have an epel7-infra DistTag?)
by Neal Gompa 02 Oct '18

02 Oct '18

Hey all, So as you all know, Pagure 5.0 was released[1]. With 5.0, I've taken over primary maintenance of the package outside of epel7-infra, which means that Fedora and EPEL7 packages are generally maintained by me (with packaging changes generally synced from upstream), since Pierre-Yves doesn't have time to maintain it. We've already received the approval to re-sync pagure's EPEL7 package with Fedora to update to 5.0[2]. However, this presents a bit of a problem. Pierre-Yves updates the pagure package for epel7-infra before making the general release that I can consume and push to Fedora and EPEL7. This wouldn't be an issue, except that the DistTag for epel7-infra is the same as epel7 (.el7). This means I'm locked out of that NEVR for EPEL7 updates, due to the nature of Koji considering NEVR of builds globally unique. This is frustrating because it means that I can't maintain the common spec file for both Fedora and EPEL7 (which is in turn primarily maintained upstream in the pagure git repo). Of course, I can just bump the Release to be above the epel7-infra build, but that means I'm playing NEVR games, and I don't want to do that. And if there needs to be distinct changes for epel7-infra, this is going to be an unpleasant fight. I'd like to propose that we change epel7-infra so that the DistTag defined in %dist is ".el7.infra" instead of ".el7". This will prevent the conflict and have the advantage that the infra tag is higher by default. For this current update round, I'm going to be putting a higher Release value because I need to anyway, as there's a bug in the way pagure is done that breaks packaging in Fedora and openSUSE[3]. But in the future, I'd like to have the freedom to use the correct NEVR everywhere. What do you all think? [1]: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr… [2]: https://meetbot.fedoraproject.org/teams/epel/epel.2018-09-26-17.01.html [3]: https://pagure.io/pagure/issue/3706 -- 真実はいつも一つ！/ Always, there's only one truth!

4 5

The trouble with torrents
by Kevin Fenzi 30 Sep '18

30 Sep '18

So, Fedora 29 Beta is out the door and mostly that went very smoothly as always. With one exception: torrents. Right now we are (were) using the last opensource version of 'bittorrent' python client to seed our torrents. However, I noticed yesterday that it is no longer working. It's not just one bug, but a number of them judging by the different tracebacks it's spewing. As a stopgap to get Beta happy I installed and ran rtorrent on torrent01 in a tmux session to seed our beta torrents, but thats a curses app thats not automateable. So, we really need to figure a way forward here. 1. Get permission from Council/FESCo to drop torrents (To make that case we need to show some graphs that not many people are using torrents anymore). 2. Move our torrents to amazon cloudfront. This should be doable, but means we are using a non free service so it would make me sad. 3. Try and find a open source seeding app that is manageable from ansible/config files. (Good luck, they are all old, or gui only, etc). 4. Just use rtorrent in tmux to seed everything and have to manually start/stop it when you add/remove any. 5. Some other clever idea someone has. We can discuss this in meeting tomorrow, but I figured I would start a thread here too. kevin

7 8

[PATCH] Add a RabbitMQ vhost for pubsub (fedmsg) and configure an HA policy
by Jeremy Cline 29 Sep '18

29 Sep '18

Since many of our applications will likely wish to use RabbitMQ for private things like work queues, using separate vhosts is a good idea. This sets up a single vhost to act as a fedmsg replacement (called pubsub) and configures it for high availability. The HA policy mirrors queues to all nodes in the cluster and enables automatic synchronization with a batch size of 10K messages. Automatic synchronization means that when a new node joins the cluster, all messages from existing nodes are pushed to the new node, blocking queue activity until mirroring is complete. Signed-off-by: Jeremy Cline <jcline(a)redhat.com> --- roles/rabbitmq_cluster/files/enabled_plugins | 1 - roles/rabbitmq_cluster/tasks/main.yml | 38 +++++++++++++++----- 2 files changed, 29 insertions(+), 10 deletions(-) delete mode 100644 roles/rabbitmq_cluster/files/enabled_plugins diff --git a/roles/rabbitmq_cluster/files/enabled_plugins b/roles/rabbitmq_cluster/files/enabled_plugins deleted file mode 100644 index 6a9f28b93..000000000 --- a/roles/rabbitmq_cluster/files/enabled_plugins +++ /dev/null @@ -1 +0,0 @@ -[rabbitmq_management,rabbitmq_auth_mechanism_ssl]. diff --git a/roles/rabbitmq_cluster/tasks/main.yml b/roles/rabbitmq_cluster/tasks/main.yml index 468cbc5ac..1781eb2c4 100644 --- a/roles/rabbitmq_cluster/tasks/main.yml +++ b/roles/rabbitmq_cluster/tasks/main.yml @@ -52,15 +52,6 @@ - rabbitmq_cluster - config -- name: enable plugins - copy: src=enabled_plugins dest=/etc/rabbitmq/enabled_plugins owner=root group=root mode=0644 - with_items: - - rabbitmq.config - - enabled_plugins - tags: - - rabbitmq_cluster - - config - - name: deploy configuration template: src={{item}} dest=/etc/rabbitmq/{{item}} owner=root group=root mode=0644 with_items: @@ -96,6 +87,13 @@ - rabbitmq_cluster - config +- name: Enable the HTTP management console and SSL authentication plugins + rabbitmq_plugin: + names: rabbitmq_management,rabbitmq_auth_mechanism_ssl + tags: + - rabbitmq_cluster + - config + - name: start rabbitmq service: name=rabbitmq-server state=started enabled=yes tags: @@ -106,3 +104,25 @@ tags: - rabbitmq_cluster - config + +- name: Configure the pubsub virtual host + rabbitmq_vhost: + name: /pubsub + state: present + tags: + - rabbitmq_cluster + - config + +- name: Configure the HA policy for queues on the pubsub virtual + rabbitmq_policy: + name: HA + apply_to: queues + pattern: .* + tags: + ha-mode: all + ha-sync-mode: automatic # Auto sync queues to new cluster members + ha-sync-batch-size: 10000 # Larger is faster, but must finish in 1 net_ticktime + vhost: /pubsub + tags: + - rabbitmq_cluster + - config -- 2.19.0

3 2

Data on torrents usage
by Stephen John Smoogen 29 Sep '18

29 Sep '18

So I updated my stats from 2017 and made new graphs to show how usage of torrents has progressed since 2012. https://smooge.fedorapeople.org/stats-torrents/ Due to the messiness of the logs.. the data logged is in an hourly file which gives a checksum of the image being shared without any knowledge of what that image is. [This is because tracker is meant to be open and allow anyone to track whatever they want.] The logs only say things like the number of peers, and the number of clients asking for data. It isn't clear if it is showing the amount of actual downloads completed or not. I decided to say it was.. but if that is wrong, oops. In general, I believe we should not stop producing torrents that people can use. Running a tracker and dedicated seeder for that tracker may be less useful for us because of the amount of work it takes to keep it going. The problems are that development on headless opensource trackers and seeders ended in 5-7 years ago. Someone in release engineering has to patch something each release to keep it going. In looking at various distributions, we seem to be the only one still running a tracker. The others just produce torrent files and seem to let the users find peers where-ever they can. -- Stephen J Smoogen.

1 0

MirrorManager changes - report_mirror
by Adrian Reber 28 Sep '18

28 Sep '18

During the last months most mirror problems have been related to report_mirror reporting that mirrors are up to date and the crawler then marking the mirror as not up to date. Which resulted in mirrors being added and removed from the mirrorlist every few hours. As the crawler normally sees what the user sees, means that report_mirror often reports the wrong mirror status. I would like to ignore report_mirror results for non-private mirrors in the future. For private mirrors report_mirror is the only way to know which content it has, for public mirrors it is not really necessary as the mirrors are crawled anyway and report_mirror does not contain enough information anyway right now. Any objections to ignore report_mirror reports for public mirrors after the F29 release. Adrian

3 4

fedora-tagger and statscache sunset
by Clement Verna 27 Sep '18

27 Sep '18

Dear all, As previously announced [0][1] fedora-tagger and statscache will be retired next Tuesday Oct 02 2018 at 21.00 UTC. Thanks [0] - https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr… [1] - https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr…

1 0

[Release] pagure 5.0
by Pierre-Yves Chibon 27 Sep '18

27 Sep '18

Good Morning Everyone, On Monday, I released a new version of pagure: 5.0. Here is its changelog: 5.0 (2018-09-24) ---------------- .. warning:: This release contains backward incompatible changes and fixes a CVE - Pagure supports now python2 and python3 simultaneously (Thanks to Aurélien Bompard and Neal Gompa for the testing) - New UI deployed (thanks to Ryan Lerch) - New dashboard page as index page when authenticated (Ryan Lerch) - API listing items (projects, issues, pull-requests are now paginated ( Karsten Hopp) .. warning:: Backward incompatible - Enable private projects by default (Neal Gompa) .. warning:: Backward incompatible - Change the default and sample configuration to point to localhost-friendly resources (Neal Gompa) .. warning:: Backward incompatible - Disable sending FedMsg notifications by default (Neal Gompa) .. warning:: Backward incompatible - Switch default authentication to `local` (Neal Gompa) .. warning:: Backward incompatible - Drop the dependency on python-pygments - Drop the dependency on flask-multistatic - Drop the dependency on python-trollius (in favor of python-trololio) (Neal Gompa) - Bump pygit2 requirement to 0.24.0 minimum .. warning:: Backward incompatible - Add support to re-open a pull-request (Karsten Hopp) - Fix editing a file into a fork containing a namespace - Allow creating a new API token based on an expired one - New API endpoint to submit a pull-request - Add support for making the issue tracker read-only - Add a new API endpoint allowing to update watch status on a project - Paginate the project lists on the front pages - Let the reply button append instead of replacing - Add a way to list all API tokens and don't restrict the info command ( in pagure-admin) - Expand pagure-admin to allow using it to block an user - Expand pagure-admin to allow adding new groups using it - Allow viewing commits from a git tag - Support viewing commits from a specific commit hash - Add a hook that disables creating new branches by git push (Slavek Kabrda) - Make API endpoint for creating new git branch have its own ACL - Support sorting PR's by recent activity (ymdatta) - Fix installing the API key reminder cron with systemd integration - Add reactions to comments (Lubomír Sedlář) - New API endpoint allowing to retrieve pull-requests based on their UUID (Slavek Kabrda) - Add an option to restrict emails sent to certain domains (Karsten Hopp) - Integration with repospanner (Patrick Uiterwijk) - Rework how git hooks work to rely on a single file rather than moving files around (Patrick Uiterwijk) - Add themes for pagure.io, src.fedoraproject.org (Ryan Lerch) - Add themes for OpenSUSE (hellcp) - Ensure remote PR are opened from a remote URL (CVE-2018-1002158 - reported by Patrick Uiterwijk) This release is available: - In Fedora - In Mageia - In OpenSUSE - soon in EPEL This release has been deployed in staging: stg.pagure.io and src.stg.fedoraproject.org and in production in pagure.io yesterday. We ran into some interesting issues in production that we didn't see in staging, leading to some bug fixes and a version 5.0.1 to be released soon. I take this opportunity to advertise that pagure now has two mailing lists: - pagure-devel to discuss about the development of pagure: https://lists.pagure.io/admin/lists/pagure-devel.lists.pagure.io/ - pagure-announce to follow announcement about new releases of pagure https://lists.pagure.io/admin/lists/pagure-announce.lists.pagure.io/ feel free to join them and keep in touch with pagure :) Future pagure release announcement will be sent to: pagure-announce(a)lists.pagure.io Happy hacking! Pierre

1 0

Meeting Agenda Item: Introduction Your Name
by David Shier 27 Sep '18

27 Sep '18

Hello all, My name is David Shier and I am coming back to fedora admin after a year or two away. My iRC nick is odin2016. Up until about a month and a half ago, I worked for Jackson in the US as a number of things, but most relevant to this is my last two positions, as an admin on their UNIX team, supported our RHEL (5,6,7) and AIX estates, and working in on the more or less server operations team as a senior associate with root access to the production environment. I am familiar with bash, python, and perl, as well as Ansible, which I used at work and at home. I'd like to improve my Ansible and python skills, as well as just keeping my self sharp while I search for a job after my recent move to Germany. As would be expected I am a linux user at home, having used redhat/centos, Debian/ubuntu, and Archlinux, as well as a number of forks of each, Fedora included, but I am also a distro hopper and am currently messing around with elementary on my laptop, and once the rest of my stuff gets here... whenever that is.... i'll be reimaging the raspberry pi and tossing awx on there. At my last job I worked a good bit with RHDS, and passing data into it from Active Directory, as I have experience on both sides of that fence as well. I'm sure there things here I could say, and probably should, but i've also blocked out large bits of the last year or so, so there is that. There are a number of tickets that I would *like * to work on, but before i get to deep into that I need to go and read up on how thiings int eh infra have changed since I last took a look, clone the ansible repo and read though there, and so on. I also think I saw something about updating somethings to python 3, or making sure they will work. I may take a look at those as well. I thought I saw something about the bots as well, but now I don't so may be I am crazy. I have worked with them before, so if I see it I may just grab it. I should be at the meeting Thursday (I have a thing at 17:00 CEST), unless I did the math on time offset really wrong or something, and will get myself added to the goby doc in the next little bit here. If someone could pop me back into fi-apprentice, that would be great.. fas is odin2016 as well. Thanks, Dave Shier

2 2

FBR: upgrading pagure to 5.0
by Pierre-Yves Chibon 27 Sep '18

27 Sep '18

Good Morning Everyone, I just cut a new release of pagure: 5.0, the formal announce is pending polishing things with mailman so we can include in the announce the new mailing lists to discuss/follow pagure. Pending this, I'd like to request a Freeze Break Request for upgrading pagure to 5.0. It's not critical for the release of the beta tomorrow, and it saves one day (vs waiting for Wed) for the upgrade which considering some deadlines intern to the infra team may be good. What do you think? Thanks, Pierre

5 5

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure September 2018