infrastructure October 2020

infrastructure@lists.fedoraproject.org

27 participants
27 discussions

About JS framework
by Pierre-Yves Chibon 17 Oct '22

17 Oct '22

Good Morning Everyone, Our infrastructure is mostly a python store, meaning almost all our apps are written in python and most using wsgi. However in python we are using a number of framework: * flask for most * pyramid for some of the biggest (bodhi, FAS3) * Django (askbot, Hyperkitty) * TurboGears2 (fedora-packages) * aiohttp (python3, async app: mdapi) While this makes sometime things difficult, these are fairly standard framework and most of our developers are able to help on all. However, as I see us starting to look at JS for some of our apps (fedora-hubs, wartaa...), I wonder if we could start the discussion early about the different framework and eventually see if we can unify around one. This would also allow those of us not familiar with any JS framework to look at the recommended one instead of picking one up semi-randomly. So has anyone experience with one or more JS framework? Do you have one that would you recommend? Why? Thanks for your inputs, Pierre

10 12

[PATCH 0/1] Add a flatpak-indexer openshift service
by Owen Taylor 01 Dec '20

01 Dec '20

From: "Owen W. Taylor" <otaylor(a)fishsoup.net> This is an initial attempt to create a configuration for flatpak-indexer to replace regindexer and add an image delta capability. The config here is derived from a working openshift configuration, but is untested in this form. See: https://pagure.io/fedora-infrastructure/issue/9272 Open questions: How to propagate content to the registry.fedoraproject.org reverse proxy ======================================================================== Currently the regindexer-generated content is rsync'ed from sundries to fedora-web/registry. How should this be done with flatpak-indexer running as an openshift app? Some possibilities that come to mind: - Run a rsyncd within the openshift app (either as a separate deploymentconfig or as a sidecar to the indexer) and expose a route to it internally in Fedora infrastructure. - Run a web server within the openshift app, expose a route to it internally in Fedora infrasturcture, and reverse proxy the content on fedora-web/registry instead of rsync'ing it. - Write the content onto a netapp volume, and mount that volume RO either on a host running rsyncd or directly on fedora-web/registry. What to use for a redis image ============================= Redis is used for caching and communication between the components. What redis image should be used? registry.redhat.io/rhel8/redis-5 needs configuration of a subscription docker.io/library/redis:5 centos/redis-5-centos7 don't rely on such images currently Custom Dockerfile image built from fedora:32 how would rebuilds be triggered? For the two other images needed here, I used ubi8 images - which aren't currently used elsewhere, but are presumably ok. How to handle identifying versions to build for staging/production ================================================================== I see that most openshift applications simply use 'staging'/'production' tags in the upstream repo, while a few take the approach of having specific hashes checked into the infrastructure ansible repository. Is the upstream tag approach considered sufficiently secure? (Making the service write a malicious index could allow causing users to upgrade to arbitrary application binaries.) Owen W. Taylor (1): Add a flatpak-indexer openshift service playbooks/openshift-apps/flatpak-indexer.yml | 56 +++++ .../reversepassproxy.registry-generic.conf | 34 ++- .../flatpak-indexer/files/imagestream.yml | 52 +++++ .../flatpak-indexer/files/service.yml | 16 ++ .../flatpak-indexer/files/storage.yml | 24 ++ .../flatpak-indexer/templates/buildconfig.yml | 84 +++++++ .../flatpak-indexer/templates/configmap.yml | 98 ++++++++ .../templates/deploymentconfig.yml | 221 ++++++++++++++++++ .../flatpak-indexer/templates/secret.yml | 11 + roles/regindexer/build/tasks/main.yml | 21 -- roles/regindexer/build/templates/config.yaml | 74 ------ 11 files changed, 584 insertions(+), 107 deletions(-) create mode 100644 playbooks/openshift-apps/flatpak-indexer.yml create mode 100644 roles/openshift-apps/flatpak-indexer/files/imagestream.yml create mode 100644 roles/openshift-apps/flatpak-indexer/files/service.yml create mode 100644 roles/openshift-apps/flatpak-indexer/files/storage.yml create mode 100644 roles/openshift-apps/flatpak-indexer/templates/buildconfig.yml create mode 100644 roles/openshift-apps/flatpak-indexer/templates/configmap.yml create mode 100644 roles/openshift-apps/flatpak-indexer/templates/deploymentconfig.yml create mode 100644 roles/openshift-apps/flatpak-indexer/templates/secret.yml delete mode 100644 roles/regindexer/build/tasks/main.yml delete mode 100644 roles/regindexer/build/templates/config.yaml -- 2.28.0

2 3

Another Rust MirrorManager experiment
by Adrian Reber 03 Nov '20

03 Nov '20

Our MirrorManager setup exports the current state of all mirrors every hour at :30 to a protobuf based file which is then used by the mirrorlist servers to answer the requests from yum and dnf. The Python script requires up to 10GB of memory and takes between 35 and 50 minutes. The script does a lot of SQL queries and also some really big SQL queries joining up to 6 large MirrorManager tables. I have rewritten this Python script in Rust and now it only needs around 1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB. I think the biggest difference is that I am almost not doing any joins in my SQL request. I download all the tables once and then I do a lot of loops over the downloaded tables and this seems to be massively faster. As the mirrorlist-server in Rust has proven to be extremely stable over the last months we have been using it I would also like to replace the mirrorlist protbuf input generation with my new Rust based code. I am planing to try out the new protobuf file in staging in the next days and would then try to get my new protobuf generation program into Fedora. Once it is packaged I would discuss here how and if we want to deploy in Fedora's infrastructure. Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies. Adrian

4 17

Mirrorlist bad entry for ftp.icm.edu.pl resulting 404
by Mariusz Brzezik 30 Oct '20

30 Oct '20

Dear community, recently i we faced a errors during machines creation phase due to bad entry for ftp.icm.edu.pl within EPEL mirrorlist. Checked for CentOS 7 and 8. Centos 7 entry giving 404 error: https://ftp.icm.edu.pl/pub/Linux/fedora/linux/epel/7/x86_64/ Working link for CentOS7: https://ftp.icm.edu.pl/pub/Linux/dist/fedora-epel/7/x86_64/ Centos8 entry giving 404 error: http://ftp.icm.edu.pl/pub/Linux/fedora/linux/epel/8/Everything/x86_64/ Working link for CentOS8: https://ftp.icm.edu.pl/pub/Linux/dist/fedora-epel/8/Everything/x86_64/ Who should i contact with to issue those mirrorlist errors? Thank You in advance for You help!

4 6

Fedora 33 Final Freeze now in effect!
by Kevin Fenzi 29 Oct '20

29 Oct '20

Greetings. we are now in the infrastructure freeze leading up to the Fedora 33 Final release. This is a final release freeze. We do this to ensure that our infrastructure is stable and ready to release Fedora 33 when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://infrastructure.fedoraproject.org/infra/ansible.git ansible/scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2020-10-21 (or later if release slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list. Thanks, kevin

2 5

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 28 Oct '20

28 Oct '20

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2020-10-29 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

1 0

[FBR] Bump the IoT stable keys to fedora-33
by Patrick Uiterwijk 28 Oct '20

28 Oct '20

Hi, Can I get acks for the following patch to move the stable IoT composes to the Fedora 33 key? (Note: there's a single "empty" line change as well, that was a line with purely whitespace, which my IDE automatically cleans up.) Thanks, Patrick commit 3ee8dd3f9e883414c7846d115d2926f73ec9ea99 (HEAD -> master) Author: Patrick Uiterwijk <patrick(a)puiterwijk.org> Date: Wed Oct 28 11:26:44 2020 +0100 Bump the IoT stable keys to fedora-33 Signed-off-by: Patrick Uiterwijk <patrick(a)puiterwijk.org> diff --git a/roles/robosignatory/templates/robosignatory.toml.j2 b/roles/robosignatory/templates/robosignatory.toml.j2 index 6a933bda9..70fde0d55 100644 --- a/roles/robosignatory/templates/robosignatory.toml.j2 +++ b/roles/robosignatory/templates/robosignatory.toml.j2 @@ -380,7 +380,7 @@ handlers = ["console"] key = "{{ (env == 'production')|ternary('fedora-34', 'testkey') }}" keyid = "{{ (env == 'production')|ternary('45719a39', 'd300e724') }}" type = "modular" - + # ELN Mass Rebuild [[consumer_config.koji_instances.primary.tags]] @@ -445,13 +445,13 @@ handlers = ["console"] [consumer_config.ostree_refs."fedora/stable/x86_64/iot"] directory = "/mnt/fedora_koji/koji/compose/iot/repo/" - key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}" + key = "{{ (env == 'production')|ternary('fedora-33', 'testkey') }}" [consumer_config.ostree_refs."fedora/stable/aarch64/iot"] directory = "/mnt/fedora_koji/koji/compose/iot/repo/" - key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}" + key = "{{ (env == 'production')|ternary('fedora-33', 'testkey') }}" [consumer_config.ostree_refs."fedora/stable/armhfp/iot"] directory = "/mnt/fedora_koji/koji/compose/iot/repo/" - key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}" + key = "{{ (env == 'production')|ternary('fedora-33', 'testkey') }}" [consumer_config.ostree_refs."fedora/29/x86_64/atomic-host"] directory = "/mnt/fedora_koji/koji/compose/ostree/repo/"

3 2

Auth test apps in staging
by Aurelien Bompard 27 Oct '20

27 Oct '20

Hey folks, To test authentication with the new AAA system I'd like to deploy a couple very basic apps that do nothing but auth in staging's openshift. It shouldn't touch any configuration besides the reverse proxies and the new project in openshift. And it's staging only. Is it OK? Thanks. Aurélien

4 7

Re: master mirror slow sync
by Kevin Fenzi 26 Oct '20

26 Oct '20

On Mon, Oct 26, 2020 at 04:00:06PM +0100, Jan Kasprzak wrote: > Kevin Fenzi wrote: > : On Mon, Oct 26, 2020 at 11:12:32AM +0100, Jan Kasprzak wrote: > : > Hello, > : > > : > > : > Kevin Fenzi wrote: > : > : As some of you may know, there's been reports of slow sync times for > : > : content from the master mirrors last week. > : > : (see: https://pagure.io/fedora-infrastructure/issue/9392 ) > : > > : > ftp.linux.cz Tier1 mirror admin here. Thanks for the heads up. > : > > : > : * Our two mirrors located in other datacenters: > : > : download-ib01.fedoraproject.org > : > : and > : > : download-cc-rdu01.fedoraproject.org > : > > : > None of the above contain the fedora-enchilada0 rsync package, > : > and in the fedora-releases package there is no 33 subdirectory. > : > Which package should I use for rsyncing? > : > : Odd. They do have it defined. Just like the main master mirrors. ;( > : > : It could be something to do with them having ipv6 addresses and you > : coming from a different ipv6 address? We have > : ftp.linux.cz > : in rsyncd.conf... so it should reverse lookup the ip and allow it, just > : like the iad2 masters. ;( > > OK, the reverse lookup would be a problem then. Adding -4 to the > rsync command line fixes the problem - I can now download from > download-ib101.fp.o. > > Can you add 2001:718:801:230::cd to the ACL? Unfortunately, I have > many different FQDNs pointing to my server, so PTRs can be inconsistent > (my fault, I know). We can. Would it be ok to add it in a few days? Or do you need it now? Thanks! kevin

1 0

Re: master mirror slow sync
by Kevin Fenzi 26 Oct '20

26 Oct '20

On Mon, Oct 26, 2020 at 11:12:32AM +0100, Jan Kasprzak wrote: > Hello, > > > Kevin Fenzi wrote: > : As some of you may know, there's been reports of slow sync times for > : content from the master mirrors last week. > : (see: https://pagure.io/fedora-infrastructure/issue/9392 ) > > ftp.linux.cz Tier1 mirror admin here. Thanks for the heads up. > > : * Our two mirrors located in other datacenters: > : download-ib01.fedoraproject.org > : and > : download-cc-rdu01.fedoraproject.org > > None of the above contain the fedora-enchilada0 rsync package, > and in the fedora-releases package there is no 33 subdirectory. > Which package should I use for rsyncing? Odd. They do have it defined. Just like the main master mirrors. ;( It could be something to do with them having ipv6 addresses and you coming from a different ipv6 address? We have ftp.linux.cz in rsyncd.conf... so it should reverse lookup the ip and allow it, just like the iad2 masters. ;( If you can drop by #fedora-admin on irc.freenode.net we can try and debug it? Or I can just add some hard coded IPv4/IPv6 addresses for you? > : may provide faster speeds than the dl-tier01.fedoraproject.org mirrors > : right now depending on your location/routing, so you may want to at > : least temporarily sync against them. > > So far I have manually hardlinked releases/test/33_Beta tree to > releases/33, and I am running rsync of > rsync://dl-tier1.fedoraproject.org/fedora-enchilada0/linux/releases/33/ > on top of it. The download speed around 50 Mbit/s. > > Is there a faster way how to get my mirror in sync? Thanks, Well, fedora-quick-mirror will only copy those specific files you still need and won't cause our end to have to stat the entire tree so you can compare it (like bare rsync does). But otherwise that should work... kevin

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure October 2020