infrastructure May 2015

infrastructure@lists.fedoraproject.org

55 participants
80 discussions

Agenda Item: Introduction Larrry Brower
by Brower, Larry 15 May '15

15 May '15

Hi All, My name is Larry and I would like to request permission to help with the Fedora sysadmin group. I have been using Linux since the late 90's and have over 15 years of professional experience. I have worked with large web hosting providers as well as multinational oil and gas companies. I hold the following certifications, RHCSA, CCNA R/S, CCDA, CCNP Voice and have started to study for the RHCE examination. I am willing and happy to learn new things. I would like to work with the NOC and core administration or security administration teams. Is there anyone available who is willing to sponsor me? Thank you, Larry Brower, RHCSA, CCNA R/S, CCDA, CCNP Voice Communications Technician Aramco Services Company Aramco Research Center - Houston 16300 Park Row | Houston, Tx. 77084 Office: 713.432.4516 | Mobile: 832.570.5416 | Direct: 281.647.2987 larry.brower(a)aramcoservices.com

2 3

Dist Git for Copr
by Adam Samalik 15 May '15

15 May '15

Hello everyone, I would like to make a proposal and start a discussion about Dist Git for Copr. At the moment, Copr can build packages only from source rpms, which has been uploaded in advance somewhere in the internet - so if you want to build something, you need your own file server. As people often delete the sources from their server after the build process, the builds are no longer easily reproducible. And there is also no history of the sources. We would like to use the Dist Git project[1] to store the sources. This would make all builds reproducible, there would be no need of having a file server to build and we would have a history of changes to each package. At first, we would use the Dist Git server as a storage only. Users would not have direct access. Instead, they would either: a) upload their source rpms somewhere and use Copr as before (no change in work-flow) b) submit the source rpms directly into Copr In both cases, Copr would import ($ fedpkg import) the packages into the Dist Git and build them from there. The design of git repos would be: Repo name: 'user-name/copr-project/pkg-name' Branch name: 'epel-7', 'fedora-22' At that moment, we would need a discussion on how to open the Dist Git server to users. - What would be the best web frontend? - Could we use Pagure? (used to be progit) - Which tool would they use? - How would they authenticate? There has been some unfinished discussion on https://fedorahosted.org/fedora-infrastructure/ticket/4564 The main purpouse of this is having a storage for sources, reproducible builds and history of sources. Eventualy, we can extend it with bug tracking, easy forking and merging and the other "Git-Hub-like" stuff. Thanks! Adam Samalik [1] https://github.com/release-engineering/dist-git

6 8

Freeze Break Request - reduce number of auto-disabled mirrors
by Adrian Reber 14 May '15

14 May '15

The new MM2 crawler disables mirrors which have failed to be successfully crawled for 4 consecutive crawls. This seems to be a good idea to reduce the total number of crawls by removing mirrors which are just too slow. Unfortunately the current default timeout of 2 hours is not enough. Especially for mirrors which mirror more than one category as the timeout is per host and not per category. The problem is also not network bound but it seems to be related to two crawlers updating the directories of all mirrors on the same database at the same time. To workaround this timeout problem I am now starting the crawler on the second crawler 3 hours later and I have also increased the timeout from 2 hours to 3 hours. Additionally a small fix is included to also crawl the last mirror in the database which was ignored until now. After this is applied I would also re-enable the auto-disabled hosts in the database. Can I get two +1 for these changes? Additionally I think we can remove the second crawler and just use the times when the first crawler is idle to crawl other hosts. So instead of starting a crawl every 12 hours on two crawlers we could crawl half of the mirrors every 6 hours. But that is for after the freeze. Adrian commit 0dc0b70d9790b95cb6c1f41b4d36fb9aa2c9fbfc Author: Adrian Reber <adrian(a)lisas.de> Date: Wed May 13 11:23:21 2015 +0000 Start the crawl later on the second crawler. Even with rsync as crawl method some hosts are taking a very long time to be crawled. The network connection with rsync is only open for a short time, but with both crawlers reading and writing from the database it takes a very long time until the status of all directories is updated. Therefore this patch introduces a 3 hour delay of the crawl on the second crawler. This could also be solved with two different cron.d files; one for each crawler. diff --git a/roles/mirrormanager/crawler/files/crawler.cron b/roles/mirrormanager/crawler/files/crawler.cron index 3d695ca..c74b915 100644 --- a/roles/mirrormanager/crawler/files/crawler.cron +++ b/roles/mirrormanager/crawler/files/crawler.cron @@ -1,4 +1,8 @@ # run the crawler twice a day # logs sent to /var/log/mirrormanager/crawler.log and crawl/* by default # 32GB of RAM is not enough for 75 threads, 38 seems to work so far -0 */12 * * * mirrormanager /usr/bin/mm2_crawler --threads 38 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1 +# +# [ "`hostname -s`" == "mm-crawler02" ] && sleep 3h is used to start the crawl +# later on the second crawler to reduce the number of parallel accesses to +# the database +0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 3h; /usr/bin/mm2_crawler --threads 38 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1 commit 06309516b88ffade6f00c78833bc62aa002d7f56 Author: Adrian Reber <adrian(a)lisas.de> Date: Wed May 13 11:53:16 2015 +0000 Increase crawler timeout from 2h to 3h. Since MM2 is in production about 140 mirrors have been auto-disabled due to crawler timing out after 2 hours (default). Try if it works better with 3 hours. This in combination with the previous commit to decrease the load on the database should help to auto disable less good mirrors. Especially mirrors who mirroring almost everything can hardly be crawled within the 2 hour limit. Unfortunately the limit is per host and not category. diff --git a/roles/mirrormanager/crawler/files/crawler.cron b/roles/mirrormanager/crawler/files/crawler.cron index c74b915..66801d7 100644 --- a/roles/mirrormanager/crawler/files/crawler.cron +++ b/roles/mirrormanager/crawler/files/crawler.cron @@ -5,4 +5,4 @@ # [ "`hostname -s`" == "mm-crawler02" ] && sleep 3h is used to start the crawl # later on the second crawler to reduce the number of parallel accesses on # the database -0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 3h; /usr/bin/mm2_crawler --threads 38 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1 +0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 3h; /usr/bin/mm2_crawler --timeout-minutes 180 --threads 38 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1 commit 78c19a35d1706eb43e4e0c5fa202e0a6549674de Author: Adrian Reber <adrian(a)lisas.de> Date: Wed May 13 11:59:37 2015 +0000 Also crawl the last mirror in the database. The last mirror in the database was not crawled and this adds '1' to the --stopid if necessary. diff --git a/roles/mirrormanager/crawler/files/run_crawler.sh b/roles/mirrormanager/crawler/files/run_crawler.sh index b9d642e..3269dea 100644 --- a/roles/mirrormanager/crawler/files/run_crawler.sh +++ b/roles/mirrormanager/crawler/files/run_crawler.sh @@ -26,4 +26,7 @@ for i in `seq 1 ${NUMBER_OF_CRAWLERS}`; do fi let STARTID=${STARTID}+${PART} let STOPID=${STOPID}+${PART} + if [ "${STOPID}" -eq "${MAX_HOST}" ]; then + let STOPID=${STOPID}+1 + fi done

3 3

Plan for tomorrow's Fedora Infrastructure meeting (2015-05-14)
by Kevin Fenzi 13 May '15

13 May '15

The infrastructure team will be having it's weekly meeting tomorrow, 2015-05-07 at 18:00 UTC in #fedora-meeting on the freenode network. This week we are continuing to try something new. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of this morning is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. kevin -- = Introduction = This shared document is for the next fedora infrastructure meeting. We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2015-05-14) #meetingname infrastructure #topic aloha #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson #topic New folks introductions / Apprentice feedback = Status / information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info Classroom session on new fedora infrastructure private cloud from msuchy: http://meetbot.fedoraproject.org/fedora-classroom/2015-05-11/fedora-classro… #info pagure01 is now setup in production - kevin/pingou #info new cloud nodes are now monitored in nagios - kevin #info we are now in Fedora 22 Final freeze - kevin #info Wrote up initial migration page for cloud migration:· https://fedoraproject.org/wiki/Infrastructure_private_cloud_icehouse_migrat… - kevin #info transient cloud instance playbook reworked and working - kevin #info = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus or decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic What are we going to do about meetbot (whither supybot)? - threebean #topic = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the info in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #topic Learn about: = Meeting end stuff = #topic Open Floor #endmeeting

1 0

Freeze Break Request - Declare fedmsg cert for mm2 backend
by Ralph Bean 13 May '15

13 May '15

First one of the season! We're getting a traceback from fedmsg-logger in the mm2_get_internet2_netblocks script on mm-backend01. It is complaining that it can't find the cert it needs. Here's a patch to our ansible repo that will tell it to use the cert that's already on the system in that context: Can I get two +1s to apply this and push it out to mm-backend01? diff --git a/roles/fedmsg/base/templates/ssl.py.j2 b/roles/fedmsg/base/templates/ssl.py.j2 index aebc481..ab3e6c3 100644 --- a/roles/fedmsg/base/templates/ssl.py.j2 +++ b/roles/fedmsg/base/templates/ssl.py.j2 @@ -182,9 +182,10 @@ config = dict( ("mirrormanager2.mm-frontend01", "mirrormanager2-mm-frontend01.%s" % suffix), ("shell.mm-frontend01", "shell-mm-frontend01.%s" % suffix), - ("shell.mm-backend01", "shell-mm-backend01.%s" % suffix), ("shell.mm-crawler01", "shell-mm-crawler01.%s" % suffix), ("shell.mm-crawler02", "shell-mm-crawler02.%s" % suffix), + ("shell.mm-backend01", "shell-mm-backend01.%s" % suffix), + ("mirrormanager.mm-backend01", "shell-mm-backend01.%s" % suffix), ("shell.anitya-frontend01", "shell-anitya-frontend01.fedoraproject.org"), ("anitya.anitya-frontend01", "anitya-anitya-frontend01.fedoraproject.org"),

3 3

[release] pagure: 0.1.3
by Pierre-Yves Chibon 13 May '15

13 May '15

Good morning everyone, Yesterday when I was looking into setting up pagure on EL7 again, I found out that EL7 has gitolite3 but not gitolite 2 which pagure was relying on so far. So this morning, with the help of Patrick, pagure has been ported to gitolite3 and I just cut a new pagure release. Here is the changelog of the new version: * Wed May 13 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.1.3-1 - Update to 0.1.3 - Add support for gitolite3 - Fix unit-tests suite to work on jenkins Pierre

1 0

Update on Statscache and Fedora releng dash
by Ratnadeep Debnath 12 May '15

12 May '15

Hi Ralph, I have made quite some changes to statscache[1] and statscache_plugins[2] to empower a faster prototype of releng dash[3], currently working with only the AMIs plugin. I have come up with a layout spec for releng plugin[4] to generate the UI dynamically with minimum client side code, same code working for all widgets in the dashboard. How to test the changes: 1. cd into statscache_plugins/statscahe_plugins, and run 'python -m unittest tests.releng' to seed initial data for AMIs plugin. 2. cd into statscache/ dir and run the flask server: python statscache/app.py 3. cd into fedora-releng-dash repo dir, and checkout to branch 'statscache', and open index.html in browser. You should see the AMI widgets in the page. Also, I think, we need to find a better way to pass the 'frequency' param to instantiate models and plugins. Also, the data fetching part could be moved to the plugins to allow plugin level optimizations, if needed. Please let me know your thoughts on the above changes. [1]: https://github.com/fedora-infra/statscache/pull/4 [2]: https://github.com/fedora-infra/statscache_plugins/pull/1 [3]: https://github.com/fedora-infra/fedora-releng-dash/pull/20/ [4]: https://github.com/rtnpro/statscache_plugins/commit/f1ca6c85bf2a4cb22af93b5… Regards, rtnpro -- Ratnadeep Debnath, https://www.waartaa.com GPG Fingerprint: 033C 8041 A0E9 CDBA 2E02 B785 2119 5486 F245 DFD6

3 2

Fedora Cloud classroom
by Miroslav Suchý 11 May '15

11 May '15

As previously announced, I plan to held classroom lesson. Mark your calendar: 11th May 2005 #fedora-classroom on irc.freenode.net 15:00 UTC (that is 17:00 CET, 11:00 EDT) The lesson is intended primary for Fedora Infrastructure members (or apprentices). I will try to cover following topics: * what is Fedora Cloud (spoiler: our OpenStack instance) * which machines are used * where is the documentation * where are the playbooks and which files are important and where it is stored in ansible.git * which services are enabled and why * how they are installed * where are the weak parts, where you must be careful Mirek Suchy

9 14

[release] MirrorManager2: 0.1.0
by Pierre-Yves Chibon 08 May '15

08 May '15

Hi everyone, I just cut and push to stg a new MirrorManager2 release: 0.1.0 Here is the changelog: * Thu May 07 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.1.0-1 - Update 0.1.0 - Add the possibilities to delete a site or a host - Do not only create /var/lock/mirrormanager on installation (Adrian Reber) That should fix the question from Jeff. I'll push to prod once I tested it on stg :) Pierre

2 2

May status update for Fedora Infrastructure Apprentices
by Kevin Fenzi 08 May '15

08 May '15

Greetings. You are getting this email because you are in the 'fi-apprentice' group in the fedora account system (or are reading this on the infrastructure list). Feel free to reply just directly to me, or cc the infrastructure list for everyone to see and comment on. https://fedoraproject.org/wiki/Infrastructure_Apprentice At the first of every month(or so), I am going to be sending out an email like this one. I would like feedback on how things are going for you. I'd like to ask for everyone to send me a quick reply with the following data or anything related you can think of that might help us make the apprentice program more useful. 0. Whats your fedora account system login? 1. Have you logged in and used your fi-apprentice membership to look at our machines/setup in the last month? Do you plan to? 2. Has it helped you decide any area you wish to focus on or contribute to more? 3. Have you looked at or been able to work on any of the fi-apprentice 'easyfix' tickets? https://fedorahosted.org/fedora-infrastructure/report/14 4. Do you still wish to be a member of the group? If not (for whatever reason) could you provide any hints to help others down the road? 5. Is there any help or communication or ideas you have that would help you do any of the above? 6. What do you find to be the hardest part of getting involved? Finding things to work on? Getting attention from others to help you? Finding tickets in your interest area? 7. Have you been able to make any weekly irc meetings? Do you find them helpful or interesting? 8. Have you logged into our Gobby instance and read/seen/added to our meeting agenda? https://fedoraproject.org/wiki/Gobby Any other general feedback is also quite welcome, including improvements to this email, the wiki page, etc. Any folks I do not hear from in the next week will be removed from the group. (Note that it's easy to be readded when you have time or whatever and it's nothing at all personal, we just want to keep the group up to date with active folks). Thanks, and looking forward to your feedback! kevin

6 5

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure May 2015