[Release] pagure-dist-git: 0.11.1
by Pierre-Yves Chibon
Good Morning Everyone,
I just cut a release of pagure-dist-git: 0.11.1.
Here is its changelog:
* Wed Dec 20 2017 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.11.1-1
- Update to 0.11.1
- Ensure releng has always commit ACL
- Do not consider forks when generating the extras JSON files for POC and
bugzilla
Since both fix were already applied in prod, the package has been updated to
both stg and prod.
Happy packaging!
Pierre
6 years, 4 months
[Release] pagure: 3.13.1
by Pierre-Yves Chibon
Good Morning Everyone,
I just cut a new release of pagure: 3.13.1.
Here is its changelog:
* Tue Dec 19 2017 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 3.13.1-1
- Update to 3.13.1
- Add an alembic migration removing a constraint on the DB that not only no
longer needed but even blocking regular use now
The migration worked fine in stg.pagure.io and src.stg.fp.o and the symptoms I
was seeing no longer appear.
If all goes fine I'll likely update production tomorrow morning.
Happy hacking and packaging!
Pierre
6 years, 4 months
[Release] pagure: 3.13
by Pierre-Yves Chibon
Good Morning Everyone,
I just cut a new release of pagure: 3.13.
I meant to do a 3.12.1 but a couple of feature slipped in the release and made
it a 3.13 instead.
Here is the changelog:
* Mon Dec 18 2017 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 3.13-1
- Update to 3.13
- Fix the alembic migration adjusting the pull_requests table
- Fix how is created the db in the docker development environment (Clement
Verna)
- Ensure optional dependencies remain optional
- Ensure groups cannot be created when it is not allowed
- When listing issues, include the project as well in the user's issue API
endpoint
- Sort forks by date of creation (descending) (Neha Kandpal)
- Ensure the pagination arguments are returned when a page is specified
- Make the milestone clickable on the issue page
- Make the celery tasks update their status so we know when they are running (vs
pending)
It's running in staging (both stg.pagure.io and src.stg.fp.o) but while testing
it I found out that there is a database constraint that did not get lifted in
3.12 and prevents deleting a project from working as expected.
The corresponding fix is under review, so expect a 3.13.1 soon :)
Happy hacking!
Pierre
6 years, 4 months
MBS disk usage
by Ralph Bean
Way back in May some of us were in Raleigh for a Hackathon:
https://fedoraproject.org/wiki/CI_and_Infrastructure_Hackathon_2017
There, we talked about the MBS and wondered what kind of new storage
requirements it would impose on koji. We all worried about it, but we didn't
have numbers to figure out how worried we should be.
---
Here, below, find a query that Mike McLean shared for the koji database.
It very, very roughly shows rpm storage usage per-quarter in koji, with a
couple caveats.
- It only counts rpm sizes. It doesn't count logs, or signed copied, or isos,
or anything like that.
- It is also biased towards newer quarters. More recent quarters haven't had
time yet for garbage collection to kick in, so while they are larger in
magnitude, it needs to be taken in context, with salt, etc.
Here's a run::
koji=# select date_trunc('quarter', completion_time) as bin,
round(sum(size1)/1073741824, 3) as rpm_gb,
round(sum(size2)/1073741824,3) as archive_gb,
round(sum(size1+size2)/1073741824, 3) as total_gb from (select build.id,
build.completion_time, coalesce((select sum(size) from rpminfo where
build_id=build.id),0) as size1, coalesce((select sum(size) from
archiveinfo where build_id=build.id),0) as size2 from build where
build.state=1 and build.volume_id=0) as data group by
date_trunc('quarter', completion_time) order by bin;
bin | rpm_gb | archive_gb | total_gb
--------------------+-----------------------------
2007-04-01 00:00:00 | 254.416 | 0.000 | 254.416
2007-07-01 00:00:00 | 314.764 | 0.000 | 314.764
2007-10-01 00:00:00 | 317.851 | 0.000 | 317.851
2008-01-01 00:00:00 | 427.718 | 0.000 | 427.718
2008-04-01 00:00:00 | 393.187 | 0.000 | 393.187
2008-07-01 00:00:00 | 404.171 | 0.000 | 404.171
2008-10-01 00:00:00 | 392.340 | 0.000 | 392.340
2009-01-01 00:00:00 | 565.137 | 0.000 | 565.137
2009-04-01 00:00:00 | 349.749 | 0.000 | 349.749
2009-07-01 00:00:00 | 464.736 | 0.000 | 464.736
2009-10-01 00:00:00 | 324.401 | 0.000 | 324.401
2010-01-01 00:00:00 | 331.923 | 0.000 | 331.923
2010-04-01 00:00:00 | 304.823 | 0.000 | 304.823
2010-07-01 00:00:00 | 293.123 | 0.000 | 293.123
2010-10-01 00:00:00 | 262.374 | 0.000 | 262.374
2011-01-01 00:00:00 | 307.806 | 0.000 | 307.806
2011-04-01 00:00:00 | 243.162 | 0.000 | 243.162
2011-07-01 00:00:00 | 239.072 | 0.000 | 239.072
2011-10-01 00:00:00 | 287.522 | 0.000 | 287.522
2012-01-01 00:00:00 | 388.169 | 0.000 | 388.169
2012-04-01 00:00:00 | 347.497 | 0.000 | 347.497
2012-07-01 00:00:00 | 384.210 | 0.000 | 384.210
2012-10-01 00:00:00 | 369.828 | 0.000 | 369.828
2013-01-01 00:00:00 | 522.266 | 0.000 | 522.266
2013-04-01 00:00:00 | 451.709 | 0.000 | 451.709
2013-07-01 00:00:00 | 733.928 | 0.000 | 733.928
2013-10-01 00:00:00 | 551.316 | 0.000 | 551.316
2014-01-01 00:00:00 | 530.781 | 0.000 | 530.781
2014-04-01 00:00:00 | 747.820 | 0.000 | 747.820
2014-07-01 00:00:00 | 1008.080 | 0.000 | 1008.080
2014-10-01 00:00:00 | 665.198 | 0.000 | 665.198
2015-01-01 00:00:00 | 839.060 | 0.000 | 839.060
2015-04-01 00:00:00 | 931.210 | 0.000 | 931.210
2015-07-01 00:00:00 | 887.543 | 0.000 | 887.543
2015-10-01 00:00:00 | 700.465 | 0.000 | 700.465
2016-01-01 00:00:00 | 1189.887 | 152.489 | 1342.376
2016-04-01 00:00:00 | 983.871 | 43.799 | 1027.670
2016-07-01 00:00:00 | 983.195 | 99.737 | 1082.932
2016-10-01 00:00:00 | 1165.745 | 27.005 | 1192.749
2017-01-01 00:00:00 | 1681.824 | 149.185 | 1831.009
2017-04-01 00:00:00 | 1861.706 | 8.289 | 1869.995
2017-07-01 00:00:00 | 2610.738 | 132.464 | 2743.203
2017-10-01 00:00:00 | 2236.426 | 4195.877 | 6432.303
(43 rows)
Interesting!
Now, let's see the same query, but with the MBS builds *excluded* (the MBS user
is userid 3819)::
koji=# select date_trunc('quarter', completion_time) as bin,
round(sum(size1)/1073741824, 3) as non_mbs_rpm_gb,
round(sum(size2)/1073741824,3) as non_mbs_archive_gb,
round(sum(size1+size2)/1073741824, 3) as non_mbs_total_gb from (select build.id,
build.completion_time, coalesce((select sum(size) from rpminfo where
build_id=build.id),0) as size1, coalesce((select sum(size) from
archiveinfo where build_id=build.id),0) as size2 from build where
build.state=1 and build.volume_id=0 and build.owner!=3819) as data group by
date_trunc('quarter', completion_time) order by bin;
bin | non_mbs_rpm_gb | non_mbs_archive_gb | non_mbs_total_gb
----------------------------------------------------+-----------------
2007-04-01 00:00:00 | 254.416 | 0.000 | 254.416
2007-07-01 00:00:00 | 314.764 | 0.000 | 314.764
2007-10-01 00:00:00 | 317.851 | 0.000 | 317.851
2008-01-01 00:00:00 | 427.718 | 0.000 | 427.718
2008-04-01 00:00:00 | 393.187 | 0.000 | 393.187
2008-07-01 00:00:00 | 404.171 | 0.000 | 404.171
2008-10-01 00:00:00 | 392.340 | 0.000 | 392.340
2009-01-01 00:00:00 | 565.137 | 0.000 | 565.137
2009-04-01 00:00:00 | 349.749 | 0.000 | 349.749
2009-07-01 00:00:00 | 464.736 | 0.000 | 464.736
2009-10-01 00:00:00 | 324.401 | 0.000 | 324.401
2010-01-01 00:00:00 | 331.923 | 0.000 | 331.923
2010-04-01 00:00:00 | 304.823 | 0.000 | 304.823
2010-07-01 00:00:00 | 293.123 | 0.000 | 293.123
2010-10-01 00:00:00 | 262.374 | 0.000 | 262.374
2011-01-01 00:00:00 | 307.806 | 0.000 | 307.806
2011-04-01 00:00:00 | 243.162 | 0.000 | 243.162
2011-07-01 00:00:00 | 239.072 | 0.000 | 239.072
2011-10-01 00:00:00 | 287.522 | 0.000 | 287.522
2012-01-01 00:00:00 | 388.169 | 0.000 | 388.169
2012-04-01 00:00:00 | 347.497 | 0.000 | 347.497
2012-07-01 00:00:00 | 384.210 | 0.000 | 384.210
2012-10-01 00:00:00 | 369.828 | 0.000 | 369.828
2013-01-01 00:00:00 | 522.266 | 0.000 | 522.266
2013-04-01 00:00:00 | 451.709 | 0.000 | 451.709
2013-07-01 00:00:00 | 733.928 | 0.000 | 733.928
2013-10-01 00:00:00 | 551.316 | 0.000 | 551.316
2014-01-01 00:00:00 | 530.781 | 0.000 | 530.781
2014-04-01 00:00:00 | 747.820 | 0.000 | 747.820
2014-07-01 00:00:00 | 1008.080 | 0.000 | 1008.080
2014-10-01 00:00:00 | 665.198 | 0.000 | 665.198
2015-01-01 00:00:00 | 839.060 | 0.000 | 839.060
2015-04-01 00:00:00 | 931.210 | 0.000 | 931.210
2015-07-01 00:00:00 | 887.543 | 0.000 | 887.543
2015-10-01 00:00:00 | 700.465 | 0.000 | 700.465
2016-01-01 00:00:00 | 1189.887 | 152.489 | 1342.376
2016-04-01 00:00:00 | 983.871 | 43.799 | 1027.670
2016-07-01 00:00:00 | 983.195 | 99.737 | 1082.932
2016-10-01 00:00:00 | 1165.745 | 27.005 | 1192.749
2017-01-01 00:00:00 | 1657.196 | 149.185 | 1806.381
2017-04-01 00:00:00 | 1565.237 | 8.289 | 1573.526
2017-07-01 00:00:00 | 2417.401 | 132.464 | 2549.865
2017-10-01 00:00:00 | 1932.821 | 4195.869 | 6128.691
(43 rows)
Finally, let's see the storage usage of only MBS-managed builds::
koji=# select date_trunc('quarter', completion_time) as bin,
round(sum(size1)/1073741824, 3) as mbs_rpm_gb,
round(sum(size2)/1073741824,3) as mbs_archive_gb,
round(sum(size1+size2)/1073741824, 3) as mbs_total_gb from (select build.id,
build.completion_time, coalesce((select sum(size) from rpminfo where
build_id=build.id),0) as size1, coalesce((select sum(size) from
archiveinfo where build_id=build.id),0) as size2 from build where
build.state=1 and build.volume_id=0 and build.owner=3819) as data group by
date_trunc('quarter', completion_time) order by bin;
bin | mbs_rpm_gb | mbs_archive_gb | mbs_total_gb
--------------------------------------------+-------------
2017-01-01 00:00:00 | 24.627 | 0.000 | 24.627
2017-04-01 00:00:00 | 296.469 | 0.000 | 296.469
2017-07-01 00:00:00 | 193.337 | 0.000 | 193.337
2017-10-01 00:00:00 | 303.605 | 0.008 | 303.612
(4 rows)
6 years, 4 months
December status update for Fedora Infrastructure Apprentices
by Kevin Fenzi
You are getting this email because you are in the 'fi-apprentice' group
in the fedora account system (or are reading this on the infrastructure
list).
I am doing something different this time with this email:
You should reply (to me or the list) if you have any suggestions or
ideas on how to improve the apprentice program in the coming year.
I'm not going to remove anyone for inactivity in December.
I plan on thinking on our process over the holidays and sending out a
new email on Jan 2nd about any proposed changes in process, etc.
Happy Holidays to everyone.
kevin
6 years, 4 months
Plan for Thursdays Fedora Infrastructure Meeting (2017-12-14)
by Stephen John Smoogen
The infrastructure team will be having it's weekly meeting tomorrow,
2017-12-14 at 18:00 UTC in #fedora-meeting on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
The document as of 2017-12-14 22:20 read as:
This shared document is for the next fedora infrastructure meeting.
= Introduction =
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2017-12-14)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik dgilmore threebean pingou puiterwijk
pbrobinson maxamillion
= Let new people say hello =
#topic New folks introductions
#info This is a place where people who are interested in Fedora
Infrastructure can introduce themselves
= Status / Information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need
to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info PHX2 Colo Trip, Dec 4th - 9th. Trip was success. All Fedora
servers were moved
#info Fedora Infrastructure SemiFreeze begins 2017-12-21
#info Many people on vacation.
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus /suor decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic Ticket cleanup
#info none this week.
#topic PHX2 Move
#info Lessons Learned(LL): Having a good plan does work
#info LL: Make sure that the people who need to be there are there
#info LL: New cages have 30 amp power.
#info LL: You can put 1 Dell FX per 2 30 amp power supplies
#info LL: Inventory is hard
#topic Fedora Infrastructure SemiFreeze
#info Red Hat does a year end shutdown every year.
#info Most of infrastructure is on vacation during this time
#info People will be on IRC every now and then. Do no expect replies to queries
#info All changes to infrastructure from 2017-12-21 -> 2018-01-02 need
+1 via IRC or email
#info Use email or infrastructure tickets so things are captured.
= Apprentice office hours =
#topic Apprentice Open office hours
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for
improvement,
etc. Whoever would like to do this, just add the i/nfo in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#topic Learn about:
#info none this week
= Meeting end stuff =
#topic Open Floor
#endmeeting
--
Stephen J Smoogen.
6 years, 4 months
OpenStack endpoints in fed-cloud09
by Miroslav Suchy
Hi,
I am curious about current state of fed-cloud09.
I wanted to use ansible module os_server to spin up new VM in Fedora
Cloud, but I got error that ansible cannot connect to:
http://fedorainfracloud.org:8696/v2.0/networks
That was strange as 8696 should be an internal port for neutron. HAProxy
should listen on default port 9696, handle the ssl stuff, and then
forward it to localhost:8696 which should not be propagated to outside
networks.
(And indeed curl of https://fedorainfracloud.org:9696/ works).
When I query endpoints I get:
# openstack catalog list
....
| neutron | network | RegionOne
|
| | | publicURL:
http://fedorainfracloud.org:8696/
|
| | | internalURL:
http://fedorainfracloud.org:8696/ |
| | | adminURL: http://fedorainfracloud.org:8696/
This is strange, because in our
playbooks/hosts/fed-cloud09.cloud.fedoraproject.org.yml playbook we have:
- shell: source /root/keystonerc_admin && keystone service-list | grep
'neutron' | awk '{print $2}'
check_mode: no
changed_when: false
register: SERVICE_ID
- shell: source /root/keystonerc_admin && keystone endpoint-list |
grep {{SERVICE_ID.stdout}} | awk '{print $2}'
check_mode: no
changed_when: false
register: ENDPOINT_ID
- shell: source /root/keystonerc_admin && keystone endpoint-list |grep
{{SERVICE_ID.stdout}} |grep -v {{ controller_publicname }} && (keystone
endpoint-delete {{ENDPOINT_ID.stdout}} && keystone endpoint-create
--region 'RegionOne' --service {{SERVICE_ID.stdout}} --publicurl
'https://{{ controller_publicname }}:9696/' --adminurl 'https://{{
controller_publicname }}:9696/' --internalurl 'https://{{
controller_publicname }}:9696/' ) || true
Which should set publicURL to 9696. It seems that this is set for some
time (but was not for sure in past). I before I run the fed-cloud09
playbook (which itself can break a lot of stuff if some stuff was done
manually and not put in playbook) and change the public port to 9696
back (and generaly to ssl version for all services), whether there is
some reason I should not touch it?
Mirek
6 years, 4 months
Infrastructure move -- thanks
by Paul W. Frields
I might be a bit biased from my perspective as the guy who manages the
folks on the Infrastructure team. But I wanted to take a moment to
say thank you to all the team members and everyone involved for
conducting the recent colo move, a complex and critical process, in an
outstanding manner. There were some difficulties along the way and
the team dealt with them in their usual professional way. Kudos to
all, and well done!
--
Paul W. Frields http://paul.frields.org/
gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717
http://redhat.com/ - - - - http://pfrields.fedorapeople.org/
The open source story continues to grow: http://opensource.com
6 years, 4 months
Holiday Reminder
by Kevin Fenzi
Greetings everyone.
Just a reminder that holidays are coming up and that means that many
people won't be around as much.
* If whatever it is can wait, just wait and bring it up next year after
people come back.
* If you send email don't expect quick answers.
* If it's urgent and no one is around, please file a ticket and someone
will get to it as soon as we can.
Do try not to make any infrastructure changes that might cause alerts or
down machines.
Red Hat has shutdown period starting Dec 22nd and going until Jan 1st.
So, expect all those folks to be back around jan 2nd (tuesday).
I personally am off starting monday the 11th. I should be around working
on personal projects and the like, but will try not to be working.
Hope everyone has a very happy holiday season and gets in some Open
Source contributing too! :)
kevin
6 years, 4 months
[Release] pagure: 3.12
by Pierre-Yves Chibon
Good Morning Everyone,
I just cut a new release of pagure: 3.12
Here is its changelog:
* Fri Dec 08 2017 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 3.12-1
- Update to 3.12
- Adjust the API endpoint listing project to not return a 404 when not projects
are found (Vivek Anand)
- Remove --autoreload from the docker dev deployment (Vivek Anand)
- Fix ordering issues (Patrick Uiterwijk)
- Do not log actions pertaining to private issues, PRs or projects
- Fix flagging a PR when no uid is specified
- Fix the doc about custom gitolite config
- Fix displaying the filename on the side and linking to file for remote PRs
- Add irc info in Readme (Vivek Anand)
- Make pagure compatible with newer python chardet
- Check that the identifier isn't the hash of a git tree in view_file
- Fix if the identifier provided is one of a blob instead of a commit in
view_commit
- Include the status when flagging a PR via jenkins
- Enable OpenID Connect authentication (Slavek Kabrda)
- Use the updated timestamp in the pull-request list
- Add migration to fix the project_from_id foreign key in pull_requests
- Let the SSE server to send the notifications so they can be displayed live
- Improve the createdb script to support stamping the database in the initial
run
- Specify a different connection and read timeout in pagure-ci
- Small CSS fix making the (un)subscribe show up on the PR page
It's running in staging but required two hotfixes, one of which has already been
reviewed and merged in git, the other to come shortly.
So I'll do a 3.12.1 before pushing things to prod :)
Happy hacking!
Pierre
6 years, 4 months
