About JS framework
by Pierre-Yves Chibon
Good Morning Everyone,
Our infrastructure is mostly a python store, meaning almost all our apps are
written in python and most using wsgi.
However in python we are using a number of framework:
* flask for most
* pyramid for some of the biggest (bodhi, FAS3)
* Django (askbot, Hyperkitty)
* TurboGears2 (fedora-packages)
* aiohttp (python3, async app: mdapi)
While this makes sometime things difficult, these are fairly standard framework
and most of our developers are able to help on all.
However, as I see us starting to look at JS for some of our apps (fedora-hubs,
wartaa...), I wonder if we could start the discussion early about the different
framework and eventually see if we can unify around one.
This would also allow those of us not familiar with any JS framework to look at
the recommended one instead of picking one up semi-randomly.
So has anyone experience with one or more JS framework? Do you have one that
would you recommend? Why?
Thanks for your inputs,
Pierre
1 year, 6 months
The future of pkgdb
by Pierre-Yves Chibon
(adding the releng list on CC, please keep the reply on the infra list)
Hi Everyone,
Following up on the thread about pagure on the top of dist-git started a
few days ago Ralph Bean, Matthew Prahl and I had a quick meeting just a few
minutes ago about the future of pkgdb.
Most of the content of the email is based on the notes we took during the
meeting, so I hope it is understandable for everyone. I have added some
context compared to the original notes, but feel free to ask questions if
there are any missing pieces or things that are unclear.
The idea is that with pagure becoming a front-end to dist-git, there is less
needs for pkgdb. So we tried to list all what pkgdb does and propose
solutions for each of these.
Let's start with the start: what does pkgdb do?
- Store point of contact for a package default assignee on bugzilla
- Store commit access for a package commit ACL
- Store admin access for a package approveacls ACL
- Store CC list for bug reports on the package watchbugzilla ACL
- Store CC list for changes made to the package watchcommits ACL
- Store new package requests
- Store new EPEL branch requests
- Allow creating new Fedora branch on demand
- Store which packages are orphaned
- Store which packages are retired
- Store collection/branches status
- Store anitya integration flag
- Store koshei integration flag
- Store critpath status
The main idea we have is to rely on pagure and PDC for as much as possible
and having a special project on pagure to use as a ticket/queuing system
for requests. This project could then be interacted with using a CLI tool.
Now going through each of the requirements listed above
- Store point of contact for a package (default assignee on bugzilla)
- we could use the first committer, alphabetically
- we could use the 'owner', but we need pagure to be able to "give" a
repo which it currently cannot.
- in order to "orphan" a package, we need this.
- we could list the default assignee in the yaml file in dist-git
- Not ideal since less "self-service"
- Store commit access for a package (commit ACL)
- Use the list of people with commit or admin access to the package
- Store admin access for a package (approveacls ACL)
- Handled in pagure directly by the people having admin access to the
package
- Need a way to prevent groups from having admin access (like we
prevented groups from having approveacls in pkgdb)
- Store CC list for changes made to the package (watchcommits ACL)
- use the pagure "watch" feature.
- Store CC list for bug reports on the package (watchbugzilla ACL)
- grow the ability in pagure to watch issues versus watch commits.
- in the meantime, just use watch commits for both.
- Factory 2.0 will handle this
- Store new package requests
- matt prahl is already cooking up a way to do this using a
https://pagure.io/repo-requests/issues/ queue and some scripts.
- https://pagure.io/fedrepo_req/pull-request/1
- !!! we have problems using pagure ticket queue here (api tokens, need
commit or really admin access...)
- other options:
- bugzilla
- custom made queue
- fpaste!
- patch pagure to do what we need.
-> Add the possibility to select a project in
https://pagure.io/settings/token/new and allow there the
issue_create, issue_update and issue_comment ACLs
-> Add the possibility to set the duration of the token (with
an upper limit: 365 days?) (per token with a default in the
config file?)
-> pingou will handle this
- Store new EPEL branch requests
- same as above.
- for the first seven days, don't show these in pkgdb-admin (or
equivalent) unless there are one or more comments from another package
maintainer.
- Allow creating new Fedora branch on demand
- except we want this to be for creating *any* new branch request "like
branch 2.0.x"
- make a little itty bitty approving service (IBAS) for auto-approving
these in a small way.
- it should re-use our replacement for pkgdb-admin under the hood so
we don't duplicate code.
- if some hard-coded set of checks pass then, auto-approve.
- checks like "is the EOL under 2 years?" "does the branch name
not include any non-ascii characters?" "run it through a swear
words dictionary"
- otherwise, leave in the queue for manual processing.
- Store which packages are orphaned
- these are just owned by the orphan user in pagure
- step 1, the orphan user is the only committer.
- step 2, when pagure grows the ability to "give" a project, then make
the orphan user the real owner.
- Factory 2.0 may handle this. Let pingou know
- Store which packages are retired
- in consultation with dgilmore, we don't think we need this anymore.
- the dead.package file is sufficient. we still need to propagate
blockage to koji (so listen on fedmsg for changes to that file instead
of listening for pkgdb message about status change).
- Also store this into PDC. it will be represented by the "active"
True/False flag on each branch.
- Store collection/branches status
- this is going to PDC as well (for modularity work, cf the video below)
- https://fedorapeople.org/groups/factory2/sprint-014/mprahl-pdc-sla-apis.mp4
- Store anitya integration flag
- store this in a yaml/toml file in the dist-git repo
- let the consumers
- do an http request to retrieve the file
- listen to fedmsg to catch changes to this file (and update a local
cache based on this)
- Store koshei integration flag
- store this in a yaml/toml file in the dist-git repo
- let the consumers
- do an http request to retrieve the file
- listen to fedmsg to catch changes to this file (and update a local
cache based on this)
- Store critpath status
- Store this in PDC and make it per branch
So with the proposed solution above, everything would be stored in:
- the git repo itself
- the pagure project of the package
- a pagure project for requests
- PDC
Now pkgdb interacts with a few known applications that will need to be adjusted
for this:
- the-new-hotness
- Cf the point about anitya above
- koshei
- Cf the point about koshei above
- zodbot
- will have to query the pagure API to find who has access
- bodhi
- will have to query the pagure API to find who has access (also, maybe
critpath flag?)
- FMN.rules (who is maintaining the package)
- will have to query the pagure API to find who has access
- pkgdb-admin
- mprahl is rewriting this to use a ticket queue, somewhere.
- pkgdb-cli (here we list all the different command and their description and
the possible solution)
acl Display ACLs for a given package
can be done, with no api keys
give Give package(s) according to the specified criteria
cannot be done yet. just return a "cannot be done yet" error.
list List package according to the specified criteria
can be done, with no api keys
orphan Orphan package(s) according to the specified criteria
cannot be done until "give" is done. this is just a "give" to the
orphan user.
unorphan Unorphan package(s) according to the specified criteria
cannot be done until "give" is done. this is just a "give" to another
user.
request Request ACLs on package(s) according to the specified criteria
to be bikedshed on devel about killing it. threebean says "just use
email!"
update Update ACLs on package(s) as desired
No longer needed since we don't do per branch ACLs anymore
branches List the active branches
Query PDC
pending List the pending ACLs requests
can easily be done by querying the pagure project for your tickets
with a status=Open (for package/branch requests)
for ACLs request, since we would not be tracking them down anymore,
it would have to be done by the packagers themselves and after a
while likely end up with a non-responsive maintainer procedure.
monitoring Get or Set the monitoring of a package
can easily get with no api key. (just http call to dist-git).
setting it no longer allowed.
koschei Get or Set the koschei monitoring of a package
can easily get with no api key. (just http call to dist-git).
setting it no longer allowed.
- bunch of links (long term)
I hope these notes are sufficiently clear, if not, we're happy to take any
and all questions.
I think we covered all bases and this is looking pretty straight forward.
What do you think?
Thanks for your inputs,
Pierre
6 years, 12 months
Apprentice work day (tenative 2017-05-24)
by Kevin Fenzi
Greetings.
I thought it might be a nice idea to run another apprentice work day.
What is an apprentice work day? Well, it's where everyone tries to be
around as much as they can on IRC and work on a specific goal with
apprentices.
I was thinking May 24th might be a good day.
It's the middle of the week (wed), and it's when we are in F26 Beta
freeze, so hopefully everyone is a bit less busy than normal.
For topics I'd propose one of the following (please let me know which
one seems best to you or add your own):
* Unloved apps day - Look at some of our apps we don't work on or do
many releases for anymore and fix bugs, perhaps learn how to do a
release, etc. askbot and packages are two apps that leap to mind for this.
* Triage day - Everyone goes through our issues on apps/infra/etc and
adds labels, asks for more info, fixes dupes, etc.
* PR day - like the above issues triage, but PRs.
* Lint day - run ansible-lint and clean up things in our ansible repo.
Perhaps decide on a standard syntax style and convert things to all use
that. Remove uses of things like lineinfile, etc.
Thoughts? Is the 24th good for everyone?
Does one of these sound better than the others?
Do you have some cooler idea?
kevin
6 years, 12 months
Plan for today's Fedora Infrastructure meeting 2017-04-27 18:00 UTC
by Stephen John Smoogen
The infrastructure team will be having it's weekly meeting tomorrow,
2017-04-27 at 18:00 UTC in #fedora-meeting on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
This shared document is for the next fedora infrastructure meeting.
= Introduction =
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2017-04-27)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik abadger1999 dgilmore threebean pingou
puiterwijk pbrobinson
= Let new people say hello =
#topic New folks introductions
= Status / information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need
to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info beta freeze will start 2017-05-17
#info infra hackfest in RDU 2017-05-08 to 2017-05-12 - everyone
#info Fedora Infrastructure weathered large outage last Friday. Good
job everyone
#info mass update/reboot cycle next week (2017-05-02/03) - everyone
#info bodhi 2.6.0 released. A few issues so look for 2.6.1 soon - bowlofeggs
#info
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus /suor decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic jenkins cleanup
(https://jenkins.fedorainfracloud.org/view/infrastructure/) - cverna
#topic Apprentice work day scheduling/topic - kevin
#topic BDR (bidirectional Data Replication) in postgres - kevin
= Apprentice office hours =
#topic Apprentice Open office hours
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
= Learn about some application or gsetup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for
improvement,
etc. Whoever would like to do this, just add the i/nfo in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#topic Learn about:
= Meeting end stuff =
#topic Open Floor
#endmeeting
--
Stephen J Smoogen.
7 years
2017-03-21 PHX2 Outage thoughts
by Kevin Fenzi
Greetings.
As everyong likely knows we had a long outage yesterday of our primary
datacenter (PHX2). It started around 14:45 UTC and lasted until around
06:45UTC saturday morning.
The outage was networking/routing related, but we don't know any further
details. Happy to share what we know and can share when investigations
are completed.
The purpose of this email however is to talk about what we might be able
to easily adjust to handle things like this moving forward. Happily
these sorts of outages are very rare, but there's a few things I think
we can do to make them a bit easier on us.
First, lets talk about things that went well:
* The mirrorlist containers worked great, so mirrorlist impact was low
(but not nothing, see below).
* Our distributed proxy system meant things like getfedora.org and other
static websites were fine.
* Our mirror network + metalinks/mirrorlists meant that people could
still get updates, etc just fine.
* status.fedoraproject.org was just great and even survived slashdotting
(well, thats not all it used to be :)
And some things could have been better:
* We couldn't take proxy01/10 (The two proxies at phx2) out of dns
because we didn't have the dnssec keys available anywhere to write new
data out. This meant that sometimes people would hit those in dns and
get a timeout and have to retry.
* We couldn't reach our offside backups at RDU2. Currently we access
them via a VPN that is from PHX2. We could of course have bugged someone
to get us access, but they were all busy with the outage.
* geoip was down, so might have affected installs. We may be able to
move this to a container.
* sudo anywhere depends on fas being up, but it was not, so the only way
we could get root on vm's was to go to console and login directly as root.
We also did a bit of brainstorming on IRC for what we could do if an
outage like this happened again and was longer. Without a bunch of new
hardware somewhere the buildsystem side of things wouldn't be very
practical to bring up anywhere else, but we could bring some non build
related services back up in other datacenters. Or at least improve the
errors they give.
So, IMHO, some action items we might consider:
* Create a batcave02.rdu that has all our repos, ansible keys and dns
keys. We can then use this to push things out if needed. It shouldn't be
too hard to pull from batcave01 or otherwise keep things in sync.
* Consider having a tested set of configs for a long outage like this:
haproxy/apache could be adjusted to show a outage page for all the down
services and we could bring things up more gradually.
* Consider a ipa03.rdu freeipa replicant. This would at least allow
kerberos auth (but of course with most things that use it down it
doesn't matter too much).
* Accelerate plans to move away from pam_url for sudo to something that
will work when fas is down.
kevin
7 years
Pagure on dist-git, the state of things
by Pierre-Yves Chibon
Good Morning Everyone,
I figured it has been a while since I reported progress on making pagure a
front-end for dist-git.
So here is a small status update.
What works:
- Well currently pagure is working as a front-end for dist-git in stg:
https://src.stg.fedoraproject.org/pagure/
- Hosting repos, browsing them, creating PR works all fine (w/ one , see
below)
- Features we want to exclude have been turned off (no user management on the
pagure side)
What does not work:
- Syncing the ACLs from pkgdb to gitolite takes ~3 minutes in prod, in stg, the
updated script in stg (which sync the ACLs from pkgdb to gitolite and pagure)
runs in ~30/35 minutes (if it doesn't crash, which my last run just did)
This is a blocker since it means it takes 30 to 35 minutes for someone to get
access to their git repo or (worse) their fork
- This pagure instance has the same issue as the main one, including a heisenbug
we're trying to track but have had no luck reproducing so far :(
What needs to be done:
- Fix the sync script
- Make it *way* faster than it is
- Make it creates the project on pagure using the releng user rather than
relying on the first contributor it finds in the list of maintainers
- Make fedmsg-genacls be triggered on pagure's fork fedmsg message so that we
re-generate the gitolite configuration file when someone forks a project
(and thus give them access to their fork)
- Once above is done: call for more testers
In the future:
- I think we will want to deprecate pkgdb entirely, so while the work above is
important, ideally it won't be there for too long.
- With pkgdb out of the loop, we'll need to figure some things out:
- Where/How to store the contact info for bugzilla
- Not sure relying on pagure's ACLs there is the way to go since we would
loose a level of granularity in the ACLs that I know people like and ask
for (having commit w/o being on the CC list in bugzilla or being on the CC
list w/o being a packager)
- How/when to require people be part of the packager group in FAS?
- Since one of the idea of pagure is to make it easier for "drive-by"
contribution to spec files, requiring to be a packager should only be
there for maintainers, but pagure doesn't have this level of
information/requirement, so we would need to find something or some place
to add this requirement or see if that requirement still stands
This is all I can think of for now, I'll update this thread if I come up with
more ideas/challenges.
Have a nice day,
Pierre
7 years
[release] pkgdb2: 2.6.1 and 2.6.2
by Pierre-Yves Chibon
Good Morning Everyone,
Yesterday I prepared and tagged a 2.6.1 release but I messed up with the git
history, so I ended up creating a new release 2.6.2 today.
Here are their corresponding changelogs:
* Tue Apr 25 2017 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.6.2-1
- Update to 2.6.2
- Fix git history in 2.6.2
* Mon Apr 24 2017 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.6.1-1
- Update to 2.6.1
- Make the mapping in pkgdb-sync-bugzilla script easier to adjust (by placing it
in the config file)
So 2.6.2 is now running in stg and prod and we should be free of any hotfix :)
Happy packaging,
Pierre
7 years
Weekly Koji Infra Tag Report
by Nobody
This is a list of packages in the various infrastructure koji tags
Please check and make sure there are not any that can be removed/dropped
epel6-infra
(no matching packages)
epel7-infra
Package Tag Extra Arches Owner
----------------------- ----------------------- ---------------- ---------------
freeipa-ktutils epel7-infra puiterwijk
compose-utils epel7-infra ausil
fedmsg-beaker-repoupdate epel7-infra tflink
anitya epel7-infra jcline
the-new-hotness epel7-infra jcline
fedocal epel7-infra pingou
python-IPy epel7-infra kevin
python-robosignatory epel7-infra puiterwijk
pdc-updater epel7-infra ralph
python-pdc epel7-infra ralph
glusterfs epel7-infra kevin
mirrormanager2 epel7-infra puiterwijk
blockerbugs epel7-infra tflink
python-django-jsonfield epel7-infra ralph
f23-infra
Package Tag Extra Arches Owner
----------------------- ----------------------- ---------------- ---------------
libphutil, f23-infra tflink
arcanist, f23-infra tflink
phabricator f23-infra tflink
phabricator-extension-ipsilonauth f23-infra tflink
libphutil f23-infra tflink
arcanist f23-infra tflink
f24-infra
Package Tag Extra Arches Owner
----------------------- ----------------------- ---------------- ---------------
mediawiki-openid f24-infra kevin
phabricator-extension-oauth f24-infra tflink
python-twill f24-infra codeblock
stickynotes2modernpaste f24-infra codeblock
python-flask-testing f24-infra codeblock
modern-paste f24-infra codeblock
mediawiki-skin-fedora f24-infra puiterwijk
mediawiki-FedoraBadges f24-infra kevin
basset f24-infra puiterwijk
phabricator f24-infra tflink
mediawiki-Lockdown f24-infra kevin
libphutil f24-infra tflink
arcanist f24-infra tflink
mediawiki-RSS f24-infra kevin
mirrormanager2 f24-infra puiterwijk
f25-infra
Package Tag Extra Arches Owner
----------------------- ----------------------- ---------------- ---------------
python-flask-testing f25-infra codeblock
modern-paste f25-infra codeblock
python-coveralls f25-infra codeblock
mdapi f25-infra pingou
basset f25-infra puiterwijk
mediawiki-FedoraBadges f25-infra kevin
mediawiki-Lockdown f25-infra kevin
mediawiki-RSS f25-infra kevin
mediawiki-openid f25-infra kevin
plus-plus-service f25-infra pingou
python-pdc f25-infra ralph
python-django-cors-headers f25-infra ralph
python-django-rest-framework-composed-permissions f25-infra ralph
patternfly1 f25-infra ralph
fas f25-infra kevin
libphutil, f25-infra tflink
arcanist, f25-infra tflink
phabricator f25-infra tflink
phabricator-extension-ipsilonauth f25-infra tflink
libphutil f25-infra tflink
arcanist f25-infra tflink
python-twill f25-infra codeblock
stickynotes2modernpaste f25-infra codeblock
mediawiki-skin-fedora f25-infra kevin
f26-infra
(no matching packages)
f27-infra
(no matching packages)
7 years
