About JS framework
by Pierre-Yves Chibon
Good Morning Everyone,
Our infrastructure is mostly a python store, meaning almost all our apps are
written in python and most using wsgi.
However in python we are using a number of framework:
* flask for most
* pyramid for some of the biggest (bodhi, FAS3)
* Django (askbot, Hyperkitty)
* TurboGears2 (fedora-packages)
* aiohttp (python3, async app: mdapi)
While this makes sometime things difficult, these are fairly standard framework
and most of our developers are able to help on all.
However, as I see us starting to look at JS for some of our apps (fedora-hubs,
wartaa...), I wonder if we could start the discussion early about the different
framework and eventually see if we can unify around one.
This would also allow those of us not familiar with any JS framework to look at
the recommended one instead of picking one up semi-randomly.
So has anyone experience with one or more JS framework? Do you have one that
would you recommend? Why?
Thanks for your inputs,
Pierre
1 year, 6 months
Proposed zchunk file format - V2
by Jonathan Dieter
Neal, thanks for the feedback. After taking your comments into
consideration, here's version 2.
+-+-+-+-+-+------------------+-+-+-+-+-+-+-+-+
| ID | Compression type | Index size |
+-+-+-+-+-+------------------+-+-+-+-+-+-+-+-+
+==================+=================+
| Compressed Index | Compressed Dict |
+==================+=================+
+===========+===========+
| Chunk | Chunk | ==> More chunks
+===========+===========+
ID
'\0ZCK1', identifies file as zchunk version 1 file
Compression type
Type of compression used to compress dict and chunks
Current values:
0 - Uncompressed
2 - zstd
Index size
This is a 64-bit unsigned integer containing the size of compressed
index.
Compressed Index
This is the index, which is described in the next section. The index
is compressed without a custom dictionary.
Compressed Dict (optional)
This is a custom dictionary used when compressing each chunk.
Because each chunk is compressed completely separately from the
others, the custom dictionary gives us much better overall
compression. The custom dictionary is compressed without a custom
dictionary (for obvious reasons).
Chunk
This is a chunk of data, compressed with the custom dictionary
provided above.
The index:
+---------------+======================+
| Checksum type | Checksum of all data |
+---------------+======================+
+================+-+-+-+-+-+-+-+-+
| Dict checksum | End of dict |
+================+-+-+-+-+-+-+-+-+
+================+-+-+-+-+-+-+-+-+
| Chunk checksum | End of chunk | ==> More
+================+-+-+-+-+-+-+-+-+
Checksum type
This is the type of checksum used to generate the checksums in the
index.
Current values:
0 = SHA-256
Checksum of all data
This is the checksum of the compressed dict and all the compressed
chunks, used to verify that the file is actually the same, even in
the unlikely event of a hash collision for one of the chunks
Dict checksum
This is the checksum of the compressed dict, used to detect whether
two dicts are identical. If there is no dict, the checksum must be
all zeros.
End of dict
This is the location of the end of the dict starting from the end of
the index. This gives us the information we need to find and
decompress the dict. If there is no dict, the checksum must be all
zeros.
Chunk checksum
This is the checksum of the compressed chunk, used to detect whether
any two chunks are identical.
End of chunk
This is the location of the end of the chunk starting from the end of
the index. This gives us the information we need to find and
decompress each chunk.
The index is designed to be able to be extracted from the file on the
server and downloaded separately, to facilitate downloading only the
parts of the file that are needed, but must then be re-embedded when
assembling the file so the user only needs to keep one file.
6 years
Fedora 27 Beta Freeze now in effect
by Kevin Fenzi
Greetings.
we are now in the infrastructure freeze leading up to the Fedora 27 Beta
release. This is a pre-release freeze.
We do this to ensure that our infrastructure is stable and ready to
release the Fedora 27 Beta when it's available.
You can see a list of hosts that do not freeze by checking out the
ansible repo and running the freezelist script:
git clone
https://infrastructure.fedoraproject.org/infra/ansible.git
ansible/scripts/freezelist -i inventory
Any hosts listed as freezes is frozen until 2017-09-19 (or later if Beta
slips). Frozen hosts should have no changes made to them without a
sign-off on the change from at least 2 sysadmin-main or rel-eng members,
along with (in most cases) a patch of the exact change to be made to
this list.
Thanks,
kevin
6 years
VDO
by Miroslav Suchý
After I seen the talk about VDO:
https://www.youtube.com/watch?v=7CGr5LEAfRY
I went ahead and tried it.
I tried it on small (12GB) sample of Copr data and I saved 20-30% data.
I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB.
Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of
VDO, which should work on Fedora. No RPM available for Fedora yet though.
Here are my notes:
# vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T
# mkfs.ext4 /dev/mapper/vdo1
Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning.
# mount -o discard /dev/mapper/vdo1 /mnt/a
Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that.
When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and
copy the data.
In my case, due limited space for migration I created a small LVM volume "vdosrv":
# lvcreate -L 80G -n vdosrv vol0
I converted it to VDO which claims to have 2TB of vitual total size
# vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T
# mkfs.ext4 /dev/mapper/srv18
See how many space is *actually* free.
# vdostats
# mkdir /mnt/srv18
# mount -o discard /dev/mapper/srv18 /mnt/srv18
Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I
enlarged the new LV.
# lvresize -L +400G /dev/vol0/vdosrv
and let know vdo that it can use the new space:
# vdo growPhysical -n srv18
I repeated this several times.
I have to say that my experience is so far good.
I hope that my notes help someone.
Miroslav
6 years, 1 month
fedp.org domain
by Dusty Mabe
I bought fedp.org last year thinking it's a shorter version
of fedoraproject.org that we could use if we wanted to. It's
about to expire. Should I let it go or do we want to use it?
Dusty
6 years, 1 month
Plan for tomorrow's Fedora Infrastructure Meeting 2018-02-08
by Stephen John Smoogen
Here is the current agenda. Please update the gobby with items to be discussed.
This shared document is for the next fedora infrastructure meeting.
= Preamble =
The infrastructure team will be having its weekly meeting tomorrow,
2018-03-01 at 18:00 UTC in #fedora-meeting on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
= Introduction =
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2018-03-01)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik pingou puiterwijk tflink
= Let new people say hello =
#topic New folks introductions
#info This is a place where people who are interested in Fedora
Infrastructure can introduce themselves
= Status / Information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need
to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info Infrastructure hackfest in april:
https://fedoraproject.org/wiki/Infrastructure_Hackathon_2018
#info The problem with 503's looks contained. Now they are 408s on the
server to look at
#info Rebuild mirrormanager container to be more resilient
#info Staging Koji synced with production - mizdebsk
#info New s390x builder in staging Koji, moved from prod - mizdebsk
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus /suor decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic Ticket cleanup
#info none this week.
#topic
= Apprentice office hours =
#topic Apprentice Open office minutes
#info A time where apprentices may ask for help or look at problems.
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for
improvement,
etc. Whoever would like to do this, just add the i/nfo in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#topic Learn about:
#info none this week
= Meeting end stuff =
#topic Open Floor
#endmeeting
--
Stephen J Smoogen.
6 years, 1 month
[PATCH] bodhi: remove atomic-config.py.j2
by Dusty Mabe
We used to use this when we used fedmsg-atomic-composer but since the
move to pungi I don't think this is used any longer.
---
roles/bodhi2/backend/templates/atomic-config.py.j2 | 412 ---------------------
1 file changed, 412 deletions(-)
delete mode 100644 roles/bodhi2/backend/templates/atomic-config.py.j2
diff --git a/roles/bodhi2/backend/templates/atomic-config.py.j2 b/roles/bodhi2/backend/templates/atomic-config.py.j2
deleted file mode 100644
index d75269cf2..000000000
--- a/roles/bodhi2/backend/templates/atomic-config.py.j2
+++ /dev/null
@@ -1,412 +0,0 @@
-# Check if we're running on RHEL6 and disable
-# `mock --new-chroot` and `rpm-ostree --workdir-tmpfs`
-import platform
-dist = platform.dist()
-rhel6 = dist[0] == 'redhat' and int(float(dist[1])) == 6
-
-config = dict(
- releases={
- 'f26-updates': {
- 'name': 'f26-updates',
- 'repo': 'updates',
- 'version': '26',
- 'arch': 'x86_64',
-
- # OSTree treefile configuration
- # https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-host.json',
- 'ref': 'fedora/26/x86_64/updates/atomic-host',
- 'repos': ['fedora-26', 'fedora-26-updates', 'updates'],
- 'packages': [],
- },
-
- # The name of the mock container to build and maintain
- 'mock': 'fedora-26-updates-x86_64',
-
- # The git branch to use in the `git_repo` for the parent
- # treefile & repo configurations
- 'git_branch': 'f26',
-
- # Add or overwrite yum repository name:urls. This lets you
- # compose trees against your own repositories.
- 'repos': {},
- },
- 'f26-updates-testing': {
- 'name': 'f26-updates-testing',
- 'repo': 'updates-testing',
- 'version': '26',
- 'arch': 'x86_64',
-
- # OSTree treefile configuration
- # https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-host.json',
- 'ref': 'fedora/26/x86_64/testing/atomic-host',
- 'repos': ['fedora-26', 'fedora-26-updates', 'updates-testing'],
- 'packages': [],
- },
-
- # The name of the mock container to build and maintain
- 'mock': 'fedora-26-updates-testing-x86_64',
-
- # The git branch to use in the `git_repo` for the parent
- # treefile & repo configurations
- 'git_branch': 'f26',
-
- # Add or overwrite yum repository name:urls. This lets you
- # compose trees against your own repositories.
- 'repos': {},
- },
- 'f25-updates': {
- 'name': 'f25-updates',
- 'repo': 'updates',
- 'version': '25',
- 'arch': 'x86_64',
-
- # OSTree treefile configuration
- # https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-docker-host.json',
- 'ref': 'fedora-atomic/25/x86_64/docker-host',
- 'repos': ['fedora-25', 'fedora-25-updates', 'updates'],
- 'packages': [],
- },
-
- # The name of the mock container to build and maintain
- 'mock': 'fedora-25-updates-x86_64',
-
- # The git branch to use in the `git_repo` for the parent
- # treefile & repo configurations
- 'git_branch': 'f25',
-
- # Add or overwrite yum repository name:urls. This lets you
- # compose trees against your own repositories.
- 'repos': {},
- },
- 'f25-updates-testing': {
- 'name': 'f25-updates-testing',
- 'repo': 'updates-testing',
- 'version': '25',
- 'arch': 'x86_64',
-
- # OSTree treefile configuration
- # https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-docker-host.json',
- 'ref': 'fedora-atomic/25/x86_64/testing/docker-host',
- 'repos': ['fedora-25', 'fedora-25-updates', 'updates-testing'],
- 'packages': [],
- },
-
- # The name of the mock container to build and maintain
- 'mock': 'fedora-25-updates-testing-x86_64',
-
- # The git branch to use in the `git_repo` for the parent
- # treefile & repo configurations
- 'git_branch': 'f25',
-
- # Add or overwrite yum repository name:urls. This lets you
- # compose trees against your own repositories.
- 'repos': {},
- },
-
- 'f24-updates': {
- 'name': 'f24-updates',
- 'repo': 'updates',
- 'version': '24',
- 'arch': 'x86_64',
-
- # OSTree treefile configuration
- # https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-docker-host.json',
- 'ref': 'fedora-atomic/24/x86_64/docker-host',
- 'repos': ['fedora-24', 'fedora-24-updates', 'updates'],
- 'packages': [],
- },
-
- # The name of the mock container to build and maintain
- 'mock': 'fedora-24-updates-x86_64',
-
- # The git branch to use in the `git_repo` for the parent
- # treefile & repo configurations
- 'git_branch': 'f24',
-
- # Add or overwrite yum repository name:urls. This lets you
- # compose trees against your own repositories.
- 'repos': {},
- },
- 'f24-updates-testing': {
- 'name': 'f24-updates-testing',
- 'repo': 'updates-testing',
- 'version': '24',
- 'arch': 'x86_64',
-
- # OSTree treefile configuration
- # https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-docker-host.json',
- 'ref': 'fedora-atomic/24/x86_64/testing/docker-host',
- 'repos': ['fedora-24', 'fedora-24-updates', 'updates-testing'],
- 'packages': [],
- },
-
- # The name of the mock container to build and maintain
- 'mock': 'fedora-24-updates-testing-x86_64',
-
- # The git branch to use in the `git_repo` for the parent
- # treefile & repo configurations
- 'git_branch': 'f24',
-
- # Add or overwrite yum repository name:urls. This lets you
- # compose trees against your own repositories.
- 'repos': {},
- },
-
- 'f23-updates': {
- 'name': 'f23-updates',
- 'repo': 'updates',
- 'version': '23',
- 'arch': 'x86_64',
-
- # OSTree treefile configuration
- # https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-docker-host.json',
- 'ref': 'fedora-atomic/f23/x86_64/docker-host',
- 'repos': ['fedora-23', 'fedora-23-updates'],
- 'packages': [],
- },
-
- # The name of the mock container to build and maintain
- 'mock': 'fedora-23-updates-x86_64',
-
- # The git branch to use in the `git_repo` for the parent
- # treefile & repo configurations
- 'git_branch': 'f23',
-
- # Add or overwrite yum repository name:urls. This lets you
- # compose trees against your own repositories.
- 'repos': {},
- },
- 'f23-updates-testing': {
- 'name': 'f23-updates-testing',
- 'repo': 'updates-testing',
- 'version': '23',
- 'arch': 'x86_64',
-
- # OSTree treefile configuration
- # https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-docker-host.json',
- 'ref': 'fedora-atomic/f23/x86_64/testing/docker-host',
- 'repos': ['fedora-23', 'fedora-23-updates', 'updates-testing'],
- 'packages': [],
- },
-
- # The name of the mock container to build and maintain
- 'mock': 'fedora-23-updates-testing-x86_64',
-
- # The git branch to use in the `git_repo` for the parent
- # treefile & repo configurations
- 'git_branch': 'f23',
-
- # Add or overwrite yum repository name:urls. This lets you
- # compose trees against your own repositories.
- 'repos': {},
- },
-
- 'f22-updates': {
- 'name': 'f22-updates',
- 'repo': 'updates',
- 'version': '22',
- 'arch': 'x86_64',
-
- # OSTree treefile configuration
- # https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-docker-host.json',
- 'ref': 'fedora-atomic/f22/x86_64/docker-host',
- 'repos': ['fedora-22', 'updates'],
- 'packages': [],
- },
-
- # The name of the mock container to build and maintain
- 'mock': 'fedora-22-updates-x86_64',
-
- # The git branch to use in the `git_repo` for the parent
- # treefile & repo configurations
- 'git_branch': 'f22',
-
- # Add or overwrite yum repository name:urls. This lets you
- # compose trees against your own repositories.
- 'repos': {},
- },
- 'f22-updates-testing': {
- 'name': 'f22-updates-testing',
- 'repo': 'updates-testing',
- 'version': '22',
- 'arch': 'x86_64',
-
- # OSTree treefile configuration
- # https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-docker-host.json',
- 'ref': 'fedora-atomic/f22/x86_64/testing/docker-host',
- 'repos': ['fedora-22', 'updates', 'updates-testing'],
- 'packages': [],
- },
-
- # The name of the mock container to build and maintain
- 'mock': 'fedora-22-updates-testing-x86_64',
-
- # The git branch to use in the `git_repo` for the parent
- # treefile & repo configurations
- 'git_branch': 'f22',
-
- # Add or overwrite yum repository name:urls. This lets you
- # compose trees against your own repositories.
- 'repos': {},
- },
-
- 'f21-updates': {
- 'name': 'f21-updates',
- 'repo': 'updates',
- 'version': '21',
- 'arch': 'x86_64',
-
- # OSTree treefile configuration
- # https://github.com/projectatomic/rpm-ostree/blob/master/doc/treefile.md
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-docker-host.json',
- 'ref': 'fedora-atomic/f21/x86_64/docker-host',
- 'repos': ['fedora-21', 'updates'],
- },
-
- # The name of the mock container to build and maintain
- 'mock': 'fedora-21-updates-x86_64',
-
- # The git branch to use in the `git_repo` for the parent
- # treefile & repo configurations
- 'git_branch': 'f21',
-
- # Add or overwrite yum repository name:urls. This lets you
- # compose trees against your own repositories.
- 'repos': {},
- },
-
- 'f21-updates-testing': {
- 'name': 'f21-updates-testing',
- 'repo': 'updates-testing',
- 'version': '21',
- 'arch': 'x86_64',
- 'tree': 'docker-host',
- 'treefile': {
- 'include': 'fedora-atomic-docker-host.json',
- 'ref': 'fedora-atomic/f21/x86_64/updates-testing/docker-host',
- 'repos': ['fedora-21', 'updates', 'updates-testing'],
- },
- 'git_branch': 'f21',
- 'mock': 'fedora-21-updates-testing-x86_64',
- 'repos': {},
- },
- },
-
- # Package repositories to use in the mock container and ostree compose
- repos={
-{% if env == 'production' %}
- 'fedora-{version}': 'file:///pub/fedora/linux/releases/{version}/Everything/{arch}/os',
- 'fedora-{version}-updates': 'file:///pub/fedora/linux/updates/{version}/{arch}/',
- 'updates': 'file:///pub/fedora/linux/updates/{version}/{arch}/',
- 'updates-testing': 'file:///pub/fedora/linux/updates/testing/{version}/{arch}/',
-{% else %}
- 'fedora-{version}': 'http://download01.phx2.fedoraproject.org/pub/fedora/linux/releases/{versi...',
- 'updates': 'http://download01.phx2.fedoraproject.org/pub/fedora/linux/updates/{versio...',
- 'updates-testing': 'http://download01.phx2.fedoraproject.org/pub/fedora/linux/updates/testing...',
-{% endif %}
- },
-
- # Output directories
-{% if env == 'production' %}
- prod_dir='/mnt/koji/mash/atomic',
-{% else %}
- prod_dir='/var/cache/bodhi/atomic',
-{% endif %}
- canonical_dir='{prod_dir}/{version}',
- work_dir='/srv/fedora-atomic',
- output_dir='{work_dir}/{version}/{arch}/{tree}',
- log_dir='{work_dir}/logs/{version}/{arch}/{repo}/{tree}',
-
-{% if env == 'production' %}
- mount_dirs=['/pub/fedora', '/mnt/koji/mash/updates'],
-{% else %}
- mount_dirs=[],
-{% endif %}
-
- # The git repo containing our parent treefiles and yum repos
- git_repo='https://pagure.io/fedora-atomic.git',
- git_cache='{work_dir}/fedora-atomic.git',
-
- # Some branches contain custom .repo files that we don't want to use
- delete_repo_files=True,
-
- # Mock command
- mock_cmd='/usr/bin/mock%s-r {mock}' % (rhel6 and ' ' or ' --new-chroot '),
- #mock_cmd='/usr/bin/mock -r {mock}',
- mock_clean=True,
-
- # OSTree commands
- ostree_init='/usr/bin/ostree --repo={output_dir} init --mode=archive-z2',
- ostree_compose='/usr/bin/rpm-ostree compose tree' +
- (rhel6 and ' ' or ' --workdir-tmpfs ') + '--repo={output_dir} %s',
- ostree_summary='/usr/bin/ostree --repo={output_dir} summary --update',
-
- # rsync commands
-{% if env == 'production' %}
- rsync_in_objs='/usr/bin/rsync -rvp --ignore-existing {canonical_dir}/objects/ {output_dir}/objects/',
- rsync_in_rest='/usr/bin/rsync -rvp --exclude=objects/ {canonical_dir}/ {output_dir}/',
- rsync_out_objs='/usr/bin/rsync -rvp --ignore-existing {output_dir}/objects/ {canonical_dir}/objects/',
- rsync_out_rest='/usr/bin/rsync -rvp --exclude=objects/ {output_dir}/ {canonical_dir}/',
-{% else %}
- rsync_in_objs='',
- rsync_in_rest='',
- rsync_out_objs='',
- rsync_out_rest='',
-{% endif %}
-
- map_to_release=('work_dir', 'prod_dir', 'output_dir', 'log_dir',
- 'git_repo', 'git_cache', 'mock_cmd', 'ostree_init',
- 'ostree_compose', 'ostree_summary', 'canonical_dir',
- 'repos', 'rsync_in_objs', 'rsync_in_rest', 'rsync_out_objs',
- 'rsync_out_rest', 'mount_dirs', 'mock_clean',
- 'delete_repo_files'),
-)
-
-# Map and expand certain variables to each release
-for key in config.get('map_to_release', []):
- for name, release in config['releases'].items():
- if isinstance(config[key], dict):
- release[key] = {}
- for k, v in config[key].items():
- k = k.format(**release)
- release[key][k] = v.format(**release)
- elif isinstance(config[key], (list, tuple)):
- release[key] = []
- for item in config[key]:
- release[key].append(item.format(**release))
- elif isinstance(config[key], bool):
- release[key] = config[key]
- else:
- release[key] = config[key].format(**release)
--
2.14.3
6 years, 2 months
