Upcoming changes on datanomer/datagrepper
by Ant Carroll
Hey folks,
One of the initiatives the CPE team has taken over this quarter is updating
and improving the performances of datagrepper/datanomer.
That work is based upon the investigation done earlier in the year whose
results can be found at:
https://fedora-arc.readthedocs.io/en/latest/datanommer_datagrepper/index....
One important item to note is that we will introduce a default value for
the `delta` argument.
Currently, if you do not specify one, datagrepper will not limit its
queries by time, which is very resource intensive and often so much so that
you do not get any results back.
By setting a default `delta` argument we can ensure that just browsing the
UI does not throw random time-out errors and increase the performance of
some of the queries.
However, this is a **non-backward compatible change**!
If you have scripts relying on the fact that the messages returned can be
of any date, they will break once the default delta is in place. You should
thus adjust them to specify a delta (which will override the default one).
The default delta will likely be set to "one day" but that's not fixed yet.
Once it has been fully confirmed we'll update again, so please treat this
as an FYI, but also start looking at your scripts. Specifying the `delta`
today won't break the current behavior and will ensure they keep working
once the change has landed.
Secondly, the grouped feature [2] will be dropped as it's not supported in
the new library. If you're using it, please reach out to us so we can
figure out mitigation plans.
Enjoy Nest if you're attending and have a nice weekend!
Cheers,
Ant
[1]
https://fedora-arc.readthedocs.io/en/latest/datanommer_datagrepper/index....
[2] See the section "Formatting arguments" at:
https://apps.fedoraproject.org/datagrepper/reference
--
Ant Carroll
Associate Manager, Software Engineering
Red Hat Waterford <http://www.redhat.com>
Communications House
Cork Road, Waterford City
ancarrol(a)redhat.com
M: +353876213163 IM: ancarrol
@redhatjobs <https://twitter.com/redhatjobs> redhatjobs
<https://www.facebook.com/redhatjobs> @redhatjobs
<https://instagram.com/redhatjobs>
<https://www.redhat.com/>
2 years, 8 months
Community Platform Engineering Team July Report
by Aoife Moloney
Hi all,
Here is a quick snapshot of some of the work delivered by the
Community Platform Engineering team during the month of July.
If you have trouble reading this in email format, feel free to visit
the hackmd link where it is published
https://hackmd.io/@Ap8CkTlpSfmjb44UGV-kWA/rkW5c8W1t
## Fedora
### Infrastructure & Release Engineering
* Mass rebuild completed
* Additional power8 boxes installed in one of the datacentres
### Scheduled Q3 Projects
#### Datanomer/Datagrepper
**About** These apps are currently used to retrieve historical
information about messages on the fedmsg bus and add them to a
postgresql database. Datanomer reads-in messages from the bus and
stores them in the database and Datagrepper exposes the messages in
the database via an API with different filtering capacities. We want
to upgrade these applications to use fedora-messaging and increase the
performance of the applications for users.
Work Tracker: https://github.com/orgs/fedora-infra/projects/10
**Delivered in July**
* Converted datanomer.consumer to use fedora-messaging
* Timescaledb is deployed in staging
* Vagrant setup created to help new contributors hack on datagrepper easier
#### Metrics for Apps on Openshift
**About** The project team will deliver on the installation of newest
Openshift in Fedora infra to be then configured with prometheus so
applications can be hooked into this service and can be monitored and
metricized through this tech stack for more comprehensive
understanding of app behaviour, performance and troubleshooting.
Work Tracker: https://pagure.io/fedora-infra/metrics-for-apps/boards/metrics-for-apps
**Delivered in July**
* Bootstrap Openshift Install
* Staging cluster of Openshift 4 up
#### DNF Counting Project
**About**: The DNF counting project will enhance the currently
existing program that captures a ‘countme’ value from computers
running Fedora weekly. There are multiple scripts running from a
server which parses the data into a csv file for graphing and
displaying trends that had been misbehaving a lot. This program gives
a good indication for how many computers use fedora so the project
team will work on improving these scripts and the program overall to
make it a more reliable and maintainable solution.
Work Tracker: https://pagure.io/mirrors-countme/boards/DNF%20Counting%20Initiative
**Delivered in July**:
* Zuul CI testing enabled
* Added F34 stats
* Improved the syncHttpLogs script so that it retries on failure and
has a timeout and no longer hangs quietly for long periods of time
## CentOS
### Infrastructure & Release Engineering
https://pagure.io/centos-infra/issues
* New Automotive SIG set up complete
* New Infra SIG member added - Mark O'Brien (IRC:mobrien)
### CentOS Linux
* Kmod secure boot signing is now in place
### CentOS Stream
#### Stream 8
* Legacy composeinfo is now being generated for every compose
* Kernel-4.18.0-326.el8 has been exported to git.centos.org and pushed
to the mirrors for CentOS Stream 8
#### Stream 9
* Daily composes now being generated for Stream 9
* Stream 9 composes now does ec2 and generic qcow2 images
* Tooling identified for package signing automation
Thank you for your continued support and collaboration with our team
across all communities, the work we achieve is not possible without
that, and I look forward to updating you all on what we have
accomplished at the end of August!
Kindest regards & on behalf of the CPE team,
Aoife
--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
2 years, 8 months
CPE Feedback Survey
by Ant Carroll
Hey folks,
Over the last several months we’ve been trying to continue our improvements
with how we interact and share information with you all based on the
feedback of the last surveys. From the blog posts, to mails and how we work
on the tickets you send us.
Here is a link to our very short survey where you can give us that
feedback: https://fedoraproject.limequery.com/5
If you could take the time (5mins max) to complete it for us it would be
massively appreciated as always. We're also moving the cadence of this to
once every 6 months. The shorter survey times at the start were great to
give us that initial burst of improvements to work on, and the increases in
happiness we've seen from you reflect the work the team has done.
Hope to see some of you at Nest,
Ant
--
Ant Carroll
Associate Manager, Software Engineering
Red Hat Waterford <http://www.redhat.com>
Communications House
Cork Road, Waterford City
ancarrol(a)redhat.com
M: +353876213163 IM: ancarrol
@redhatjobs <https://twitter.com/redhatjobs> redhatjobs
<https://www.facebook.com/redhatjobs> @redhatjobs
<https://instagram.com/redhatjobs>
<https://www.redhat.com/>
2 years, 8 months