So, I think we have two groups approaching a replacement app for packages?
Can we perhaps decide which way we want to go and look at deploying something soon? This has been more important since we didn't move the old broken packages app from the old datacenter and people keep asking about it.
We could wait until we have staging back, but I don't think we need to wait for that to decide what we want to do here?
Thoughts?
kevin
Hi,
Timothée and I are working on fedora-packages-static. It's a lot simpler than the old packages app, it just generates static files that you can serve. The old packages app was missing packages and would time out if an external service took too long to respond. This approach eliminates those problems by not having the web server do that work on demand. Because these are just static files, search will need to be managed by something else (E.g. YaCy, Algolia). I have generated an example page and put a link below.
Example: https://mymindstorm.fedorapeople.org/pkgs-demo/pkgs/numix-icon-theme-square/
Pagure: https://pagure.io/fedora-packages-static
--
Brendan Early
On 7/16/20 12:06 PM, Kevin Fenzi wrote:
So, I think we have two groups approaching a replacement app for packages?
Can we perhaps decide which way we want to go and look at deploying something soon? This has been more important since we didn't move the old broken packages app from the old datacenter and people keep asking about it.
We could wait until we have staging back, but I don't think we need to wait for that to decide what we want to do here?
Thoughts?
kevin
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Thu, Jul 16, 2020 at 02:56:08PM -0500, Brendan Early wrote:
Hi,
Timothée and I are working on fedora-packages-static. It's a lot simpler than the old packages app, it just generates static files that you can serve. The old packages app was missing packages and would time out if an external service took too long to respond. This approach eliminates those problems by not having the web server do that work on demand. Because these are just static files, search will need to be managed by something else (E.g. YaCy, Algolia). I have generated an example page and put a link below.
Example: https://mymindstorm.fedorapeople.org/pkgs-demo/pkgs/numix-icon-theme-square/
Yeah, and msuchy (and others) have been working on: https://github.com/xsuchy/fedora-packages-ng
My point is that we shouldn't have 2 groups working on the replacement, that seems like it's taking time from one group we don't decide to use the app from. Or is there a possibily to merge the two apps or split the functionality up so both are complementary?
kevin --
Pagure: https://pagure.io/fedora-packages-static
--
Brendan Early
On 7/16/20 12:06 PM, Kevin Fenzi wrote:
So, I think we have two groups approaching a replacement app for packages?
Can we perhaps decide which way we want to go and look at deploying something soon? This has been more important since we didn't move the old broken packages app from the old datacenter and people keep asking about it.
We could wait until we have staging back, but I don't think we need to wait for that to decide what we want to do here?
Thoughts?
kevin
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 7/17/20 6:40 PM, Kevin Fenzi wrote:
On Thu, Jul 16, 2020 at 02:56:08PM -0500, Brendan Early wrote:
Hi,
Timothée and I are working on fedora-packages-static. It's a lot simpler than the old packages app, it just generates static files that you can serve. The old packages app was missing packages and would time out if an external service took too long to respond. This approach eliminates those problems by not having the web server do that work on demand. Because these are just static files, search will need to be managed by something else (E.g. YaCy, Algolia). I have generated an example page and put a link below.
Example:
https://mymindstorm.fedorapeople.org/pkgs-demo/pkgs/numix-icon-theme-square/
Yeah, and msuchy (and others) have been working on: https://github.com/xsuchy/fedora-packages-ng
My point is that we shouldn't have 2 groups working on the replacement, that seems like it's taking time from one group we don't decide to use the app from. Or is there a possibily to merge the two apps or split the functionality up so both are complementary?
I didn't realize at the time that there were two solutions being developed. Timothée was giving lots of status updates and you and Clement were active in those discussions, so it wasn't very obvious from my point of view as a fairly new contributor here. I don't know if it's possible to merge the two. I don't see the point of spitting the functionality as they mostly do the same thing.
Brendan Early
kevin
Pagure: https://pagure.io/fedora-packages-static
--
Brendan Early
On 7/16/20 12:06 PM, Kevin Fenzi wrote:
So, I think we have two groups approaching a replacement app for packages?
Can we perhaps decide which way we want to go and look at deploying something soon? This has been more important since we didn't move the old broken packages app from the old datacenter and people keep asking about it.
We could wait until we have staging back, but I don't think we need to wait for that to decide what we want to do here?
Thoughts?
kevin
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to
infrastructure-leave@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to
infrastructure-leave@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to
infrastructure-leave@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Tue, Jul 21, 2020 at 04:14:39PM -0500, Brendan Early wrote:
I didn't realize at the time that there were two solutions being developed. Timothée was giving lots of status updates and you and Clement were active in those discussions, so it wasn't very obvious from my point of view as a fairly new contributor here. I don't know if it's possible to merge the two. I don't see the point of spitting the functionality as they mostly do the same thing.
Right. I'm happy for any good solution...I wasn't aware that msuchy was working on a replacement (other than just saying he would if he found time) until he appeared with the initial project. ;)
kevin --
Brendan Early
kevin
Pagure: https://pagure.io/fedora-packages-static
--
Brendan Early
On 7/16/20 12:06 PM, Kevin Fenzi wrote:
So, I think we have two groups approaching a replacement app for packages?
Can we perhaps decide which way we want to go and look at deploying something soon? This has been more important since we didn't move the old broken packages app from the old datacenter and people keep asking about it.
We could wait until we have staging back, but I don't think we need to wait for that to decide what we want to do here?
Thoughts?
kevin
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to
infrastructure-leave@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to
infrastructure-leave@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to
infrastructure-leave@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... -----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEEuO7QAvcjGvjf8zLJc9/feL/WqkFAl8XWqsACgkQJc9/feL/ WqmQgA/+Lki6d5XZjItBp9k8jtapnaaCPQkClYbZ+j1WE51onGhktD89x9LE/GtJ WAqdbEdMRcgJPMAtOiYAQzlWXwny6qvODw+XoOqFf0Zp5VyAeOkxyhczdLDguP7N NYoWUj86S0QK+SFZyKMin8ExD9tnICIX/e9J9UCKNmDGxvMADtm3A5iO3LRxOaN4 8Ko+S3EE4FujBDpIN/+WMZbCSXKU4j1N2KD2tPXaeimzyKCShFXXggGc4kAsJI0/ xsCJNl8WO7gZ/8kw/hjZU17q+l68LR4e01+/OBEsHZhCVbxG/RGkNsXIA2UBYF4A sH7mibpgFcULJr+5fhPjgyRTeTl5dFK6Wy2ikneNzDH3v44DT34Y1OaWyvdnaSuE gz0+ouZ9UOYKNPxi3kysDZkPj9A+n/Oq0Hze2c5CP1gtkROUzjpQdDeO/0Qy9upu Kt3VjkqMB320OVCdHrbLhhiirazePWxOoQnFBwIrkyRGVKrIrYhQQmpVwxcIwp3Y MW2xlPphqRkpzaVkblkTBQ5rQdFJc4B3X04q3u2+su48TMztFoEzQRC0U1DTRK2Q c+Q8crkE9Tvt25Y2OyIADPABUnVwhtXFxRZCrVemWy87kGxNY0Uf7lHJ+RQyZjst B69+We2MlU47Vw0KWMeIPbKn65J3YYG7jzlKY2Qe/1aH8/VUUzY= =hbr0 -----END PGP SIGNATURE-----
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
Dne 16. 07. 20 v 21:56 Brendan Early napsal(a):
and would time out if an external service took too long to respond.
Only if you wanted to list builds, bugs.... the overview page is more or less static (i.e. it is rendered just from local DB). Fedora-packages-static does not list them at all. This is ok when we agree that this is what people find sufficient.
What I am curious about fedora-packages-static - how long it takes to generate all those pages? When I tried to retrieve initial data fedora-packages-ng, it took several hours. Which may be too much for daily job. How much data it consumes? What is yacysearchserver-pkgs-playground.apps.os.fedorainfracloud.org? Or when we delegate search to Google, how long it takes to refresh the search index? Will be people ok with the speed?
On 7/27/20 9:40 AM, Miroslav Suchý wrote:
Dne 16. 07. 20 v 21:56 Brendan Early napsal(a):
and would time out if an external service took too long to respond.
Only if you wanted to list builds, bugs.... the overview page is more or less static (i.e. it is rendered just from local DB).
I was basing that statement off of a quick skim of old fedora packages source. I may be mistaken as I'm still new to python, but I was under the impression that it always queried koji when it was given a request.
https://github.com/fedora-infra/fedora-packages/blob/master/fedoracommunity/...
Fedora-packages-static does not list them at all. This is ok when we agree that this is what people find sufficient.
This is intentional, I can easily write some client-side JS to add that functionality if it is needed.
https://pagure.io/fedora-packages-static/pull-request/1#comment-0
What I am curious about fedora-packages-static - how long it takes to generate all those pages? When I tried to retrieve initial data fedora-packages-ng, it took several hours. Which may be too much for daily job.
How are you getting the data in ng?! From a clean state on my computer, it takes packages-static around 140 seconds to get the package metadata into the db and about 195 seconds to generate all the static pages.
How much data it consumes?
Not anything monumental, just ~6.3G.
What is yacysearchserver-pkgs-playground.apps.os.fedorainfracloud.org?
A staging YaCy server that Timothée had on communishift.
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
Or when we delegate search to Google, how long it takes to refresh the search index?
I have code to generate sitemaps, which can be used to tell Google how often we want to be crawled and it will make sure every page is indexed. This should improve discoverability, I don't know how well the old one was indexed.
Will be people ok with the speed?
I don't understand exactly what you are referring to here. If you mean the speed of search, I don't specifically know about the speed of Solr. In the event hosting something ourselves does not work out we could always go with Algolia, which is near-instant.
Dne 28. 07. 20 v 1:18 Brendan Early napsal(a):
How are you getting the data in ng?! From a clean state on my computer,
The code remained unchanged. The most time the code spend quering PDC. Here: https://github.com/xsuchy/fedora-packages-ng/blob/master/fedoracommunity/sea...
it takes packages-static around 140 seconds to get the package metadata into the db and about 195 seconds to generate all the static pages.
How much data it consumes?> Not anything monumental, just ~6.3G.
Nice.
Will be people ok with the speed?
I don't understand exactly what you are referring to here. If you mean the speed of search,
I meant the speed of update of Google index.
On 7/28/20 5:33 AM, Miroslav Suchý wrote:
Will be people ok with the speed?
I don't understand exactly what you are referring to here. If you mean the speed of search,
I meant the speed of update of Google index.
Google doesn't say how long it takes them to index. Assuming we submit the sitemap to Google, hopefully the initial index shouldn't be more than a few weeks. We can set the page description to the package description so that constant crawls aren't necessary.
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure@lists.fedoraproject.org
-
Brendan Early -
Kevin Fenzi -
Miroslav Suchý