Good Morning everyone,
Over the last few days I have been working on a small app: mdapi. It is aimed at serving the metadata from our repos simply and *fast*, offering information from koji, rawhide, all our active branches and epel (you'll have to specify which you want) and for each it will return you the first hit it finds in the testing, updates or release repo (it says which in the json returned).
I deployed it for testing in our cloud, at: http://209.132.184.236/
This morning I took a look at what it would take to package mdapi for epel7. mdapi has 5 dependencies: - aiohttp -> Has an epel7 branch but was never built, open a ticket on the bugzilla asking it - requests -> Exists in RHEL channels but not as py3, so will need a new package - simplejson -> Asked the current PoC if it could be built with the py3 sub-package in epel7 - sqlalchemy -> Looked at it (I'm the PoC for epel7), but requires: - python3-setuptools (same situation as requests) - requires: python3-pip, python3-wheel, python3-mock, python3-pytest - python3-nose (same situation as requests and setuptools as requests - werkzeug -> same situation as requests, setuptools...
The dependency list as well as the current situation of the guidelines for packaging python3 app in epel [1], makes me proposing that we deploy mdapi in Fedora (at least for now).
The second point I would like to raise is how we deploy this application. This isn't a standard wsgi application (since it's async) and it cannot run with apache (afaiu). So far the other async application we have (in pagure) have been deployed simply as a systemd service. I wonder if we want to use the same approach here or if we should investigate things like gunicorn/nginx or so. Does someone have experience in this field? Any advice/feedback?
Thanks for your attention, Pierre
"PC" == Pierre-Yves Chibon pingou@pingoured.fr writes:
PC> The dependency list as well as the current situation of the PC> guidelines for packaging python3 app in epel [1], makes me proposing PC> that we deploy mdapi in Fedora (at least for now).
Note that there's nothing blocking importation of python3 packages into epel. The module packaging is just far uglier than we'd like, with potentially three different python versions for which you have to build. None of this should have anything to do with an application anyway.
- J<
On Tue, Oct 27, 2015 at 12:04:07PM -0500, Jason L Tibbitts III wrote:
"PC" == Pierre-Yves Chibon pingou@pingoured.fr writes:
PC> The dependency list as well as the current situation of the PC> guidelines for packaging python3 app in epel [1], makes me proposing PC> that we deploy mdapi in Fedora (at least for now).
Note that there's nothing blocking importation of python3 packages into epel. The module packaging is just far uglier than we'd like, with potentially three different python versions for which you have to build. None of this should have anything to do with an application anyway.
Ok so we should just be able to create the python34-foo* packages we need in epel7 and open their corresponding review request.
Thanks!
Pierre
* python34-foo or python3-foo then?
"PC" == Pierre-Yves Chibon pingou@pingoured.fr writes:
PC> Ok so we should just be able to create the python34-foo* packages we PC> need in epel7 and open their corresponding review request.
Assuming they're already in Fedora, you just need to get them branched and get the spec going. You wouldn't generally name the source package python34-*, you'd just call it python-*.
- J<
On Tue, Oct 27, 2015 at 04:27:39PM -0500, Jason L Tibbitts III wrote:
"PC" == Pierre-Yves Chibon pingou@pingoured.fr writes:
PC> Ok so we should just be able to create the python34-foo* packages we PC> need in epel7 and open their corresponding review request.
Assuming they're already in Fedora, you just need to get them branched and get the spec going. You wouldn't generally name the source package python34-*, you'd just call it python-*.
My problem is more for the package in Fedora also present in RHEL, thus where we only want the python3 version as otherwise we conflict.
Pierre
"PC" == Pierre-Yves Chibon pingou@pingoured.fr writes:
PC> My problem is more for the package in Fedora also present in RHEL, PC> thus where we only want the python3 version as otherwise we PC> conflict.
That's where either you don't build the python2 version on RHEL, by using a separate spec or with a mass of ifdefs. I guess this hinges on whether having an _srpm_ with the same name as one in RHEL would cause an issue for EPEL, even if there's no conflict with the binary packages. I would guess not as I haven't seen any mention of mass-reviews (or exemptions for such) for python3-* packages in EPEL.
- J<
"JLT" == Jason L Tibbitts tibbs@math.uh.edu writes:
JLT> I guess this hinges on whether having an _srpm_ with the same name JLT> as one in RHEL would cause an issue for EPEL, even if there's no JLT> conflict with the binary packages.
Just confirmed that this isn't an issue, since it's already done in a few cases.
- J<
On Wed, 28 Oct 2015 10:40:56 -0500 Jason L Tibbitts III tibbs@math.uh.edu wrote:
"PC" == Pierre-Yves Chibon pingou@pingoured.fr writes:
PC> My problem is more for the package in Fedora also present in RHEL, PC> thus where we only want the python3 version as otherwise we PC> conflict.
That's where either you don't build the python2 version on RHEL, by using a separate spec or with a mass of ifdefs. I guess this hinges on whether having an _srpm_ with the same name as one in RHEL would cause an issue for EPEL, even if there's no conflict with the binary packages. I would guess not as I haven't seen any mention of mass-reviews (or exemptions for such) for python3-* packages in EPEL.
koji operates on package names. It will only get rpms from a package in one place, it won't mix them at all.
So, if RHEL has python-foo and EPEL creates a package named python-foo, it will cause koji to completely and utterly ignore the RHEL python-foo and all rpms it makes.
So, you will need to make python3-foo for python3 support in EPEL where there's a python2 version in RHEL.
kevin
On Wed, Oct 28, 2015 at 10:06:53AM -0600, Kevin Fenzi wrote:
koji operates on package names. It will only get rpms from a package in one place, it won't mix them at all.
So, if RHEL has python-foo and EPEL creates a package named python-foo, it will cause koji to completely and utterly ignore the RHEL python-foo and all rpms it makes.
In which case does it not mix packages? It is allowed for limited arch packages to use the same package name as RHEL: https://fedoraproject.org/wiki/EPEL:Packaging#Limited_Arch_Packages
Regards Till
On Wed, 28 Oct 2015 19:16:35 +0100 Till Maas opensource@till.name wrote:
On Wed, Oct 28, 2015 at 10:06:53AM -0600, Kevin Fenzi wrote:
koji operates on package names. It will only get rpms from a package in one place, it won't mix them at all.
So, if RHEL has python-foo and EPEL creates a package named python-foo, it will cause koji to completely and utterly ignore the RHEL python-foo and all rpms it makes.
In which case does it not mix packages? It is allowed for limited arch packages to use the same package name as RHEL: https://fedoraproject.org/wiki/EPEL:Packaging#Limited_Arch_Packages
Koji doesn't mix those. When thats done the entire package is taken from EPEL.
So, if you buildrequire foo and it's in both epel and rhel it will use the epel version. koji will not use the rhel version for anything at all.
The reason that policy has:
"Change the release of the package to have a leading 0. EXAMPLE: RHEL has foobar-1.0-1, you change it to foobar-1.0-0.1 for EPEL."
Is not for koji, but for end users. That makes the EPEL version lower E-N-V-R than the rhel one for arches that rhel ships it, so people get the rhel version.
kevin
On Tue, Oct 27, 2015 at 04:00:56PM +0100, Pierre-Yves Chibon wrote:
Good Morning everyone,
Over the last few days I have been working on a small app: mdapi. It is aimed at serving the metadata from our repos simply and *fast*, offering information from koji, rawhide, all our active branches and epel (you'll have to specify which you want) and for each it will return you the first hit it finds in the testing, updates or release repo (it says which in the json returned).
I deployed it for testing in our cloud, at: http://209.132.184.236/
[...]
The second point I would like to raise is how we deploy this application. This isn't a standard wsgi application (since it's async) and it cannot run with apache (afaiu). So far the other async application we have (in pagure) have been deployed simply as a systemd service. I wonder if we want to use the same approach here or if we should investigate things like gunicorn/nginx or so. Does someone have experience in this field? Any advice/feedback?
And regarding this point, anyone has experience with these services?
Thanks, Pierre
On 10/29/2015 03:53 PM, Pierre-Yves Chibon wrote:
On Tue, Oct 27, 2015 at 04:00:56PM +0100, Pierre-Yves Chibon wrote:
Good Morning everyone,
Over the last few days I have been working on a small app: mdapi. It is aimed at serving the metadata from our repos simply and *fast*, offering information from koji, rawhide, all our active branches and epel (you'll have to specify which you want) and for each it will return you the first hit it finds in the testing, updates or release repo (it says which in the json returned).
I deployed it for testing in our cloud, at: http://209.132.184.236/
[...]
The second point I would like to raise is how we deploy this application. This isn't a standard wsgi application (since it's async) and it cannot run with apache (afaiu). So far the other async application we have (in pagure) have been deployed simply as a systemd service. I wonder if we want to use the same approach here or if we should investigate things like gunicorn/nginx or so. Does someone have experience in this field? Any advice/feedback?
And regarding this point, anyone has experience with these services?
Yes, this setup works well with django/flask when you need maximum performance or you don't want to run apache.
Deployment is easy. It required supervisord but that is no longer the case as now we have systemd that can take care of running gunicron.
Cheers,
On Thu, Oct 29, 2015 at 04:21:41PM +0100, Richard Marko wrote:
On 10/29/2015 03:53 PM, Pierre-Yves Chibon wrote:
On Tue, Oct 27, 2015 at 04:00:56PM +0100, Pierre-Yves Chibon wrote:
Good Morning everyone,
Over the last few days I have been working on a small app: mdapi. It is aimed at serving the metadata from our repos simply and *fast*, offering information from koji, rawhide, all our active branches and epel (you'll have to specify which you want) and for each it will return you the first hit it finds in the testing, updates or release repo (it says which in the json returned).
I deployed it for testing in our cloud, at: http://209.132.184.236/
[...]
The second point I would like to raise is how we deploy this application. This isn't a standard wsgi application (since it's async) and it cannot run with apache (afaiu). So far the other async application we have (in pagure) have been deployed simply as a systemd service. I wonder if we want to use the same approach here or if we should investigate things like gunicorn/nginx or so. Does someone have experience in this field? Any advice/feedback?
And regarding this point, anyone has experience with these services?
Yes, this setup works well with django/flask when you need maximum performance or you don't want to run apache.
You mean using nginx as a replacement for apache, correct?
Deployment is easy. It required supervisord but that is no longer the case as now we have systemd that can take care of running gunicron.
So the application is written in aiohttp, I'm pretty sure I can make it run as a systemd service but I have no experience of gunicorn. Would you advice for it? Does it bring us much?
Thanks, Pierre
On 10/29/2015 04:46 PM, Pierre-Yves Chibon wrote:
On Thu, Oct 29, 2015 at 04:21:41PM +0100, Richard Marko wrote:
On 10/29/2015 03:53 PM, Pierre-Yves Chibon wrote:
On Tue, Oct 27, 2015 at 04:00:56PM +0100, Pierre-Yves Chibon wrote:
Good Morning everyone,
Over the last few days I have been working on a small app: mdapi. It is aimed at serving the metadata from our repos simply and *fast*, offering information from koji, rawhide, all our active branches and epel (you'll have to specify which you want) and for each it will return you the first hit it finds in the testing, updates or release repo (it says which in the json returned).
I deployed it for testing in our cloud, at: http://209.132.184.236/
[...]
The second point I would like to raise is how we deploy this application. This isn't a standard wsgi application (since it's async) and it cannot run with apache (afaiu). So far the other async application we have (in pagure) have been deployed simply as a systemd service. I wonder if we want to use the same approach here or if we should investigate things like gunicorn/nginx or so. Does someone have experience in this field? Any advice/feedback?
And regarding this point, anyone has experience with these services?
Yes, this setup works well with django/flask when you need maximum performance or you don't want to run apache.
You mean using nginx as a replacement for apache, correct?
Correct. Apache is often replaced with nginx/lighttpd when you don't need features like kerberos, WebDAV, ...
Deployment is easy. It required supervisord but that is no longer the case as now we have systemd that can take care of running gunicron.
So the application is written in aiohttp, I'm pretty sure I can make it run as a systemd service but I have no experience of gunicorn. Would you advice for it? Does it bring us much?
Depends on how much load do you expect on mdapi. I haven't done any testing performance wise so I can't tell if it's actually worth. I would say try it as there's not much difference in effort required for running an app with apache/wsgi compared to nginx/gunicorn. nginx/gunicorn configuration is quite easy, I would say even easier now (no supervisord required) than apache/wsgi.
On Thu, Oct 29, 2015 at 05:05:06PM +0100, Richard Marko wrote:
On 10/29/2015 04:46 PM, Pierre-Yves Chibon wrote:
On Thu, Oct 29, 2015 at 04:21:41PM +0100, Richard Marko wrote:
On 10/29/2015 03:53 PM, Pierre-Yves Chibon wrote:
On Tue, Oct 27, 2015 at 04:00:56PM +0100, Pierre-Yves Chibon wrote:
Good Morning everyone,
Over the last few days I have been working on a small app: mdapi. It is aimed at serving the metadata from our repos simply and *fast*, offering information from koji, rawhide, all our active branches and epel (you'll have to specify which you want) and for each it will return you the first hit it finds in the testing, updates or release repo (it says which in the json returned).
I deployed it for testing in our cloud, at: http://209.132.184.236/
[...]
The second point I would like to raise is how we deploy this application. This isn't a standard wsgi application (since it's async) and it cannot run with apache (afaiu). So far the other async application we have (in pagure) have been deployed simply as a systemd service. I wonder if we want to use the same approach here or if we should investigate things like gunicorn/nginx or so. Does someone have experience in this field? Any advice/feedback?
And regarding this point, anyone has experience with these services?
<snip>
Deployment is easy. It required supervisord but that is no longer the case as now we have systemd that can take care of running gunicron.
So the application is written in aiohttp, I'm pretty sure I can make it run as a systemd service but I have no experience of gunicorn. Would you advice for it? Does it bring us much?
Depends on how much load do you expect on mdapi. I haven't done any testing performance wise so I can't tell if it's actually worth. I would say try it as there's not much difference in effort required for running an app with apache/wsgi compared to nginx/gunicorn. nginx/gunicorn configuration is quite easy, I would say even easier now (no supervisord required) than apache/wsgi.
Well, the application is asynchronous, so apache is actually not an option :) So my question is really mdapi alone vs mdapi in nginx/gunicorn You seem to encourage checking gunicorn, do you have a preferred resource/howto for it?
Pierre
On 10/29/2015 05:34 PM, Pierre-Yves Chibon wrote:
On Thu, Oct 29, 2015 at 05:05:06PM +0100, Richard Marko wrote:
On 10/29/2015 04:46 PM, Pierre-Yves Chibon wrote:
On Thu, Oct 29, 2015 at 04:21:41PM +0100, Richard Marko wrote:
On 10/29/2015 03:53 PM, Pierre-Yves Chibon wrote:
On Tue, Oct 27, 2015 at 04:00:56PM +0100, Pierre-Yves Chibon wrote:
Good Morning everyone,
Over the last few days I have been working on a small app: mdapi. It is aimed at serving the metadata from our repos simply and *fast*, offering information from koji, rawhide, all our active branches and epel (you'll have to specify which you want) and for each it will return you the first hit it finds in the testing, updates or release repo (it says which in the json returned).
I deployed it for testing in our cloud, at: http://209.132.184.236/
[...]
The second point I would like to raise is how we deploy this application. This isn't a standard wsgi application (since it's async) and it cannot run with apache (afaiu). So far the other async application we have (in pagure) have been deployed simply as a systemd service. I wonder if we want to use the same approach here or if we should investigate things like gunicorn/nginx or so. Does someone have experience in this field? Any advice/feedback?
And regarding this point, anyone has experience with these services?
<snip>
Deployment is easy. It required supervisord but that is no longer the case as now we have systemd that can take care of running gunicron.
So the application is written in aiohttp, I'm pretty sure I can make it run as a systemd service but I have no experience of gunicorn. Would you advice for it? Does it bring us much?
Depends on how much load do you expect on mdapi. I haven't done any testing performance wise so I can't tell if it's actually worth. I would say try it as there's not much difference in effort required for running an app with apache/wsgi compared to nginx/gunicorn. nginx/gunicorn configuration is quite easy, I would say even easier now (no supervisord required) than apache/wsgi.
Well, the application is asynchronous, so apache is actually not an option :) So my question is really mdapi alone vs mdapi in nginx/gunicorn You seem to encourage checking gunicorn, do you have a preferred resource/howto for it?
This page has very nice documentation about deployment options: http://docs.gunicorn.org/en/latest/deploy.html
Cheers,
infrastructure@lists.fedoraproject.org
-
Jason L Tibbitts III -
Kevin Fenzi -
Pierre-Yves Chibon -
Richard Marko -
Till Maas