Greetings.
I thought I would throw out some plans for hosted moving forward and see if we could hash out a plan and some timetable for implementing things.
== Current status/setup/problems ==
Hosted is 2 machines. One primary (hosted01) and one spare (hosted02). There is an hourly rsync of data from 01->02. 02 is completely standby, it takes none of the load or requests, it's simply there in case hosted01 dies.
Hosted01 has the following items on it:
* apache/httpd gitweb other scm web trac mailman archive http access source downloads (tar.gz, etc) loggerhead reviewboard
* rsync * scm checkouts * scm checkins * lists.fedorahosted.org mailman * spamassassin for lists.
Issues with current setup:
* hosted01 is under heavy load much of the time. Needs to be more responsive. * hosted01 is rhel5 and very old trac. rhel6 has a much newer trac. * If hosted01 dies we could loose up to 1 hour of data with switching to hosted02. * hosted01/02 are at serverbeach. If they are off line, hosted is off line.
=== Short term plan ===
* Create a hosted-lists01/02 (or whatever we want to call them) sync all lists and list data to it. make sure dns is happy on it. Move all lists.fedorahosted.org over to these new machines.
This will move some of the load off the existing hosted01/02 setup, and should be a pretty easy thing to do. We will just setup them like collab01/02 and hosted01/02 are with a live machine and hot spare (unless we can use some shared fs somehow?)
Questions:
- Should these be at serverbeach? Or one there and one somewhere else? (There is approx 23GB of mailman archives on hosted01 currently).
* Create a hosted03/04 Setup rhel6 versions of hosted01/02 Setup in similar way, except with no lists (see above).
Ask hosted project owners to 'opt-in' to moving to the new machines. Get a small pool of things at first, then move more as any issues are fixed. Once everything looks stable, announce a cut over date, and just move all the rest over then.
Questions: - Should these be at serverbeach? Or one there and one somewhere else? (There is approx 90GB of data on hosted01 currently).
* Look at some caching. Could we add memcached on hosted03/04 easily? What about varnish?
Questions: - What of the things we have would use caching well?
=== Long term plan ===
We kicked around some plans here in several of our meetings, but did a poor job of recording that anywhere. ;)
Some things we talked about:
* A caching frontend of some kind. Could cache web requests and take load off the main machines. * Some way to distribute the data, so if serverbeach were off line we could still switch to and use another machine. * Adding more/different services. IRC commit bot redmine (still not in fedora, but people are working on it) your idea here.
So, what do folks think? Should we try and come up with more concrete longer term plans before acting on the short term ones? Are there other paths forward that make sense here?
kevin
On Mon, 8 Aug 2011, Kevin Fenzi wrote:
Greetings.
I thought I would throw out some plans for hosted moving forward and see if we could hash out a plan and some timetable for implementing things.
== Current status/setup/problems ==
Hosted is 2 machines. One primary (hosted01) and one spare (hosted02). There is an hourly rsync of data from 01->02. 02 is completely standby, it takes none of the load or requests, it's simply there in case hosted01 dies.
Hosted01 has the following items on it:
apache/httpd gitweb other scm web trac mailman archive http access source downloads (tar.gz, etc) loggerhead reviewboard
rsync
scm checkouts
scm checkins
lists.fedorahosted.org mailman
spamassassin for lists.
Issues with current setup:
- hosted01 is under heavy load much of the time. Needs to be more responsive.
- hosted01 is rhel5 and very old trac. rhel6 has a much newer trac.
- If hosted01 dies we could loose up to 1 hour of data with switching to hosted02.
- hosted01/02 are at serverbeach. If they are off line, hosted is off line.
=== Short term plan ===
- Create a hosted-lists01/02 (or whatever we want to call them) sync all lists and list data to it. make sure dns is happy on it. Move all lists.fedorahosted.org over to these new machines.
This will move some of the load off the existing hosted01/02 setup, and should be a pretty easy thing to do. We will just setup them like collab01/02 and hosted01/02 are with a live machine and hot spare (unless we can use some shared fs somehow?)
Questions:
- Should these be at serverbeach? Or one there and one somewhere else?
(There is approx 23GB of mailman archives on hosted01 currently).
- Create a hosted03/04 Setup rhel6 versions of hosted01/02 Setup in similar way, except with no lists (see above).
Ask hosted project owners to 'opt-in' to moving to the new machines. Get a small pool of things at first, then move more as any issues are fixed. Once everything looks stable, announce a cut over date, and just move all the rest over then.
Questions:
- Should these be at serverbeach? Or one there and one somewhere else?
(There is approx 90GB of data on hosted01 currently).
- Look at some caching. Could we add memcached on hosted03/04 easily?
What about varnish?
My understanding of the load issues has always been that the trac git plugin is very inefficient. It'd probably be worth it to see if the load problems magically vanish by updating trac and the git plugin before committing to any archtectural changes related to performance.
-Mike
On Mon, 8 Aug 2011 12:01:33 -0500 (CDT) Mike McGrath mmcgrath@redhat.com wrote:
My understanding of the load issues has always been that the trac git plugin is very inefficient. It'd probably be worth it to see if the load problems magically vanish by updating trac and the git plugin before committing to any archtectural changes related to performance.
Yeah, that could well be the case. I know it leaves around defunct processes all the time. ;)
However, I looking today, the issue seems to be I/O.
There's a mailman archive process: mailman 1933 17.8 1.3 98980 92432 ? R Aug01 1821:31 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s and the rsync to hosted02: root 5955 4.9 0.4 63772 34224 ? Ds 15:53 3:53 rsync --daemon
sucking up most of the I/O. httpd is trying to squeeze in there. ;)
(As a side note, we should get apache-status working on there, might tell us more).
kevin
On Mon, 8 Aug 2011, Kevin Fenzi wrote:
On Mon, 8 Aug 2011 12:01:33 -0500 (CDT) Mike McGrath mmcgrath@redhat.com wrote:
My understanding of the load issues has always been that the trac git plugin is very inefficient. It'd probably be worth it to see if the load problems magically vanish by updating trac and the git plugin before committing to any archtectural changes related to performance.
Yeah, that could well be the case. I know it leaves around defunct processes all the time. ;)
However, I looking today, the issue seems to be I/O.
There's a mailman archive process: mailman 1933 17.8 1.3 98980 92432 ? R Aug01 1821:31 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s and the rsync to hosted02: root 5955 4.9 0.4 63772 34224 ? Ds 15:53 3:53 rsync --daemon
sucking up most of the I/O. httpd is trying to squeeze in there. ;)
(As a side note, we should get apache-status working on there, might tell us more).
https://admin.fedoraproject.org/status/hosted01
We've got that. The IO is what I saw with git as well. It doesn't properly cache so all page loads have to craw the related git repo. Compounding this issue is web crawlers but I think we have/had something in place to help limit this.
-Mike
On Mon, 8 Aug 2011 14:30:15 -0500 (CDT) Mike McGrath mmcgrath@redhat.com wrote:
Cool. ;)
We've got that. The IO is what I saw with git as well. It doesn't properly cache so all page loads have to craw the related git repo.
Yeah. :(
Compounding this issue is web crawlers but I think we have/had something in place to help limit this.
Yeah, there's some robots.txt in place I think now, which has mitigated this somewhat.
kevin
On Aug 8, 2011, at 10:01, Mike McGrath mmcgrath@redhat.com wrote:
On Mon, 8 Aug 2011, Kevin Fenzi wrote:
Greetings.
I thought I would throw out some plans for hosted moving forward and see if we could hash out a plan and some timetable for implementing things.
== Current status/setup/problems ==
Hosted is 2 machines. One primary (hosted01) and one spare (hosted02). There is an hourly rsync of data from 01->02. 02 is completely standby, it takes none of the load or requests, it's simply there in case hosted01 dies.
Hosted01 has the following items on it:
apache/httpd gitweb other scm web trac mailman archive http access source downloads (tar.gz, etc) loggerhead reviewboard
rsync
scm checkouts
scm checkins
lists.fedorahosted.org mailman
spamassassin for lists.
Issues with current setup:
- hosted01 is under heavy load much of the time. Needs to be more
responsive.
- hosted01 is rhel5 and very old trac. rhel6 has a much newer trac.
- If hosted01 dies we could loose up to 1 hour of data with switching
to hosted02.
- hosted01/02 are at serverbeach. If they are off line, hosted is off
line.
=== Short term plan ===
- Create a hosted-lists01/02 (or whatever we want to call them) sync all lists and list data to it. make sure dns is happy on it. Move all lists.fedorahosted.org over to these new machines.
This will move some of the load off the existing hosted01/02 setup, and should be a pretty easy thing to do. We will just setup them like collab01/02 and hosted01/02 are with a live machine and hot spare (unless we can use some shared fs somehow?)
Questions:
- Should these be at serverbeach? Or one there and one somewhere else?
(There is approx 23GB of mailman archives on hosted01 currently).
- Create a hosted03/04 Setup rhel6 versions of hosted01/02 Setup in similar way, except with no lists (see above).
Ask hosted project owners to 'opt-in' to moving to the new machines. Get a small pool of things at first, then move more as any issues are fixed. Once everything looks stable, announce a cut over date, and just move all the rest over then.
Questions:
- Should these be at serverbeach? Or one there and one somewhere else?
(There is approx 90GB of data on hosted01 currently).
- Look at some caching. Could we add memcached on hosted03/04 easily?
What about varnish?
My understanding of the load issues has always been that the trac git plugin is very inefficient. It'd probably be worth it to see if the load problems magically vanish by updating trac and the git plugin before committing to any archtectural changes related to performance.
-Mike
In my experience the git plugin has been the major performance hit on memory strapped systems, even with the devel branches. Just throwing this out there but getting people to use gitweb over trac browser might help performance although is not ideal.
--Brennan Ashton
On Mon, 2011-08-08 at 10:54 -0600, Kevin Fenzi wrote:
Hosted01 has the following items on it:
- apache/httpd gitweb other scm web trac mailman archive http access source downloads (tar.gz, etc) loggerhead reviewboard
I've been talking with Mike McGrath about ReviewBoard. I really want to get ReviewBoard off of Hosted, as the performance is incredibly poor and the FAS integration causes problems that I have not had a chance to identify.
Furthermore, starting with ReviewBoard 1.5.6 (being released upstream soon), I've submitted patches to make it possible to use ReviewBoard against any FedoraHosted git repository remotely. The main reason that ReviewBoard was located on FedoraHosted to begin with is because it needed direct access to the git repositories. So I'd like to move ReviewBoard to one of the app servers or into an OpenShift instance.
Of course, we still have issues regarding the FAS integration. For reasons I've still not been able to nail down, it causes us to lose access to the server. I was hoping to switch over to using OpenID with the release of ReviewBoard 1.6, but unfortunately they've deferred that feature until 1.7.
So I'm proposing the following options:
1) Move our existing ReviewBoard instance to one of the app servers. This will significantly improve the performance and responsiveness, but we'll still have no email notification support (due to as-yet-unknown negative interaction with FAS integration) 2) Move ReviewBoard to an app server and drop integration with FAS and allow standard enrollment for users, be they Fedora users or not. This will solve the performance and email issues, but results in a server running on Fedora systems that is not using Fedora accounts. Also I'm not sure we can maintain the existing review histories for the few projects currently using the system. 3) Turn ReviewBoard into a turnkey OpenShift virtual instance and allow any Fedora Hosted project to spin one up. This instance would use standard enrollment (rather than FAS integration, which is impossible outside the Infra firewall). Each project could have its own complete instance to maintain on its own. Upsides: less work for Fedora Admins, support for email and better performance. Downsides: no centrally-managed user accounts and projects need to do more of the maintaining of the system themselves.
I'm all ears for a fourth (or fifth...) option.
On Mon, 8 Aug 2011, Stephen Gallagher wrote:
On Mon, 2011-08-08 at 10:54 -0600, Kevin Fenzi wrote:
Hosted01 has the following items on it:
- apache/httpd gitweb other scm web trac mailman archive http access source downloads (tar.gz, etc) loggerhead reviewboard
I've been talking with Mike McGrath about ReviewBoard. I really want to get ReviewBoard off of Hosted, as the performance is incredibly poor and the FAS integration causes problems that I have not had a chance to identify.
Furthermore, starting with ReviewBoard 1.5.6 (being released upstream soon), I've submitted patches to make it possible to use ReviewBoard against any FedoraHosted git repository remotely. The main reason that ReviewBoard was located on FedoraHosted to begin with is because it needed direct access to the git repositories. So I'd like to move ReviewBoard to one of the app servers or into an OpenShift instance.
Of course, we still have issues regarding the FAS integration. For reasons I've still not been able to nail down, it causes us to lose access to the server. I was hoping to switch over to using OpenID with the release of ReviewBoard 1.6, but unfortunately they've deferred that feature until 1.7.
So I'm proposing the following options:
- Move our existing ReviewBoard instance to one of the app servers.
This will significantly improve the performance and responsiveness, but we'll still have no email notification support (due to as-yet-unknown negative interaction with FAS integration) 2) Move ReviewBoard to an app server and drop integration with FAS and allow standard enrollment for users, be they Fedora users or not. This will solve the performance and email issues, but results in a server running on Fedora systems that is not using Fedora accounts. Also I'm not sure we can maintain the existing review histories for the few projects currently using the system. 3) Turn ReviewBoard into a turnkey OpenShift virtual instance and allow any Fedora Hosted project to spin one up. This instance would use standard enrollment (rather than FAS integration, which is impossible outside the Infra firewall). Each project could have its own complete instance to maintain on its own. Upsides: less work for Fedora Admins, support for email and better performance. Downsides: no centrally-managed user accounts and projects need to do more of the maintaining of the system themselves.
FYI, just in the last hour or so we've got full support for this in OpenShift. Here's a howto on running reviewboard:
https://github.com/openshift/reviewboard-example
-Mike
On Mon, 08 Aug 2011 14:32:25 -0400 Stephen Gallagher sgallagh@redhat.com wrote:
...snip...
So I'm proposing the following options:
- Move our existing ReviewBoard instance to one of the app servers.
This will significantly improve the performance and responsiveness, but we'll still have no email notification support (due to as-yet-unknown negative interaction with FAS integration)
Yeah, that doesn't seem ideal. ;(
- Move ReviewBoard to an app server and drop integration with FAS and
allow standard enrollment for users, be they Fedora users or not. This will solve the performance and email issues, but results in a server running on Fedora systems that is not using Fedora accounts. Also I'm not sure we can maintain the existing review histories for the few projects currently using the system.
Ditto.
- Turn ReviewBoard into a turnkey OpenShift virtual instance and
allow any Fedora Hosted project to spin one up. This instance would use standard enrollment (rather than FAS integration, which is impossible outside the Infra firewall). Each project could have its own complete instance to maintain on its own. Upsides: less work for Fedora Admins, support for email and better performance. Downsides: no centrally-managed user accounts and projects need to do more of the maintaining of the system themselves.
This is pretty interesting... I assume after following the steps they would have a persistent instance they could use moving forward. It doesn't need anything special to talk to their project on hosted? Does it end up costing the end project anything? ;)
What happens if someone sets up an instance and then disappears? Does the project have any way to deal with that? Or just make a new one?
Would someone be interested in trying this out and seeing how well it actually works? Is there a project or two that are really wanting to use reviewboard that we could ask?
I'm all ears for a fourth (or fifth...) option.
Well, there's https://www.rbcommons.com/plans/ but thats a cost/month. (Which might be worth it for some projects).
kevin
On Wed, 2011-08-10 at 09:15 -0600, Kevin Fenzi wrote:
- Turn ReviewBoard into a turnkey OpenShift virtual instance and
allow any Fedora Hosted project to spin one up. This instance would use standard enrollment (rather than FAS integration, which is impossible outside the Infra firewall). Each project could have its own complete instance to maintain on its own. Upsides: less work for Fedora Admins, support for email and better performance. Downsides: no centrally-managed user accounts and projects need to do more of the maintaining of the system themselves.
This is pretty interesting... I assume after following the steps they would have a persistent instance they could use moving forward. It doesn't need anything special to talk to their project on hosted? Does it end up costing the end project anything? ;)
I've submitted a patch upstream to ReviewBoard to add easy configuration of Fedora Hosted source repositories: http://reviews.reviewboard.org/r/2505/
I have confirmation from Christian Hammond (the upstream project lead) that it will be included in the 1.5.6 and 1.6.0 releases (aka imminently).
So there's very little that the projects need to do in order to connect to the hosted repo. As I said above, they lose the centrally-managed users available to FAS, so they'd need to manage their own groups themselves. On the other side, this does mean that they gain much finer control over permissions (since they can define their own project-specific groups rather than relying on FAS groups).
What happens if someone sets up an instance and then disappears? Does the project have any way to deal with that? Or just make a new one?
That's a good question for Mike McGrath. I suspect that it would be prudent to recommend that projects set up several administrators so that a disappearance of one doesn't result in the loss of all administrators.
Also, it's possible to promote a user to admin status if you have database privileges as well by setting the admin flag on their user account, but of course that assumes you have access to a DB admin.
A final option would be to modify the openshift instance to always install a recovery admin with a random password that was escrowed by the Fedora project, but I'm not sure whether that's realistic. Mike, can you speak to that?
Would someone be interested in trying this out and seeing how well it actually works? Is there a project or two that are really wanting to use reviewboard that we could ask?
Well, AutoQA has been using the FedoraHosted instance (without email support and poor performance) very heavily. They might be interested.
I would also be interested in piloting it for the SSSD and FreeIPA projects (for which I originally got invested in setting this up).
I'm all ears for a fourth (or fifth...) option.
Well, there's https://www.rbcommons.com/plans/ but thats a cost/month. (Which might be worth it for some projects).
True, but as we already have the OpenShift option (thanks Mike!), I think it's worth looking into making this part of our Hosted offering.
On Wed, 10 Aug 2011, Stephen Gallagher wrote:
On Wed, 2011-08-10 at 09:15 -0600, Kevin Fenzi wrote:
- Turn ReviewBoard into a turnkey OpenShift virtual instance and
allow any Fedora Hosted project to spin one up. This instance would use standard enrollment (rather than FAS integration, which is impossible outside the Infra firewall). Each project could have its own complete instance to maintain on its own. Upsides: less work for Fedora Admins, support for email and better performance. Downsides: no centrally-managed user accounts and projects need to do more of the maintaining of the system themselves.
This is pretty interesting... I assume after following the steps they would have a persistent instance they could use moving forward. It doesn't need anything special to talk to their project on hosted? Does it end up costing the end project anything? ;)
I've submitted a patch upstream to ReviewBoard to add easy configuration of Fedora Hosted source repositories: http://reviews.reviewboard.org/r/2505/
I have confirmation from Christian Hammond (the upstream project lead) that it will be included in the 1.5.6 and 1.6.0 releases (aka imminently).
So there's very little that the projects need to do in order to connect to the hosted repo. As I said above, they lose the centrally-managed users available to FAS, so they'd need to manage their own groups themselves. On the other side, this does mean that they gain much finer control over permissions (since they can define their own project-specific groups rather than relying on FAS groups).
What happens if someone sets up an instance and then disappears? Does the project have any way to deal with that? Or just make a new one?
That's a good question for Mike McGrath. I suspect that it would be prudent to recommend that projects set up several administrators so that a disappearance of one doesn't result in the loss of all administrators.
Also, it's possible to promote a user to admin status if you have database privileges as well by setting the admin flag on their user account, but of course that assumes you have access to a DB admin.
A final option would be to modify the openshift instance to always install a recovery admin with a random password that was escrowed by the Fedora project, but I'm not sure whether that's realistic. Mike, can you speak to that?
We should have multiple ssh key support soon to allow multiple admins of a single project. Having said that the apps are tied to an individual user account. At the moment if someone wanted to move it they'd have to make a snapshot and restore it elsewhere (if for some reason the app needed to change owners or something)
To do the "fedora project recovery" bits, we'd have to setup the fedora project as an intermediary (basically running everything as a fedora user). The rest could all be scripted fairly easily. At the moment though there's a limit of 5 apps per account so the best bet is to have users setup their own for the time being.
-Mike
On Monday, August 08, 2011 11:54:30 AM Kevin Fenzi wrote:
Greetings.
I thought I would throw out some plans for hosted moving forward and see if we could hash out a plan and some timetable for implementing things.
== Current status/setup/problems ==
Hosted is 2 machines. One primary (hosted01) and one spare (hosted02). There is an hourly rsync of data from 01->02. 02 is completely standby, it takes none of the load or requests, it's simply there in case hosted01 dies.
Hosted01 has the following items on it:
apache/httpd gitweb other scm web trac mailman archive http access source downloads (tar.gz, etc) loggerhead reviewboard
rsync
scm checkouts
scm checkins
lists.fedorahosted.org mailman
spamassassin for lists.
Issues with current setup:
- hosted01 is under heavy load much of the time. Needs to be more responsive.
- hosted01 is rhel5 and very old trac. rhel6 has a much newer trac.
- If hosted01 dies we could loose up to 1 hour of data with switching to hosted02.
- hosted01/02 are at serverbeach. If they are off line, hosted is off line.
=== Short term plan ===
- Create a hosted-lists01/02 (or whatever we want to call them) sync all lists and list data to it. make sure dns is happy on it. Move all lists.fedorahosted.org over to these new machines.
This will move some of the load off the existing hosted01/02 setup, and should be a pretty easy thing to do. We will just setup them like collab01/02 and hosted01/02 are with a live machine and hot spare (unless we can use some shared fs somehow?)
Questions:
- Should these be at serverbeach? Or one there and one somewhere else?
(There is approx 23GB of mailman archives on hosted01 currently).
- Create a hosted03/04 Setup rhel6 versions of hosted01/02 Setup in similar way, except with no lists (see above).
Ask hosted project owners to 'opt-in' to moving to the new machines. Get a small pool of things at first, then move more as any issues are fixed. Once everything looks stable, announce a cut over date, and just move all the rest over then.
Questions:
- Should these be at serverbeach? Or one there and one somewhere else?
(There is approx 90GB of data on hosted01 currently).
- Look at some caching. Could we add memcached on hosted03/04 easily?
What about varnish?
Questions:
- What of the things we have would use caching well?
=== Long term plan ===
We kicked around some plans here in several of our meetings, but did a poor job of recording that anywhere. ;)
Some things we talked about:
- A caching frontend of some kind. Could cache web requests and take load off the main machines.
- Some way to distribute the data, so if serverbeach were off line we could still switch to and use another machine.
- Adding more/different services. IRC commit bot redmine (still not in fedora, but people are working on it) your idea here.
So, what do folks think? Should we try and come up with more concrete longer term plans before acting on the short term ones? Are there other paths forward that make sense here?
kevin
another thing ive wanted to test and look into is moving to using something like glusterfs as the backend storage so that we can scale beyond more than one box without users needing to know what box there proect is stored on.
Dennis
On Mon, 2011-08-08 at 15:23 -0500, Dennis Gilmore wrote:
another thing ive wanted to test and look into is moving to using something like glusterfs as the backend storage so that we can scale beyond more than one box without users needing to know what box there proect is stored on.
I looked at glusterfs and cloudfs(now hekafs) for this and the answer is 'maybe' if we're on a lan - not on a wan and 'maybe' if we can cope with the groups/acls/selinux limitations and/or wait for patches to be upstreamed.
-sv
On Mon, 2011-08-08 at 10:54 -0600, Kevin Fenzi wrote:
Greetings.
I thought I would throw out some plans for hosted moving forward and see if we could hash out a plan and some timetable for implementing things.
Some things we talked about:
- A caching frontend of some kind. Could cache web requests and take load off the main machines.
- Some way to distribute the data, so if serverbeach were off line we could still switch to and use another machine.
- Adding more/different services. IRC commit bot redmine (still not in fedora, but people are working on it) your idea here.
We talked a bit about this yesterday and I wanted to write it up here:
1. move the mailman instance over to collab1 and have them share an infrastructure - mailman is not (usually) easy to spread to lots of servers in a very clean way and it is not (usually) the source of a massive load (the archiver not withstanding)
2. I think we should seriously consider building things for hosted 2.0 such that slices are possible:
you still go to fedorahosted.org/projectname
but it redirects you to project-##.fedorahosted.org - based on the first 2 letters of the name of your project or what not.
for commits you have to commit to: git-##.fedorahosted.org (for example) or svn-##.fedorahosted.org
it would allow us to expand out horizontally as things get busier.
It would require a config change and/or a flag day for our users but it's not the end of the world now. Advantages: a. this keeps all of our eggs out of one basket b. it means replicating the infrastructure is important, repeatedly :)
it seems like an obvious course of action that scales well for what we need.
3. Think HARD about limiting support for additional services on hosted.
thoughts?
-sv
On Wed, 2011-08-17 at 14:24 -0400, seth vidal wrote:
We talked a bit about this yesterday and I wanted to write it up here:
- move the mailman instance over to collab1 and have them share an
infrastructure - mailman is not (usually) easy to spread to lots of servers in a very clean way and it is not (usually) the source of a massive load (the archiver not withstanding)
- I think we should seriously consider building things for hosted 2.0
such that slices are possible:
you still go to fedorahosted.org/projectname
but it redirects you to project-##.fedorahosted.org - based on the first 2 letters of the name of your project or what not.
for commits you have to commit to: git-##.fedorahosted.org (for example) or svn-##.fedorahosted.org
Actually - on further thought just making it work for:
git-$projectname.fedorahosted.org (for example)
should work just fine and be no more effort - just an CNAME on the backend.
Easier to document, too
-sv
On Wed, Aug 17, 2011 at 12:24, seth vidal skvidal@fedoraproject.org wrote:
On Mon, 2011-08-08 at 10:54 -0600, Kevin Fenzi wrote:
Greetings.
I thought I would throw out some plans for hosted moving forward and see if we could hash out a plan and some timetable for implementing things.
Some things we talked about:
- A caching frontend of some kind. Could cache web requests and take
load off the main machines.
- Some way to distribute the data, so if serverbeach were off line we
could still switch to and use another machine.
- Adding more/different services.
IRC commit bot redmine (still not in fedora, but people are working on it) your idea here.
We talked a bit about this yesterday and I wanted to write it up here:
- move the mailman instance over to collab1 and have them share an
infrastructure - mailman is not (usually) easy to spread to lots of servers in a very clean way and it is not (usually) the source of a massive load (the archiver not withstanding)
- I think we should seriously consider building things for hosted 2.0
such that slices are possible:
you still go to fedorahosted.org/projectname
but it redirects you to project-##.fedorahosted.org - based on the first 2 letters of the name of your project or what not.
Just an idea to bring up when we implement:
Due to the fact that we would have huge groupings on certain letters ( re, fe, li py, sy) use the first 2 letters of the md5sum of the name. Hopefully that would spread out the load across enough. The project build script would automate this so people don't have to figure it out.
I like the below.
for commits you have to commit to: git-##.fedorahosted.org (for example) or svn-##.fedorahosted.org
it would allow us to expand out horizontally as things get busier.
It would require a config change and/or a flag day for our users but it's not the end of the world now. Advantages: a. this keeps all of our eggs out of one basket b. it means replicating the infrastructure is important, repeatedly :)
it seems like an obvious course of action that scales well for what we need.
- Think HARD about limiting support for additional services on hosted.
4. Talk to something like github if we can meet up and grow something togehter?
thoughts?
-sv
infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Wed, 2011-08-17 at 13:01 -0600, Stephen John Smoogen wrote:
Due to the fact that we would have huge groupings on certain letters ( re, fe, li py, sy) use the first 2 letters of the md5sum of the name. Hopefully that would spread out the load across enough. The project build script would automate this so people don't have to figure it out.
md5sum is too complicated - maintaining a few thousand cnames, though, isn't tough at all.
- Talk to something like github if we can meet up and grow something togehter?
I agree with jesse that github, being closed source, is a complete non-starter.
-sv
On Wed, Aug 17, 2011 at 13:53, seth vidal skvidal@fedoraproject.org wrote:
On Wed, 2011-08-17 at 13:01 -0600, Stephen John Smoogen wrote:
Due to the fact that we would have huge groupings on certain letters ( re, fe, li py, sy) use the first 2 letters of the md5sum of the name. Hopefully that would spread out the load across enough. The project build script would automate this so people don't have to figure it out.
md5sum is too complicated - maintaining a few thousand cnames, though, isn't tough at all.
- Talk to something like github if we can meet up and grow something togehter?
I agree with jesse that github, being closed source, is a complete non-starter.
Sorry I meant whichever one is the opensource one :) [Sorry, they are all just names without faces to me]. However if their code is not better than the problems we have with trac then I can move along.
-sv
infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Excerpts from seth vidal's message of Wed Aug 17 15:53:42 -0400 2011:
On Wed, 2011-08-17 at 13:01 -0600, Stephen John Smoogen wrote:
- Talk to something like github if we can meet up and grow something togehter?
I agree with jesse that github, being closed source, is a complete non-starter.
Allura, the code behind the shiny new SourceForge, is open source.
http://sourceforge.net/p/allura
Based on the many conversations I've had with them at PyCons, they would definitely be interested in working together with us on hosting.
luke
On Wed, Aug 17, 2011 at 15:49, Luke Macken lmacken@redhat.com wrote:
Excerpts from seth vidal's message of Wed Aug 17 15:53:42 -0400 2011:
On Wed, 2011-08-17 at 13:01 -0600, Stephen John Smoogen wrote:
- Talk to something like github if we can meet up and grow something togehter?
I agree with jesse that github, being closed source, is a complete non-starter.
Allura, the code behind the shiny new SourceForge, is open source.
http://sourceforge.net/p/allura
Based on the many conversations I've had with them at PyCons, they would definitely be interested in working together with us on hosting.
This does look interesting. I am going to see what it would take to get going.
luke _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Wed, 17 Aug 2011 13:01:42 -0600 Stephen John Smoogen smooge@gmail.com wrote:
On Wed, Aug 17, 2011 at 12:24, seth vidal skvidal@fedoraproject.org wrote:
On Mon, 2011-08-08 at 10:54 -0600, Kevin Fenzi wrote:
Greetings.
I thought I would throw out some plans for hosted moving forward and see if we could hash out a plan and some timetable for implementing things.
Some things we talked about:
- A caching frontend of some kind. Could cache web requests and
take load off the main machines.
- Some way to distribute the data, so if serverbeach were off line
we could still switch to and use another machine.
- Adding more/different services.
IRC commit bot redmine (still not in fedora, but people are working on it) your idea here.
We talked a bit about this yesterday and I wanted to write it up here:
- move the mailman instance over to collab1 and have them share an
infrastructure - mailman is not (usually) easy to spread to lots of servers in a very clean way and it is not (usually) the source of a massive load (the archiver not withstanding)
- I think we should seriously consider building things for hosted
2.0 such that slices are possible:
you still go to fedorahosted.org/projectname
but it redirects you to project-##.fedorahosted.org - based on the first 2 letters of the name of your project or what not.
Just an idea to bring up when we implement:
Due to the fact that we would have huge groupings on certain letters ( re, fe, li py, sy) use the first 2 letters of the md5sum of the name. Hopefully that would spread out the load across enough. The project build script would automate this so people don't have to figure it out.
How about just even more simple:
01 02 03
when 01 fills up/gets loaded, move to 02, or add start just cycling thru them...
kevin
On Thu, 2011-08-18 at 09:29 -0600, Kevin Fenzi wrote:
How about just even more simple:
01 02 03
when 01 fills up/gets loaded, move to 02, or add start just cycling thru them...
works for me - the only reason I suggested:
git-$projectname.fedorahosted.org
is so I don't have to constantly go lookup which numbered machine a project is on before I can go check something out.
-sv
On Thu, Aug 18, 2011 at 11:39:08AM -0400, seth vidal wrote:
On Thu, 2011-08-18 at 09:29 -0600, Kevin Fenzi wrote:
How about just even more simple:
01 02 03
when 01 fills up/gets loaded, move to 02, or add start just cycling thru them...
works for me - the only reason I suggested:
git-$projectname.fedorahosted.org
is so I don't have to constantly go lookup which numbered machine a project is on before I can go check something out.
+1 for cnames.
We get a lot of "Why can't I git pull foobar today? I was able to do that yesterday." If we can avoid having to lookup which machine is serving that it'll make things easier.
-Toshio
On Thu, 18 Aug 2011 09:18:06 -0700 Toshio Kuratomi a.badger@gmail.com wrote:
On Thu, Aug 18, 2011 at 11:39:08AM -0400, seth vidal wrote:
On Thu, 2011-08-18 at 09:29 -0600, Kevin Fenzi wrote:
How about just even more simple:
01 02 03
when 01 fills up/gets loaded, move to 02, or add start just cycling thru them...
works for me - the only reason I suggested:
git-$projectname.fedorahosted.org
is so I don't have to constantly go lookup which numbered machine a project is on before I can go check something out.
+1 for cnames.
We get a lot of "Why can't I git pull foobar today? I was able to do that yesterday." If we can avoid having to lookup which machine is serving that it'll make things easier.
Yeah, I am fine with that too. I made my 01/02/03 suggestion before I read down to the $scm-$projectname idea. Makes sense to me.
kevin
On Wed, 17 Aug 2011 14:24:12 -0400 seth vidal skvidal@fedoraproject.org wrote:
We talked a bit about this yesterday and I wanted to write it up here:
- move the mailman instance over to collab1 and have them share an
infrastructure - mailman is not (usually) easy to spread to lots of servers in a very clean way and it is not (usually) the source of a massive load (the archiver not withstanding)
Yeah, I think this will be a win... althought it means more eggs in one basket. ;)
- I think we should seriously consider building things for hosted 2.0
such that slices are possible:
you still go to fedorahosted.org/projectname
but it redirects you to project-##.fedorahosted.org - based on the first 2 letters of the name of your project or what not.
for commits you have to commit to: git-##.fedorahosted.org (for example) or svn-##.fedorahosted.org
it would allow us to expand out horizontally as things get busier.
It would require a config change and/or a flag day for our users but it's not the end of the world now. Advantages: a. this keeps all of our eggs out of one basket b. it means replicating the infrastructure is important, repeatedly :)
it seems like an obvious course of action that scales well for what we need.
Actually - on further thought just making it work for:
git-$projectname.fedorahosted.org (for example)
should work just fine and be no more effort - just an CNAME on the backend
Easier to document, too
Sounds reasonable. We can setup the new hosted machines to use this, and when we move over everything we convert them?
- Think HARD about limiting support for additional services on
hosted.
Sure, I think we should use the same methods for any request for resource. Ie, we need someone able to support it, it's got an upstream, it's in epel, we have sop's for it, etc.
thoughts?
-sv
kevin
On Mon, 2011-08-08 at 10:54 -0600, Kevin Fenzi wrote:
- Adding more/different services. IRC commit bot redmine (still not in fedora, but people are working on it)
If we move from python/trac I'd suggest that we have a look at indefero as well, being php it might be easier to deploy/integrate in our current infrastructure.
Pierre
On Thu, 18 Aug 2011 13:15:22 +0200 Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Mon, 2011-08-08 at 10:54 -0600, Kevin Fenzi wrote:
- Adding more/different services. IRC commit bot redmine (still not in fedora, but people are working on it)
If we move from python/trac I'd suggest that we have a look at indefero as well, being php it might be easier to deploy/integrate in our current infrastructure.
Yeah, currently redmine in a nonstarter due to not being in fedora, but I think some folks are working on that.
I'd think we would want to look over any replacement with the same eye we do to any new resources we need to support...
kevin
infrastructure@lists.fedoraproject.org
-
Adam M. Dutko -
Brennan Ashton -
Dennis Gilmore -
Jesse Keating -
Kevin Fenzi -
Luke Macken -
Mike McGrath -
Pierre-Yves Chibon -
Seth Vidal -
Stephen Gallagher -
Stephen John Smoogen -
Toshio Kuratomi