On 04/03/2012 01:25 AM, Richard Su wrote: > Hello, > > Here's a start of a feature outline. Contents of the wiki page is listed > below. > https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Dbomatic_testa... > > > > What I am unsure of is which part of dbomatic performance needs > addressing? From my limited experience, I have seen potential issues > with RHEV and the speed in which it starts instances. But that may have > been specific to the environment I was using. > > If you have experienced issues with dbomatic, this would be a great time > to get them out in the open so that we may include them in this > refactoring of dbomatic. > > Thanks. > > ---- > > # Summary > > Dbomatic is currently responsible for updating instance statuses and ip > addresses and updating realms. It also starts freshly created RHEV > instances. > > We do not have any test coverage for dbomatic and it is written in a way > that makes testing it difficult. We would like to refactor dbomatic to > make it more readable and in a way that allows us to plug in tests. > > As part of the refactoring, we should also explore ways to improve its > performance. For example, there is currently a 30 second time window > allocated for each provider account where dbomatic communicates with > deltacloud to perform status update, launch instances, etc. This 30 > second window may be too short especially if dbomatic is asked to launch > a large number of RHEV instances. > > # Owner > > Richard Su (rwsu) > and others who have interest > > # Current status > > Planning phase > > # Screencast Demo > > Run dbomatic tests. > Demonstrate dbomatic can handle status updates and launches where they > are a large number of instances. This can be through a unit test. > > # Implementation tasks > > This feature is being tracked at task #3094. > > Tests to add > > * State change reported by deltacloud is then reported in dbomatic > * Verify extraction and population of public and private ip addresses > * For RHEV, verify instance that is in STOPPED state and had previously > not been in running state is then started > * Realms refresh > * When provider is unavailable, dbomatic stays up and doesn't fail > and exit > > Performance related tasks > > * Dbomatic should not timeout if is asked to launch a large number of > RHEV instances. Mock out a delay to simulate RHEV instance starts. > Eliminate the 30 second deltacloud timeouts. > * Profile RHEV instance launch performance from dbomatic, and optimize > code where needed. > > # Open Questions > > * Which part of dbomatic performance are we worried about? Sooner today I proposed in thread "RFC feature planning - robust instance launching" to replace dbomatic with a background job tool which supports both delayed and recurring jobs: - delayed jobs would be used for launching instances - recurring jobs would be used for all stuff done by dbomatic The code in "check_one_account" method could be part of ProviderAccount model, so it can be covered by common rspec tests we have. The rest of script is just about parsing params, and forking processes in an interval - we get all this logic for free when using a bg job tool. Jan

On 04/03/2012 06:52 PM, Richard Su wrote: >On 04/03/2012 07:13 AM, Jan Provaznik wrote: >>On 04/03/2012 01:25 AM, Richard Su wrote: >>>Hello, >>> >>>Here's a start of a feature outline. Contents of the wiki page >>>is listed >>>below. >>>https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Dbomatic_testability_and_performance >>> >>> >>> >>>What I am unsure of is which part of dbomatic performance needs >>>addressing? From my limited experience, I have seen potential issues >>>with RHEV and the speed in which it starts instances. But that may have >>>been specific to the environment I was using. >>> >>>If you have experienced issues with dbomatic, this would be a >>>great time >>>to get them out in the open so that we may include them in this >>>refactoring of dbomatic. >>> >>>Thanks. >>> >>>---- >>> >>># Summary >>> >>>Dbomatic is currently responsible for updating instance statuses and ip >>>addresses and updating realms. It also starts freshly created RHEV >>>instances. >>> >>>We do not have any test coverage for dbomatic and it is written >>>in a way >>>that makes testing it difficult. We would like to refactor dbomatic to >>>make it more readable and in a way that allows us to plug in tests. >>> >>>As part of the refactoring, we should also explore ways to improve its >>>performance. For example, there is currently a 30 second time window >>>allocated for each provider account where dbomatic communicates with >>>deltacloud to perform status update, launch instances, etc. This 30 >>>second window may be too short especially if dbomatic is asked >>>to launch >>>a large number of RHEV instances. >>> >>># Owner >>> >>>Richard Su (rwsu) >>>and others who have interest >>> >>># Current status >>> >>>Planning phase >>> >>># Screencast Demo >>> >>>Run dbomatic tests. >>>Demonstrate dbomatic can handle status updates and launches where they >>>are a large number of instances. This can be through a unit test. >>> >>># Implementation tasks >>> >>>This feature is being tracked at task #3094. >>> >>>Tests to add >>> >>>* State change reported by deltacloud is then reported in dbomatic >>>* Verify extraction and population of public and private ip addresses >>>* For RHEV, verify instance that is in STOPPED state and had previously >>>not been in running state is then started >>>* Realms refresh >>>* When provider is unavailable, dbomatic stays up and doesn't >>>fail and exit >>> >>>Performance related tasks >>> >>>* Dbomatic should not timeout if is asked to launch a large number of >>>RHEV instances. Mock out a delay to simulate RHEV instance starts. >>>Eliminate the 30 second deltacloud timeouts. >>>* Profile RHEV instance launch performance from dbomatic, and optimize >>>code where needed. >>> >>># Open Questions >>> >>>* Which part of dbomatic performance are we worried about? >> >> >>Sooner today I proposed in thread "RFC feature planning - robust >>instance launching" to replace dbomatic with a background job tool >>which supports both delayed and recurring jobs: >>- delayed jobs would be used for launching instances >>- recurring jobs would be used for all stuff done by dbomatic >> >>The code in "check_one_account" method could be part of >>ProviderAccount model, so it can be covered by common rspec tests >>we have. >>The rest of script is just about parsing params, and forking >>processes in an interval - we get all this logic for free when >>using a bg job tool. >> >>Jan > >Thanks for the feedback Jan. I like the idea of replacing dbomatic >with a background job tool. Placing a task on a queue to launch an >instance is much better approach to having dbomatic poll in the >background. It also eliminates the dbomatic timeout issues when >launching a large number of instances. > >As for the instance status checks, do we need recurring jobs at all? >I wonder if we can simply create a delayed_job that runs and then >queues itself again upon completion. > > Would requiring all recurring jobs to re-scedule the next iteration be less reliable than making sure the job scheduler component can handle recurring jobs natively? We could get around that a bit by making sure wrap the whole job impl with a "rescue everything" block which queues the next iteration no matter what. What do other rails-based projects do about recurring background jobs? Scott

>>>> >>>> Sooner today I proposed in thread "RFC feature planning - robust >>>> instance launching" to replace dbomatic with a background job tool >>>> which supports both delayed and recurring jobs: >>>> - delayed jobs would be used for launching instances >>>> - recurring jobs would be used for all stuff done by dbomatic >>>> >>>> The code in "check_one_account" method could be part of >>>> ProviderAccount model, so it can be covered by common rspec tests >>>> we have. >>>> The rest of script is just about parsing params, and forking >>>> processes in an interval - we get all this logic for free when >>>> using a bg job tool. >>>> >>>> Jan >>> >>> Thanks for the feedback Jan. I like the idea of replacing dbomatic >>> with a background job tool. Placing a task on a queue to launch an >>> instance is much better approach to having dbomatic poll in the >>> background. It also eliminates the dbomatic timeout issues when >>> launching a large number of instances. >>>

>>> As for the instance status checks, do we need recurring jobs at all? >>> I wonder if we can simply create a delayed_job that runs and then >>> queues itself again upon completion. >>> >>> >> Would requiring all recurring jobs to re-scedule the next iteration >> be less reliable than making sure the job scheduler component can >> handle recurring jobs natively? We could get around that a bit by >> making sure wrap the whole job impl with a "rescue everything" block >> which queues the next iteration no matter what. What do other >> rails-based projects do about recurring background jobs? >> >> Scott >>

> > I have to say, I don't immediately like the idea of having the job > need to reschedule itself. It feels error-prone and tightly coupled to > me. Agreed, if a task gets interrupted and lost (someone pulls the plug on a machine) it will be so forever.

> I also strongly feel a full task/feature/whatever is needed here > to: > * list out what we would actually use such a daemon for, intial > candidates off the top of my head include everything currently in > dbomatic plus ldap user/group syncing, though I am sure there are > others.

Did we ever take care of handling instance actions that get invoked outside the scope of Aeolus? Eg if an instance is started on a cloud provider can we detect it? Can see an external tool (eg outside of conductor, such as dbomatic) being useful in this sort of situation. -Mo