Greetings,
I've been doing some looking at versioning issues related to QMF schemas and wanted to share some notes to make sure that I understand correctly and that we're all on the same page. I will be creating a wiki page to capture this information, including related sample code.
On the agent side of things:
1) We can add new properties without affecting backwards compatibility.
2) We can add new methods without affecting backwards compatibility.
3) We can add new arguments to methods without affecting backwards compatibility, as long as the agent code is written in such a way that it does not _expect_ the new arguments to always be present.
4) Removing things should be avoided as much as possible once they hit a supported release.
On the console side:
1) Consoles must always be written against the same or older version of the schema as the agent. A console may not include arguments to a method that the agent does not know about.
2) If a console would like to operate against multiple versions of an agent that have different schemas, it can inspect the schema presented by the agent. To make this a bit easier, I have written some helper functions which you can find here:
https://github.com/russellb/matahari/blob/qmfutils/src/include/matahari/qmf_...
These functions make it easy to do some schema checks against a new agent. For example, let's say you get a CONSOLE_AGENT_ADD event notifying you that a new agent has shown up, you could then do checks against it like this:
qmf::Agent agent(event.getAgent());
// Does this agent have the hostname property? std::cout << "Has hostname: " << matahari::qmf::schema_has_property(agent, "org.matahariproject", "Host", "hostname") << std::endl;
// Does this agent have the get_uuid method? std::cout << "Has get_uuid method: " << matahari::qmf::schema_has_method(agent, "org.matahariproject", "Host", "get_uuid") << std::endl;
// Does this agent take the lifetime argument to the get_uuid method? std::cout << "Has lifetime argument to get_uuid_method: " << matahari::qmf::schema_has_property(agent, "org.matahariproject", "Host", "get_uuid", "lifetime") << std::endl;
I implemented this in the matahari code base just to flesh out the ideas. This functionality probably belongs as a part of the qmf::Agent class upstream.
Comments welcome. Thanks.
-- Russell Bryant
On Tue, Aug 16, 2011 at 04:02:57PM -0400, Russell Bryant wrote:
Greetings,
I've been doing some looking at versioning issues related to QMF schemas and wanted to share some notes to make sure that I understand correctly and that we're all on the same page. I will be creating a wiki page to capture this information, including related sample code.
On the agent side of things:
We can add new properties without affecting backwards compatibility.
We can add new methods without affecting backwards compatibility.
We can add new arguments to methods without affecting backwards
compatibility, as long as the agent code is written in such a way that it does not _expect_ the new arguments to always be present.
- Removing things should be avoided as much as possible once they hit
a supported release.
On the console side:
- Consoles must always be written against the same or older version
of the schema as the agent. A console may not include arguments to a method that the agent does not know about.
- If a console would like to operate against multiple versions of an
agent that have different schemas, it can inspect the schema presented by the agent. To make this a bit easier, I have written some helper functions which you can find here:
https://github.com/russellb/matahari/blob/qmfutils/src/include/matahari/qmf_utils.h
These functions make it easy to do some schema checks against a new agent. For example, let's say you get a CONSOLE_AGENT_ADD event notifying you that a new agent has shown up, you could then do checks against it like this:
qmf::Agent agent(event.getAgent()); // Does this agent have the hostname property? std::cout << "Has hostname: " <<
matahari::qmf::schema_has_property(agent, "org.matahariproject", "Host", "hostname") << std::endl;
// Does this agent have the get_uuid method? std::cout << "Has get_uuid method: " <<
matahari::qmf::schema_has_method(agent, "org.matahariproject", "Host", "get_uuid") << std::endl;
// Does this agent take the lifetime argument to the get_uuid method? std::cout << "Has lifetime argument to get_uuid_method: " <<
matahari::qmf::schema_has_property(agent, "org.matahariproject", "Host", "get_uuid", "lifetime") << std::endl;
You mean: schema_method_has_arg()
I implemented this in the matahari code base just to flesh out the ideas. This functionality probably belongs as a part of the qmf::Agent class upstream.
Comments welcome. Thanks.
-- Russell Bryant _______________________________________________ Matahari mailing list Matahari@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/matahari
On Tue, Aug 16, 2011 at 5:51 PM, Angus Salkeld asalkeld@redhat.com wrote:
On Tue, Aug 16, 2011 at 04:02:57PM -0400, Russell Bryant wrote:
// Does this agent take the lifetime argument to the get_uuid method? std::cout << "Has lifetime argument to get_uuid_method: " << matahari::qmf::schema_has_property(agent, "org.matahariproject", "Host", "get_uuid", "lifetime") << std::endl;
You mean: schema_method_has_arg()
Yes. Copy/paste fail.
On 16/08/11 22:02, Russell Bryant wrote:
Greetings,
I've been doing some looking at versioning issues related to QMF schemas and wanted to share some notes to make sure that I understand correctly and that we're all on the same page. I will be creating a wiki page to capture this information, including related sample code.
On the agent side of things:
We can add new properties without affecting backwards compatibility.
We can add new methods without affecting backwards compatibility.
We can add new arguments to methods without affecting backwards
compatibility, as long as the agent code is written in such a way that it does not _expect_ the new arguments to always be present.
- Removing things should be avoided as much as possible once they hit
a supported release.
On the console side:
- Consoles must always be written against the same or older version
of the schema as the agent. A console may not include arguments to a method that the agent does not know about.
- If a console would like to operate against multiple versions of an
agent that have different schemas, it can inspect the schema presented by the agent. To make this a bit easier, I have written some helper functions which you can find here:
This is all consistent with my understanding.
https://github.com/russellb/matahari/blob/qmfutils/src/include/matahari/qmf_utils.h
These functions make it easy to do some schema checks against a new agent. For example, let's say you get a CONSOLE_AGENT_ADD event notifying you that a new agent has shown up, you could then do checks against it like this:
qmf::Agent agent(event.getAgent()); // Does this agent have the hostname property? std::cout<< "Has hostname: "<<
matahari::qmf::schema_has_property(agent, "org.matahariproject", "Host", "hostname")<< std::endl;
// Does this agent have the get_uuid method? std::cout<< "Has get_uuid method: "<<
matahari::qmf::schema_has_method(agent, "org.matahariproject", "Host", "get_uuid")<< std::endl;
// Does this agent take the lifetime argument to the get_uuid method? std::cout<< "Has lifetime argument to get_uuid_method: "<<
matahari::qmf::schema_has_property(agent, "org.matahariproject", "Host", "get_uuid", "lifetime")<< std::endl;
I implemented this in the matahari code base just to flesh out the ideas. This functionality probably belongs as a part of the qmf::Agent class upstream.
I am totally on board with the principle that we need to make this kind of thing simpler for users. The QMF API is obtuse and not especially well documented (yet). For example, formal parameters in header files are not even named, and even in the private implementations they all have single-character names. So all you can tell in many cases is that a method takes two strings and an int as parameters. Try to guess what they mean! It's like some sort of cruel puzzle. You have to dive through another level of indirection to the private Impl code to even find out what the parameters are called, let alone what they mean. So we need to improve things through some combination of providing helpers in matahari and pushing improvements upstream.
Having said that, I think this patch is tackling too narrow a problem (existence of a particular schema/property/method/argument). The real issue here is that you can't look up a particular schema/property/method/argument without iterating through a list of them. This is unfortunate and bizarre, IMHO, since all of these things are primarily identified by name, not by order. When you do callMethod() you pass it the name of the method you want to call, not the order it appeared in the schema. It's even more unfortunate that it has been implemented in such a way that the consumer has to do all the iteration itself, since that places unnecessary limits on the internal representation... although looking at the code it's already gratuitously inefficient, so maybe that's not such an issue. A more C++ish way to implement this would have been by providing an iterator.
The good news, however, is that the Schema, SchemaMethod and SchemaProperty classes all provide operator bool() and operator!(), so if you just want to know about existence of something in the schema then
if (schema_has_property(agent, "org.matahariproject", "Host", "hostname")) { // i can has hostname }
is equivalent to
::qmf::Schema host(agent_get_schema_by_name(agent, "org.matahariproject", "Host")); if (host && schema_get_property_by_name(host, "hostname")) { // i can has hostname }
but the latter is more flexible: it gives the caller access to the actual Schema object itself and makes it easy to, e.g. lookup multiple properties in the same Schema object without having to copy & paste all of the parameters each time.
Obviously, as with your suggestions, the best place to ultimately implement this would be in the qmf::Agent, qmf::Schema and qmf::SchemaMethod classes.
(I also added some specific implementation suggestions on GitHub.)
Comments welcome. Thanks.
Ditto ;)
cheers, Zane.
On Wed, Aug 17, 2011 at 12:28:07PM +0200, Zane Bitter wrote:
On 16/08/11 22:02, Russell Bryant wrote:
Greetings,
I've been doing some looking at versioning issues related to QMF schemas and wanted to share some notes to make sure that I understand correctly and that we're all on the same page. I will be creating a wiki page to capture this information, including related sample code.
On the agent side of things:
We can add new properties without affecting backwards compatibility.
We can add new methods without affecting backwards compatibility.
We can add new arguments to methods without affecting backwards
compatibility, as long as the agent code is written in such a way that it does not _expect_ the new arguments to always be present.
- Removing things should be avoided as much as possible once they hit
a supported release.
On the console side:
- Consoles must always be written against the same or older version
of the schema as the agent. A console may not include arguments to a method that the agent does not know about.
- If a console would like to operate against multiple versions of an
agent that have different schemas, it can inspect the schema presented by the agent. To make this a bit easier, I have written some helper functions which you can find here:
This is all consistent with my understanding.
https://github.com/russellb/matahari/blob/qmfutils/src/include/matahari/qmf_utils.h
These functions make it easy to do some schema checks against a new agent. For example, let's say you get a CONSOLE_AGENT_ADD event notifying you that a new agent has shown up, you could then do checks against it like this:
qmf::Agent agent(event.getAgent()); // Does this agent have the hostname property? std::cout<< "Has hostname: "<<
matahari::qmf::schema_has_property(agent, "org.matahariproject", "Host", "hostname")<< std::endl;
// Does this agent have the get_uuid method? std::cout<< "Has get_uuid method: "<<
matahari::qmf::schema_has_method(agent, "org.matahariproject", "Host", "get_uuid")<< std::endl;
// Does this agent take the lifetime argument to the get_uuid method? std::cout<< "Has lifetime argument to get_uuid_method: "<<
matahari::qmf::schema_has_property(agent, "org.matahariproject", "Host", "get_uuid", "lifetime")<< std::endl;
I implemented this in the matahari code base just to flesh out the ideas. This functionality probably belongs as a part of the qmf::Agent class upstream.
I am totally on board with the principle that we need to make this kind of thing simpler for users. The QMF API is obtuse and not especially well documented (yet). For example, formal parameters in header files are not even named, and even in the private implementations they all have single-character names. So all you can tell in many cases is that a method takes two strings and an int as parameters. Try to guess what they mean! It's like some sort of cruel puzzle. You have to dive through another level of indirection to the private Impl code to even find out what the parameters are called, let alone what they mean. So we need to improve things through some combination of providing helpers in matahari and pushing improvements upstream.
Having said that, I think this patch is tackling too narrow a problem (existence of a particular schema/property/method/argument). The real issue here is that you can't look up a particular schema/property/method/argument without iterating through a list of them. This is unfortunate and bizarre, IMHO, since all of these things are primarily identified by name, not by order. When you do callMethod() you pass it the name of the method you want to call, not the order it appeared in the schema. It's even more unfortunate that it has been implemented in such a way that the consumer has to do all the iteration itself, since that places unnecessary limits on the internal representation... although looking at the code it's already gratuitously inefficient, so maybe that's not such an issue. A more C++ish way to implement this would have been by providing an iterator.
The good news, however, is that the Schema, SchemaMethod and SchemaProperty classes all provide operator bool() and operator!(), so if you just want to know about existence of something in the schema then
if (schema_has_property(agent, "org.matahariproject", "Host", "hostname")) { // i can has hostname }
is equivalent to
::qmf::Schema host(agent_get_schema_by_name(agent, "org.matahariproject", "Host")); if (host && schema_get_property_by_name(host, "hostname")) { // i can has hostname }
but the latter is more flexible: it gives the caller access to the actual Schema object itself and makes it easy to, e.g. lookup multiple properties in the same Schema object without having to copy & paste all of the parameters each time.
Obviously, as with your suggestions, the best place to ultimately implement this would be in the qmf::Agent, qmf::Schema and qmf::SchemaMethod classes.
(I also added some specific implementation suggestions on GitHub.)
I vote for better doxygen comments and non-trivial examples. I think they are more useful than helper functions. Since once you understand what is going on it is easy to write.
I think that anyone that writes a non-trivial QMF program will have to write heaps of wrapper classes according to their own needs (versioning/schema is only one tiny part of it).
https://github.com/pacemaker-cloud/pacemaker-cloud/blob/master/src/qmf_objec... https://github.com/pacemaker-cloud/pacemaker-cloud/blob/master/src/qmf_multi... https://github.com/pacemaker-cloud/pacemaker-cloud/blob/master/src/qmf_agent...
- QMF doesn't have async_method_call timeouts - If an agent fails what do you do with outstanding async requests? - Better handling of multiple agents on one console connection - timing and logging of methos calls for diagnostics - better handling of failed agent connections (if you notice an agent dies before QMF sends you an AGENT_DEL event you need to make sure not to try and re-connect to it)
-Angus
Comments welcome. Thanks.
Ditto ;)
cheers, Zane. _______________________________________________ Matahari mailing list Matahari@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/matahari
On 08/17/2011 08:32 AM, Angus Salkeld wrote:
On Wed, Aug 17, 2011 at 12:28:07PM +0200, Zane Bitter wrote:
I am totally on board with the principle that we need to make this kind of thing simpler for users. The QMF API is obtuse and not especially well documented (yet). For example, formal parameters in header files are not even named, and even in the private implementations they all have single-character names. So all you can tell in many cases is that a method takes two strings and an int as parameters. Try to guess what they mean! It's like some sort of cruel puzzle. You have to dive through another level of indirection to the private Impl code to even find out what the parameters are called, let alone what they mean. So we need to improve things through some combination of providing helpers in matahari and pushing improvements upstream.
Having said that, I think this patch is tackling too narrow a problem (existence of a particular schema/property/method/argument). The real issue here is that you can't look up a particular schema/property/method/argument without iterating through a list of them. This is unfortunate and bizarre, IMHO, since all of these things are primarily identified by name, not by order. When you do callMethod() you pass it the name of the method you want to call, not the order it appeared in the schema. It's even more unfortunate that it has been implemented in such a way that the consumer has to do all the iteration itself, since that places unnecessary limits on the internal representation... although looking at the code it's already gratuitously inefficient, so maybe that's not such an issue. A more C++ish way to implement this would have been by providing an iterator.
The good news, however, is that the Schema, SchemaMethod and SchemaProperty classes all provide operator bool() and operator!(), so if you just want to know about existence of something in the schema then
if (schema_has_property(agent, "org.matahariproject", "Host", "hostname")) { // i can has hostname }
is equivalent to
::qmf::Schema host(agent_get_schema_by_name(agent, "org.matahariproject", "Host")); if (host&& schema_get_property_by_name(host, "hostname")) { // i can has hostname }
but the latter is more flexible: it gives the caller access to the actual Schema object itself and makes it easy to, e.g. lookup multiple properties in the same Schema object without having to copy& paste all of the parameters each time.
Obviously, as with your suggestions, the best place to ultimately implement this would be in the qmf::Agent, qmf::Schema and qmf::SchemaMethod classes.
(I also added some specific implementation suggestions on GitHub.)
I vote for better doxygen comments and non-trivial examples. I think they are more useful than helper functions. Since once you understand what is going on it is easy to write.
I think that anyone that writes a non-trivial QMF program will have to write heaps of wrapper classes according to their own needs (versioning/schema is only one tiny part of it).
https://github.com/pacemaker-cloud/pacemaker-cloud/blob/master/src/qmf_objec... https://github.com/pacemaker-cloud/pacemaker-cloud/blob/master/src/qmf_multi... https://github.com/pacemaker-cloud/pacemaker-cloud/blob/master/src/qmf_agent...
- QMF doesn't have async_method_call timeouts
- If an agent fails what do you do with outstanding async requests?
- Better handling of multiple agents on one console connection
- timing and logging of methos calls for diagnostics
- better handling of failed agent connections (if you notice an agent dies before QMF sends you an AGENT_DEL event you need to make sure not to try and re-connect to it)
This is all great feedback. It's much appreciated. Here are some thoughts on where to go from here:
1) I'm going to capture the general rules for schema changes on a wiki page. We should expand this with example code once the related QMF APIs (and potentially Matahari helper code) settles down.
2) The qmf schema helpers were a good learning exercise for me, but shouldn't be taken any further for now. The QMF APIs could use some usability love from our perspective and that should be done first. I'm going to work on an upstream QMF patch based on the exchange between Zane and I above.
3) We should help with the QMF documentation effort to whatever degree makes sense. For example, if I go off and start working on a patch to QMF APIs, I should add documentation along the way. If any one of us spends time reading their code to understand something, document what you learn in the headers. It is obviously in our best interest to help make QMF easier to adopt.
Doxygen docs for QMF is in the Qpid issue tracker as well. According to the issue, it's targeted for 0.11.
https://issues.apache.org/jira/browse/QPID-3257
4) We should be thinking about our own documentation for QMF console writers against Matahari APIs. This should include both trivial and non-trivial console implementation examples. As far as I know, we don't have much in this area right now, but those people are our user base and should be our top priority for documentation efforts in our project.
Thanks,
On 08/17/2011 12:44 PM, Russell Bryant wrote:
- I'm going to capture the general rules for schema changes on a wiki
page. We should expand this with example code once the related QMF APIs (and potentially Matahari helper code) settles down.
https://github.com/matahari/matahari/wiki/Schemas
On 08/17/2011 12:44 PM, Russell Bryant wrote:
- The qmf schema helpers were a good learning exercise for me, but
shouldn't be taken any further for now. The QMF APIs could use some usability love from our perspective and that should be done first. I'm going to work on an upstream QMF patch based on the exchange between Zane and I above.
A first patch:
https://issues.apache.org/jira/browse/QPID-3436
matahari@lists.fedorahosted.org