There are a few ways certs get regenerated: 1. During refresh pools if (a) the product name/attributes change (b) provided products are added/removed (c) subscription dates changed (d) subscription was revoked - quantity changes could lead to certs getting revoked if we're over the limit, but does not regenerate anything. - we do not regenerate on content set changes (we do for provided products, but not just content sets within one of those provided products). we do not store content data locally in some deployments so there's nothing to compare against to see if anything changed.

- I am assuming the business rules would require us to immediately revoke for (d), but to use lazy revocation for all of the others.

2. An explicit call to regenerate everything for some product, I do not know where/if IT uses this: PUT /entitlements/product/{productId} Lazy regeneration should probably be used here. (doesn't matter what changed, we don't really know, but we can rest assured it wasn't anyone losing their subscription access)

3. An explicit call to regenerate everything for a given consumer: PUT /consumers/{uuid}/certificates. IMO we should not do any lazy regeneration here, I think the use case for this call was for security reasons and the like.

4. If configured for environments, whenever content is promoted/demoted to an environment, we mass regenerate for those consumers. Lazy regen seems good here.

This seems fairly straightforward so far. The request was for distributor consumer types only. - Add a "dirty" column to cp_entitlement, flip it to true as desired for entitlements when they need regeneration during (1) or (2) above. - In GET /consumers/{uuid}/export check all entitlements for the dirty flag, if any are found regenerate them, then proceed normally. jbowes has proposed possibly leveraging lazy regen for all consumers of all types for some cert v2 purposes. (might need more info from you on this jbowes) I think this just implies expanding the hooks for triggering regeneration to:

GET /consumers/{uuid} GET /consumers/{uuid}/entitlements GET /consumers/{uuid}/certificates (both JSON and ZIP calls) GET /consumers/{uuid}/certificates/serials I don't think this is much more work, the method to scan for dirty entitlements and regenerate would be shared across all these API calls. The implementation is cleaner particularly in the locations where we decide to set dirty flag or not, if this is not specific to just one consumer type and path through the code. Thoughts? Thanks,

On Fri 25 May 2012 09:54:20 AM ADT, Bryan Kearney wrote: > On 05/24/2012 03:30 PM, Devan Goodwin wrote: >> There are a few ways certs get regenerated: >> >> 1. During refresh pools if >> (a) the product name/attributes change >> (b) provided products are added/removed >> (c) subscription dates changed >> (d) subscription was revoked >> - quantity changes could lead to certs getting revoked if we're over the >> limit, but does not regenerate anything. >> - we do not regenerate on content set changes (we do for provided >> products, but not just content sets within one of those provided >> products). we do not store content data locally in some deployments so >> there's nothing to compare against to see if anything changed. > > Isnt this why we had a refresh based on product? If we dont have this, > I have a new backlog item :) > >> - I am assuming the business rules would require us to immediately >> revoke for (d), but to use lazy revocation for all of the others. > > Agreed. > >> >> 2. An explicit call to regenerate everything for some product, I do not >> know where/if IT uses this: PUT /entitlements/product/{productId} Lazy >> regeneration should probably be used here. (doesn't matter what changed, >> we don't really know, but we can rest assured it wasn't anyone losing >> their subscription access) > > > agreed... answers my question from above. I thought we had this already.

> >> >> 3. An explicit call to regenerate everything for a given consumer: PUT >> /consumers/{uuid}/certificates. IMO we should not do any lazy >> regeneration here, I think the use case for this call was for security >> reasons and the like. > > No, I would here as well. Or, give an option: > > PUT /consumers/{uuid}/certificates?lazy=HellNo Ok if there's no known use case for demanding immediate regen for a consumer's certs, then may as well make it default to lazy and stay consistent with the rest of the API. > >> >> 4. If configured for environments, whenever content is promoted/demoted >> to an environment, we mass regenerate for those consumers. Lazy regen >> seems good here. > Need to define lazy for me here. I would assume we would not have > distributors and environments. > Only in like a nested Katello scenario, but my hope is that we can proceed with this for all consumers, not just distributors. So when content is promoted and demoted, all affected consumers would get their entitlements dirty flag set and will regen when they next check in. This actually alleviates one of the big concerns I had after the environments work where promotion/demotion could be really slow. Now, you can promote/demote multiple times without triggering any mass regen, should work a lot better actually. >> >> >> >> This seems fairly straightforward so far. The request was for >> distributor consumer types only. >> >> - Add a "dirty" column to cp_entitlement, flip it to true as desired for >> entitlements when they need regeneration during (1) or (2) above. >> >> - In GET /consumers/{uuid}/export check all entitlements for the dirty >> flag, if any are found regenerate them, then proceed normally. >> >> jbowes has proposed possibly leveraging lazy regen for all consumers of >> all types for some cert v2 purposes. (might need more info from you on >> this jbowes) I think this just implies expanding the hooks for >> triggering regeneration to: > > I agree. It may be nice to have the pain of creating the new certs at > next machine checkin. This would speed up the repool by spreading out > the pain. Cool, I am liking the sound of it as well. Thanks, Devan > >> >> GET /consumers/{uuid} >> GET /consumers/{uuid}/entitlements >> GET /consumers/{uuid}/certificates (both JSON and ZIP calls) >> GET /consumers/{uuid}/certificates/serials >> >> I don't think this is much more work, the method to scan for dirty >> entitlements and regenerate would be shared across all these API calls. >> The implementation is cleaner particularly in the locations where we >> decide to set dirty flag or not, if this is not specific to just one >> consumer type and path through the code. >> >> Thoughts? >> >> Thanks, > > -- bk > > > _______________________________________________ > candlepin mailing list > candlepin(a)lists.fedorahosted.org > https://fedorahosted.org/mailman/listinfo/candlepin

On Fri 25 May 2012 09:54:20 AM ADT, Bryan Kearney wrote: > On 05/24/2012 03:30 PM, Devan Goodwin wrote: >> There are a few ways certs get regenerated: >> >> 1. During refresh pools if >> (a) the product name/attributes change >> (b) provided products are added/removed >> (c) subscription dates changed >> (d) subscription was revoked >> - quantity changes could lead to certs getting revoked if we're over the >> limit, but does not regenerate anything. >> - we do not regenerate on content set changes (we do for provided >> products, but not just content sets within one of those provided >> products). we do not store content data locally in some deployments so >> there's nothing to compare against to see if anything changed. > > Isnt this why we had a refresh based on product? If we dont have this, > I have a new backlog item :) > >> - I am assuming the business rules would require us to immediately >> revoke for (d), but to use lazy revocation for all of the others. > > Agreed. > >> >> 2. An explicit call to regenerate everything for some product, I do not >> know where/if IT uses this: PUT /entitlements/product/{productId} Lazy >> regeneration should probably be used here. (doesn't matter what changed, >> we don't really know, but we can rest assured it wasn't anyone losing >> their subscription access) > > > agreed... answers my question from above. I thought we had this already. > >> >> 3. An explicit call to regenerate everything for a given consumer: PUT >> /consumers/{uuid}/certificates. IMO we should not do any lazy >> regeneration here, I think the use case for this call was for security >> reasons and the like. > > No, I would here as well. Or, give an option: > > PUT /consumers/{uuid}/certificates?lazy=HellNo Ok if there's no known use case for demanding immediate regen for a consumer's certs, then may as well make it default to lazy and stay consistent with the rest of the API. > >> >> 4. If configured for environments, whenever content is promoted/demoted >> to an environment, we mass regenerate for those consumers. Lazy regen >> seems good here. > Need to define lazy for me here. I would assume we would not have > distributors and environments. > Only in like a nested Katello scenario, but my hope is that we can proceed with this for all consumers, not just distributors. So when content is promoted and demoted, all affected consumers would get their entitlements dirty flag set and will regen when they next check in. This actually alleviates one of the big concerns I had after the environments work where promotion/demotion could be really slow. Now, you can promote/demote multiple times without triggering any mass regen, should work a lot better actually.

jbowes has proposed possibly leveraging lazy regen for all consumers of all types for some cert v2 purposes. (might need more info from you on this jbowes) I think this just implies expanding the hooks for triggering regeneration to: GET /consumers/{uuid} GET /consumers/{uuid}/entitlements GET /consumers/{uuid}/certificates (both JSON and ZIP calls) GET /consumers/{uuid}/certificates/serials I don't think this is much more work, the method to scan for dirty entitlements and regenerate would be shared across all these API calls. The implementation is cleaner particularly in the locations where we decide to set dirty flag or not, if this is not specific to just one consumer type and path through the code. Thoughts?

Thanks, Devan _______________________________________________ candlepin mailing list candlepin(a)lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/candlepin

On 05/25/2012 02:50 PM, James Bowes wrote: >On Thu, May 24, 2012 at 04:30:32PM -0300, Devan Goodwin wrote: ><snip> >>jbowes has proposed possibly leveraging lazy regen for all consumers >>of all types for some cert v2 purposes. (might need more info from >>you on this jbowes) I think this just implies expanding the hooks >>for triggering regeneration to: >> >>GET /consumers/{uuid} >>GET /consumers/{uuid}/entitlements >>GET /consumers/{uuid}/certificates (both JSON and ZIP calls) >>GET /consumers/{uuid}/certificates/serials >> >>I don't think this is much more work, the method to scan for dirty >>entitlements and regenerate would be shared across all these API >>calls. The implementation is cleaner particularly in the locations >>where we decide to set dirty flag or not, if this is not specific to >>just one consumer type and path through the code. >> >>Thoughts? >> > >My thinking on making regens lazy for all consumer types: > 1. spread x509 related load out across time. systems check in every 4 > hours or whatever, not all at once. > 2. with multiple certificate formats, there's no need to know what > version the consumer wants before they ask for it. we simply > regenerate what they ask for (via api or looking at a header on the > request or whatever). > 3. fewer special cases make for cleaner code > 4. if we want to be nice to distributors and let them use certs that > aren't exactly current, shouldn't we extend the same courtesy to system > consumers? > +1 to all. >Doing a lazy regen implies that the real work of deciding if a >certificate is still good is when we put that old certificate on the CRL >list. We'd only add a certificate to the CRL immediately if it has >recieved a detectable net loss in access to content, like: >- quantity of entitlement has decreased (not sure if this is a real > thing or not) >- content set is removed from a product (not detectable?) >- content set url is changed on a product (not detectable?) >- product is removed from a subscription > >If we hit one of those cases on a refreshpools, we add the cert cerial >to the CRL, and mark it as dirty, letting the client trigger the regen >at some later point. Quantity decreasing should probably do as it does now and go ahead and revoke, this isn't really a regeneration but an outright revocation, and probably is not important for the use case we're trying to solve. I vote we let this immediately go to CRL and revoke the cert as we do today. (it's probably a pretty rare situation anyhow)

Content set changes are indeed not detectable, but possibly being triggered by IT to mass regenerate via that product specific API call. We will not know why we're being triggered though so as it is today, we cannot decide if we're going to CRL now or later when we regenerate. However as per next point I am hoping we can keep it simple and have only two situations. For products being removed I will wager we still do not want to add to the CRL immediately, driver here was to not invalidate distributor certificates and my guess is we should not be doing that even if we're removing access to a particular repo from their subscription. Can we get clarification on this? (should we immediately invalidate certificates if a product is removed from a subscription)

Hopefully we can just stay with these two situations: 1. we're either adding to CRL + revoking (overconsumption, normal unbind, no work required because this is how it is today) 2. we're flagging entitlement dirty now, and adding to CRL + regenerating on next check-in or manifest generation > >If we don't hit one of those cases, like if we've added a new product, >we simply mark the cert as dirty. That dirty cert is still valid until >the new regen is done, but why not just keep it valid for as long as >the newly regenerated version? that is, can we reuse the cert serial on >the new cert, keeping the old version still valid, and reducing both the >clutter in our CRL and the rate at which we consume cert serials? I >think that would work. Having multiple certificates out in the wild with different contents but the same serial, all still valid, seems strange and like a probable source of future confusion, and perhaps an issue for our database. Each unique cert seems like it should probably have it's own serial to me.

If CRL clutter is a current known problem then it's worth considering, but otherwise I'd propose we leave that for another day and solve it if the issue arises.

Cheers, Devan

On 05/25/2012 04:02 PM, James Bowes wrote: > I mean the quantity in the entitlement decreases, rather than the > quantity in the pool. Ah yes, that might need more fleshing out, I think I missed that there is a path to regenerate, not just revoke in there. I will revisit that.

> >> Content set changes are indeed not detectable, but possibly being >> triggered by IT to mass regenerate via that product specific API >> call. We will not know why we're being triggered though so as it is >> today, we cannot decide if we're going to CRL now or later when we >> regenerate. However as per next point I am hoping we can keep it >> simple and have only two situations. >> >> For products being removed I will wager we still do not want to add >> to the CRL immediately, driver here was to not invalidate >> distributor certificates and my guess is we should not be doing that >> even if we're removing access to a particular repo from their >> subscription. Can we get clarification on this? (should we >> immediately invalidate certificates if a product is removed from a >> subscription) >> > > I'm pretty sure this is the exact situation where we do want to add to > the CRL. removing a product or a content set means you should no longer > have access to that content. If all a nefarious consumer has to do to > keep accessing content they're not rightly entitled to is to not ask for > a new cert, candlepin isn't doing its job. It's possible but I think we need to check, if I am reading the user story notes correctly, removal of content sets is listed explicitly as one of the scenarios where we want lazy revocation. i.e. we removed some content from your entitlement (in this case a provided product and it's content), but we do not want to immediately break all those downstream consumers. Candlepin could still be doing it's job if we reject immediately on loss of entitlement, but take a tolerant approach to rejecting certificates because we changed something on our end. Highly unlikely a nefarious consumer is going to anticipate a change to our product data, and shut off all consumer check-ins as far as I can forsee. It all depends on why this is being requested though.

On 05/25/2012 04:38 PM, Devan Goodwin wrote: >On Fri 25 May 2012 04:18:02 PM ADT, Devan Goodwin wrote: >>On 05/25/2012 04:02 PM, James Bowes wrote: >>>I mean the quantity in the entitlement decreases, rather than the >>>quantity in the pool. >> >>Ah yes, that might need more fleshing out, I think I missed that there >>is a path to regenerate, not just revoke in there. I will revisit that. > >Ok as per IRC discussion, if a cert needs to regen because it's quantity >changed (due to a decrease in the subscription), we will immediately >revoke and regen. > >> >>> >>>>Content set changes are indeed not detectable, but possibly being >>>>triggered by IT to mass regenerate via that product specific API >>>>call. We will not know why we're being triggered though so as it is >>>>today, we cannot decide if we're going to CRL now or later when we >>>>regenerate. However as per next point I am hoping we can keep it >>>>simple and have only two situations. >>>> >>>>For products being removed I will wager we still do not want to add >>>>to the CRL immediately, driver here was to not invalidate >>>>distributor certificates and my guess is we should not be doing that >>>>even if we're removing access to a particular repo from their >>>>subscription. Can we get clarification on this? (should we >>>>immediately invalidate certificates if a product is removed from a >>>>subscription) >>>> >>> >>>I'm pretty sure this is the exact situation where we do want to add to >>>the CRL. removing a product or a content set means you should no longer >>>have access to that content. If all a nefarious consumer has to do to >>>keep accessing content they're not rightly entitled to is to not ask for >>>a new cert, candlepin isn't doing its job. >> >>It's possible but I think we need to check, if I am reading the user >>story notes correctly, removal of content sets is listed explicitly as >>one of the scenarios where we want lazy revocation. i.e. we removed >>some content from your entitlement (in this case a provided product >>and it's content), but we do not want to immediately break all those >>downstream consumers. >> >>Candlepin could still be doing it's job if we reject immediately on >>loss of entitlement, but take a tolerant approach to rejecting >>certificates because we changed something on our end. Highly unlikely >>a nefarious consumer is going to anticipate a change to our product >>data, and shut off all consumer check-ins as far as I can forsee. It >>all depends on why this is being requested though. >> > >And for this we're going to assume content / product removal is still a >lazy operation by default, but both refresh pools for owner, and >regenerate all certs for a given product ID API calls will take an >option to force immediate certificate revocation. This is a safety net >in case something was granted that really should not have been. > > >I will write this all up on wiki and break down into tasks next week. > >Cheers, > >Devan >_______________________________________________ >candlepin mailing list &gt;candlepin(a)lists.fedorahosted.org >https://fedorahosted.org/mailman/listinfo/candlepin Design is up at https://fedorahosted.org/candlepin/wiki/LazyCertRegen Please have a look at let me know if anything needs to be clarified. Thanks,

On Sat, Jun 01, 2013 at 02:37:19PM -0300, Devan Goodwin wrote: > On 05/25/2012 04:38 PM, Devan Goodwin wrote: >> On Fri 25 May 2012 04:18:02 PM ADT, Devan Goodwin wrote: >>> On 05/25/2012 04:02 PM, James Bowes wrote: >>>> I mean the quantity in the entitlement decreases, rather than the >>>> quantity in the pool. >>> >>> Ah yes, that might need more fleshing out, I think I missed that there >>> is a path to regenerate, not just revoke in there. I will revisit that. >> >> Ok as per IRC discussion, if a cert needs to regen because it's quantity >> changed (due to a decrease in the subscription), we will immediately >> revoke and regen. >> >>> >>>> >>>>> Content set changes are indeed not detectable, but possibly being >>>>> triggered by IT to mass regenerate via that product specific API >>>>> call. We will not know why we're being triggered though so as it is >>>>> today, we cannot decide if we're going to CRL now or later when we >>>>> regenerate. However as per next point I am hoping we can keep it >>>>> simple and have only two situations. >>>>> >>>>> For products being removed I will wager we still do not want to add >>>>> to the CRL immediately, driver here was to not invalidate >>>>> distributor certificates and my guess is we should not be doing that >>>>> even if we're removing access to a particular repo from their >>>>> subscription. Can we get clarification on this? (should we >>>>> immediately invalidate certificates if a product is removed from a >>>>> subscription) >>>>> >>>> >>>> I'm pretty sure this is the exact situation where we do want to add to >>>> the CRL. removing a product or a content set means you should no longer >>>> have access to that content. If all a nefarious consumer has to do to >>>> keep accessing content they're not rightly entitled to is to not ask for >>>> a new cert, candlepin isn't doing its job. >>> >>> It's possible but I think we need to check, if I am reading the user >>> story notes correctly, removal of content sets is listed explicitly as >>> one of the scenarios where we want lazy revocation. i.e. we removed >>> some content from your entitlement (in this case a provided product >>> and it's content), but we do not want to immediately break all those >>> downstream consumers. >>> >>> Candlepin could still be doing it's job if we reject immediately on >>> loss of entitlement, but take a tolerant approach to rejecting >>> certificates because we changed something on our end. Highly unlikely >>> a nefarious consumer is going to anticipate a change to our product >>> data, and shut off all consumer check-ins as far as I can forsee. It >>> all depends on why this is being requested though. >>> >> >> And for this we're going to assume content / product removal is still a >> lazy operation by default, but both refresh pools for owner, and >> regenerate all certs for a given product ID API calls will take an >> option to force immediate certificate revocation. This is a safety net >> in case something was granted that really should not have been. >> >> >> I will write this all up on wiki and break down into tasks next week. >> >> Cheers, >> >> Devan >> _______________________________________________ >> candlepin mailing list >> candlepin(a)lists.fedorahosted.org >> https://fedorahosted.org/mailman/listinfo/candlepin > > Design is up at https://fedorahosted.org/candlepin/wiki/LazyCertRegen > > Please have a look at let me know if anything needs to be clarified. > > Thanks, under task 2 you have "Be sure". Sentance fragment or way of life? Maybe add a note that the regeneration that goes with an immediate CRL can be either lazy or immediate, depending on which is cleaner in the code. -James

On 05/29/2012 02:16 PM, Devan Goodwin wrote: > On 05/29/2012 02:53 PM, James Bowes wrote: >> On Sat, Jun 01, 2013 at 02:37:19PM -0300, Devan Goodwin wrote: >>> On 05/25/2012 04:38 PM, Devan Goodwin wrote: >>>> On Fri 25 May 2012 04:18:02 PM ADT, Devan Goodwin wrote: >>>>> On 05/25/2012 04:02 PM, James Bowes wrote: >>>>>> I mean the quantity in the entitlement decreases, rather than the >>>>>> quantity in the pool. >>>>> >>>>> Ah yes, that might need more fleshing out, I think I missed that >>>>> there >>>>> is a path to regenerate, not just revoke in there. I will revisit >>>>> that. >>>> >>>> Ok as per IRC discussion, if a cert needs to regen because it's >>>> quantity >>>> changed (due to a decrease in the subscription), we will immediately >>>> revoke and regen. >>>> >>>>> >>>>>> >>>>>>> Content set changes are indeed not detectable, but possibly being >>>>>>> triggered by IT to mass regenerate via that product specific API >>>>>>> call. We will not know why we're being triggered though so as it is >>>>>>> today, we cannot decide if we're going to CRL now or later when we >>>>>>> regenerate. However as per next point I am hoping we can keep it >>>>>>> simple and have only two situations. >>>>>>> >>>>>>> For products being removed I will wager we still do not want to add >>>>>>> to the CRL immediately, driver here was to not invalidate >>>>>>> distributor certificates and my guess is we should not be doing >>>>>>> that >>>>>>> even if we're removing access to a particular repo from their >>>>>>> subscription. Can we get clarification on this? (should we >>>>>>> immediately invalidate certificates if a product is removed from a >>>>>>> subscription) >>>>>>> >>>>>> >>>>>> I'm pretty sure this is the exact situation where we do want to >>>>>> add to >>>>>> the CRL. removing a product or a content set means you should no >>>>>> longer >>>>>> have access to that content. If all a nefarious consumer has to >>>>>> do to >>>>>> keep accessing content they're not rightly entitled to is to not >>>>>> ask for >>>>>> a new cert, candlepin isn't doing its job. >>>>> >>>>> It's possible but I think we need to check, if I am reading the user >>>>> story notes correctly, removal of content sets is listed >>>>> explicitly as >>>>> one of the scenarios where we want lazy revocation. i.e. we removed >>>>> some content from your entitlement (in this case a provided product >>>>> and it's content), but we do not want to immediately break all those >>>>> downstream consumers. >>>>> >>>>> Candlepin could still be doing it's job if we reject immediately on >>>>> loss of entitlement, but take a tolerant approach to rejecting >>>>> certificates because we changed something on our end. Highly unlikely >>>>> a nefarious consumer is going to anticipate a change to our product >>>>> data, and shut off all consumer check-ins as far as I can forsee. It >>>>> all depends on why this is being requested though. >>>>> >>>> >>>> And for this we're going to assume content / product removal is >>>> still a >>>> lazy operation by default, but both refresh pools for owner, and >>>> regenerate all certs for a given product ID API calls will take an >>>> option to force immediate certificate revocation. This is a safety net >>>> in case something was granted that really should not have been. >>>> >>>> >>>> I will write this all up on wiki and break down into tasks next week. >>>> >>>> Cheers, >>>> >>>> Devan >>>> _______________________________________________ >>>> candlepin mailing list >>>> candlepin(a)lists.fedorahosted.org >>>> https://fedorahosted.org/mailman/listinfo/candlepin >>> >>> Design is up at https://fedorahosted.org/candlepin/wiki/LazyCertRegen >>> >>> Please have a look at let me know if anything needs to be clarified. >>> >>> Thanks, >> >> under task 2 you have "Be sure". Sentance fragment or way of life? >> Maybe add a note that the regeneration that goes with an immediate CRL >> can be either lazy or immediate, depending on which is cleaner in the >> code. >> >> -James > Why get consumers/{uid}? That seems like an odd side effect? _______________________________________________ candlepin mailing list candlepin(a)lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/candlepin