You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If we see this in a way that we are the client only of the API, and the conversion hooks are properly configured for a custom resource, the migration can be seamlessly done by upgrading the operator to support JUST the most recent version of a CR.
While Kubernetes API should support multiple versions at the same time, the intention is that the clients (in this case the operator) is migrated eventually to the new version. So it's not the case that we have to support 2 versions of the same CR at the same time.
To be honest I don't fully understand what happens if the conversion strategy is None (maybe worth to try it out).
In addition to that, supporting conversion webhooks is out of the scope of the framework IMHO, or could be part of an isolated subproject of the SDK (not the framework), but that is a longer discussion and/or an other issue.
So IMHO, if nobody else sees it otherwise (@metacosm@stoetti@adam-sandor ? ) we are actually good on this. There is nothing to be done. I would even close the issue.
I gave it a try without using the conversionStrategy 'None'. In that case k8s just changes the apiVersion to the version thats marked with stored=true. This obviously leads to a deserialization-error if there is an incompatible change in the CRD.
I will work on a sample with a conversion webhook and see if there are any aspects that are worth building into a subproject. From the documentation the conversion-webhook is just a simple REST endpoint serving a specific interface.
I gave it a try without using the conversionStrategy 'None'. In that case k8s just changes the apiVersion to the version thats marked with stored=true. This obviously leads to a deserialization-error if there is an incompatible change in the CRD.
yes, just don't know how we would support that, probably does not make sense at all to do. Not sure how this is meant to be used, probably would be worth asking on kuberentes forums.
I will work on a sample with a conversion webhook and see if there are any aspects that are worth building into a subproject. From the documentation the conversion-webhook is just a simple REST endpoint serving a specific interface.
Sure, pls do, what I mean is since this is a mapping basically between json structures there is not much that can be done on a framework level to support this. Setting up such a service is already quite simple using just an application framework (spring, quarkus).
From my perspective, different versions should be backed by different POJOs (though I understand there might be cases where that's not practical), each "registered" using a different version annotation so there shouldn't be a deserialization issue. Of course, if there is no backing object, there really is nothing the SDK can do… :)
Maybe I don't understand this correctly, but the point of these conversion webhooks is to avoid to thinkin about multiple versions at the same time on client side. So having a conversion hook, basically makes it transparent for the client what version is actually used.
Thus, if somebody has an operator, and there is a new version of CRD (with a conversion servcie / webhook handler) in place, the operator can seamlessly migrate, just supporting one version in a time.
But will have to try it out.
Well, this would only work if your client can handle whatever the server returns, right? So even if you only want to keep one version to deal with on the client, there still needs to be a POJO on the client that's able to handle the converted version.
That said, I'm not super familiar with conversion webhooks so maybe I'm missing something.
This popped up again, consider a migration path from v1 to v2 of CR and related Reconciler:
Implemented an operator for version v1 CR v1
CR v2 is developed and deployed with a conversion hook (does not matter which one is the storage version)
The Operator will have a separate reconciler/controller which already asks for resources (maybe filtered by label) with version v2.
After v2 with is tested with a subset of resources, it can be switched to just use the v2 of the CR and Controller.
In other words actually to support kinda canary testing within one operator with multiple controllers, so we would allow to have a separate controller for different version of CR.
To be done from our side, would be just to remove the current check, and create an Integration test that showcases this approach.
I completely agree with this last statement @csviri .
More specifically Kubernetes does allow to serve multiple CRDs versions while only one will be marked as storage.
I do believe that the operator-sdk should provide the infrastructure to handle this situation to end-users:
Guarantee the alignment with the version marked as storage comparing the installed CRD versions and the code
Guarantee the alignment of served CRDs versions
Dispatch updates only to the correct controller filtering out the rest before deserializing the CR
The end user can than decide how to handle this situation, simply throwing an exception, logging, updating the status etc...
@andreaTP@metacosm I think we also do this after v2, checked and would like then create an e2e test for it, that would require a conversion hook etc, so this is not just for few hours to do it in good quality. Since it's not an API change will do it after v2.0.0 released, and will be one of the issues in v2.1.0.
A workaround for now is to implement both the controllers in the code base, put there a feature flag which one to register and deploy two instances of the operator, where the feature flag is the difference.
Activity
stoetti commentedon Mar 3, 2021
@csviri Is this on the roadmap in the near future?
csviri commentedon Mar 3, 2021
@stoetti yes, this should be actually one of the priorities in the close future.
pinging also @metacosm
metacosm commentedon Mar 3, 2021
This should actually work already, no? If not, could you explicit a scenario that doesn't work?
csviri commentedon Mar 3, 2021
@metacosm good question, went though again on the docs:
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning
If we see this in a way that we are the client only of the API, and the conversion hooks are properly configured for a custom resource, the migration can be seamlessly done by upgrading the operator to support JUST the most recent version of a CR.
While Kubernetes API should support multiple versions at the same time, the intention is that the clients (in this case the operator) is migrated eventually to the new version. So it's not the case that we have to support 2 versions of the same CR at the same time.
To be honest I don't fully understand what happens if the conversion strategy is None (maybe worth to try it out).
In addition to that, supporting conversion webhooks is out of the scope of the framework IMHO, or could be part of an isolated subproject of the SDK (not the framework), but that is a longer discussion and/or an other issue.
So IMHO, if nobody else sees it otherwise (@metacosm @stoetti @adam-sandor ? ) we are actually good on this. There is nothing to be done. I would even close the issue.
stoetti commentedon Mar 4, 2021
I gave it a try without using the conversionStrategy 'None'. In that case k8s just changes the apiVersion to the version thats marked with stored=true. This obviously leads to a deserialization-error if there is an incompatible change in the CRD.
I will work on a sample with a conversion webhook and see if there are any aspects that are worth building into a subproject. From the documentation the conversion-webhook is just a simple REST endpoint serving a specific interface.
csviri commentedon Mar 4, 2021
yes, just don't know how we would support that, probably does not make sense at all to do. Not sure how this is meant to be used, probably would be worth asking on kuberentes forums.
Sure, pls do, what I mean is since this is a mapping basically between json structures there is not much that can be done on a framework level to support this. Setting up such a service is already quite simple using just an application framework (spring, quarkus).
metacosm commentedon Mar 4, 2021
From my perspective, different versions should be backed by different POJOs (though I understand there might be cases where that's not practical), each "registered" using a different version annotation so there shouldn't be a deserialization issue. Of course, if there is no backing object, there really is nothing the SDK can do… :)
csviri commentedon Mar 4, 2021
(Sorry this was an accident closing it)
Maybe I don't understand this correctly, but the point of these conversion webhooks is to avoid to thinkin about multiple versions at the same time on client side. So having a conversion hook, basically makes it transparent for the client what version is actually used.
Thus, if somebody has an operator, and there is a new version of CRD (with a conversion servcie / webhook handler) in place, the operator can seamlessly migrate, just supporting one version in a time.
But will have to try it out.
metacosm commentedon Mar 4, 2021
Well, this would only work if your client can handle whatever the server returns, right? So even if you only want to keep one version to deal with on the client, there still needs to be a POJO on the client that's able to handle the converted version.
That said, I'm not super familiar with conversion webhooks so maybe I'm missing something.
8 remaining items
fix: multiple versions should actually not be supported
fix: improve duplicated controller detection, add tests
fix: improve duplicated controller detection, add tests
fix: improve duplicated controller detection, add tests
fix: improve duplicated controller detection, add tests
fix: improve duplicated controller detection, add tests
csviri commentedon Jan 10, 2022
This popped up again, consider a migration path from v1 to v2 of CR and related Reconciler:
In other words actually to support kinda canary testing within one operator with multiple controllers, so we would allow to have a separate controller for different version of CR.
To be done from our side, would be just to remove the current check, and create an Integration test that showcases this approach.
csviri commentedon Jan 10, 2022
pls let me know what do you think @metacosm @adam-sandor @andreaTP
andreaTP commentedon Jan 10, 2022
I completely agree with this last statement @csviri .
More specifically Kubernetes does allow to
servemultiple CRDs versions while only one will be marked asstorage.I do believe that the
operator-sdkshould provide the infrastructure to handle this situation to end-users:storagecomparing the installed CRD versions and the codeservedCRDs versionsThe end user can than decide how to handle this situation, simply throwing an exception, logging, updating the status etc...
csviri commentedon Jan 13, 2022
@andreaTP @metacosm I think we also do this after v2, checked and would like then create an e2e test for it, that would require a conversion hook etc, so this is not just for few hours to do it in good quality. Since it's not an API change will do it after v2.0.0 released, and will be one of the issues in v2.1.0.
csviri commentedon Jan 13, 2022
A workaround for now is to implement both the controllers in the code base, put there a feature flag which one to register and deploy two instances of the operator, where the feature flag is the difference.
csviri commentedon Jan 18, 2022
(related docs in kubebuilder: https://book.kubebuilder.io/multiversion-tutorial/tutorial.html )