Repository navigation
context.getSecondaryResources() returns incomplete list on initial reconcile after operator startup #2869
Description
Bug Report
What did you do?
I have a reconciler that manages a primary custom resource (CR-A), which owns many secondary resources (CR-B) — several hundred per namespace.
I’ve registered a SecondaryToPrimaryMapper to map CR-B back to its owning CR-A.
However, on operator startup, during the first reconciliation of CR-A, context.getSecondaryResources() returns only a partial list of its CR-B objects.
A few seconds later, subsequent reconciliations see all CR-Bs. But I need the full list during the first reconcile.
What did you expect to see?
I expect context.getSecondaryResources() to return all secondary resources immediately after the operator starts.
What did you see instead? Under which circumstances?
After starting the operator, the very first reconciliation of CR-A, context.getSecondaryResources returns a partial list of its corresponding CR-B resources.
This happened even though all CR-B resources were already created before starting the operator. A few seconds later, subsequent reconciliations were able to retrieve the full list correctly.
Environment
Kubernetes cluster type:
- docker-desktop, kubeadm, 1 node, v1.32.2
$ Mention java-operator-sdk version from pom.xml file
io.javaoperatorsdk:operator-framework-spring-boot-starter:6.1.0
$ java -version
% java -version
openjdk 21.0.7 2025-04-15 LTS
OpenJDK Runtime Environment Temurin-21.0.7+6 (build 21.0.7+6-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.7+6 (build 21.0.7+6-LTS, mixed mode, sharing)
$ kubectl version
% kubectl version
Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.32.2
Possible Solution
Additional context
Steps to Reproduce
Possible Cause (Based on Source Code)
I suspect the following might be causing the issue:
- When using InformerEventSource, context.getSecondaryResources() will attempt to retrieve secondary resources via the primaryToSecondaryIndex if no PrimaryToSecondaryMapper is configured. (InformerEventSource.java#L250)
- The primaryToSecondaryIndex is updated inside InformerEventSource.onAdd, which is triggered when a resource is added. (InformerEventSource.java#L122)
- However, onAdd is called asynchronously as part of informer initialization, so it may be invoked after the initial reconciliation is triggered.
- Ultimately, this onAdd is dispatched here (in kubernetes-client), but since it runs asynchronously, there is no guarantee when the index will be fully populated:
Activity
csviri commentedon Jul 21, 2025
Thank you @tomoyat , we will take a look.
Looping in also @shawkins .
We are starting the informer this way:
java-operator-sdk/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/InformerWrapper.java
Lines 90 to 100 in 12b1711
@shawkins I guess it is not guaranteed that events are also processed when this start check succeeds. (I will also take a look later)
csviri commentedon Jul 21, 2025
Probably the best way to fix this is to use an informer indexer rather than a custom index. Will create a PR soon.
csviri commentedon Jul 24, 2025
@tomoyat pls check the related PR, it is mostly ready, there might be asome additional improvements but it fixes this issue, feel free to test it. Thx again for reporting it.
I will wait for @metacosm to review it when back from holidays and after we will probably make a patch release.
@shawkins in case you have time pls review it also
tomoyat commentedon Jul 24, 2025
@csviri Thank you very much. I’ll take a look at the PR.
I’m not sure if this is within your scope, but we’re using
operator-framework-spring-boot-starter, so if possible, it would be helpful if the patch release could be reflected there as well.I really appreciate your quick response.
csviri commentedon Jul 31, 2025
At the end I experimented with multiple variations, first two alternarives were using informers index from fabric8 client.
However fabric8 index implementation has an issues, that is not consistent with the cache, and actally can happen that it temproarily removes a resource from an index, see:
https://github.com/csviri/kubernetes-client/blob/130f7109ec6e30966aca82c2bd7cf2952a447773/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/informers/impl/cache/CacheImpl.java#L277-L279
https://github.com/csviri/kubernetes-client/blob/130f7109ec6e30966aca82c2bd7cf2952a447773/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/informers/impl/cache/CacheImpl.java#L41-L43
while other thread reads the index, and that results in a uncomplete list returned of indexed values. (cc @shawkins )
The current solution is quite simple, it extends sync startup method with an additional step, that adds all the resource in the cahce into the
primary to secondary index. I plan to make a patch release after this merged.cc @metacosm @xstefank
csviri commentedon Aug 5, 2025
Merged and created a release, thank you @tomoyat and others!
tomoyat commentedon Aug 7, 2025
Thank you!
It looks like the changes have also been reflected in
operator-framework-spring-boot-starter, so I’ll give that a try.I really appreciate your help.
bump josdk to 5.1.2
[SPARK-53245] Update Java Operator SDK version to 5.1.2