Conversation
# Conflicts: # gateway-discovery-cm/src/main/java/org/apache/knox/gateway/topology/discovery/cm/ServiceModelGeneratorsHolder.java # gateway-discovery-cm/src/test/java/org/apache/knox/gateway/topology/discovery/cm/ServiceModelGeneratorsHolderTest.java
…figurationAnalyzer getReferencedServiceTypes and clusterReferencesExist swallowed descriptor parse/read errors silently and only caught IOException. A malformed or unreadable descriptor could silently stop rediscovery or discard a live cluster's cached configuration. Both now catch broadly and log the offending file. getReferencedServiceTypes skips the unreadable descriptor and keeps filtering on the descriptors it could read (an unparseable descriptor can't be regenerated until it is fixed, so it can't be the target of a useful rediscovery now). clusterReferencesExist instead assumes a reference remains when a descriptor can't be read, so an unreadable descriptor never causes the monitored cluster's baseline to be discarded.
…de when determining referenced services, since SimpleDescriptorHandler never regenerates them (using the gateway-site override list, not the descriptor's own read-only field). Both now catch broadly and log the offending file. getReferencedServiceTypes skips the unreadable descriptor and keeps filtering on the descriptors it could read (an unparseable descriptor can't be regenerated until it is fixed, so it can't be the target of a useful rediscovery now). clusterReferencesExist instead assumes a reference remains when a descriptor can't be read, so an unreadable descriptor never causes the monitored cluster's baseline to be discarded.
smolnar82
left a comment
There was a problem hiding this comment.
I found 2 blockers, and the following efficiency issues:
1. every descriptor is parsed twice per polling cycle
File: PollingConfigurationAnalyzer.java (getReferencedServiceTypes() / clusterReferencesExist())
clusterReferencesExist() and getReferencedServiceTypes() each independently loop over ts.getDescriptors() and re-parse every descriptor file via SimpleDescriptorFactory.parse() (around lines 423 and 602). With N descriptors on disk, each polling cycle now parses every file twice per cluster — doubling filesystem reads + JSON parsing — and duplicates near-identical IOException handling.
Fix direction: merge the two into a single pass over ts.getDescriptors() that returns both "does any descriptor reference this (address, cluster)" and "the set of referenced service names", so each descriptor is parsed once.
2. CORE_SETTINGS is re-fetched from CM once per relevant service type
File: PollingConfigurationAnalyzer.java (getCoreSettingsConfig())
getCoreSettingsConfig() issues a fresh readServices(clusterName, summary) plus a readServiceConfig() round-trip every time getCurrentServiceConfiguration() runs — i.e. once per distinct relevant service type per polling cycle. Cluster discovery, by contrast, computes CORE_SETTINGS once for the whole cluster (coreSettingsConfig(...)).
Failure path: a polling cycle with events for three service types (e.g. HDFS, HIVE_ON_TEZ, YARN) makes three separate readServices() + readServiceConfig() calls just to re-derive the same CORE_SETTINGS value, adding avoidable CM API load on every interval.
Fix direction: compute CORE_SETTINGS once per polling pass and reuse it across all services handled in that pass (cache it for the duration of the cycle).
3.: role-type exclusion config is cached for the analyzer's lifetime while service-type exclusion is read fresh
File: PollingConfigurationAnalyzer.java (isExcludedRoleType() / isExcludedServiceType())
isExcludedRoleType() lazily builds and then permanently caches its TypeNameFilter (roleTypeFilter) from gatewayConfig on first use, whereas the adjacent isExcludedServiceType() re-reads gatewayConfig.getClouderaManagerServiceDiscoveryExcludedServiceTypes() fresh on every call.
Failure path: an operator updates gateway.cloudera.manager.service.discovery.excluded.role.types on a running gateway (without restarting the CM configuration monitor thread). isExcludedServiceType honors an equivalent excluded-service-types change immediately, but isExcludedRoleType keeps using the stale filter built at first invocation and silently ignores the update.
Fix direction: pick one policy for both. Either read both config values fresh per call, or resolve both once at construction — but don't split the behavior between the two sibling checks introduced in this PR.
| roleConfigs.put(role, configList); | ||
| } | ||
| currentConfig = new ServiceConfigurationModel(svcConfig, roleConfigs); | ||
| final ApiService apiService = new ApiService().name(service).type(serviceType); |
There was a problem hiding this comment.
To reuse the discovery workflow, getCurrentServiceConfiguration() reconstructs the ApiService / ApiRole / config objects it feeds into ServiceModelFactory.generateServiceModels(). But this reconstruction doesn't populate the same fields that real discovery (ClouderaManagerServiceDiscovery.discoverService) hands to generators. Any ServiceModelGenerator that dereferences a field the reconstruction leaves null throws an NPE inside generateService(). This is not an edge case — it hits the majority of generators:
role.getHostRef().getHostname()— used by 34 of 51 generators (OozieServiceModelGenerator,SolrServiceModelGenerator, HBase, Impala, and most others). NPEs if the reconstructed role'shostRef(or its hostname) isn't populated exactly as discovery populates it.service.getClusterRef().getClusterName()— used byYarnUIServiceModelGeneratorandJobHistoryUIServiceModelGenerator. NPEs because the syntheticApiServicesets onlyname/type, noclusterRef.- Any other field a given generator's
handles()/generateService()reads that the reconstruction omits.
Failure path & blast radius: a start/restart/scale event for almost any discoverable service (YARN, Oozie, Solr, HBase, …) → hasConfigChanged → getCurrentServiceConfiguration → ServiceModelFactory.generateServiceModels → the generator's generateService() → NPE. The NPE is not an ApiException, so the method's own try/catch doesn't catch it; it propagates to monitorClusterConfigurationChanges' outer catch(Exception), aborting the entire polling cycle for all clusters. Because the triggering event is never marked processed, the same NPE recurs every polling interval — PCA is effectively dead for that gateway.
Root cause / fix direction: the reconstruction in getCurrentServiceConfiguration() diverges from how ClouderaManagerServiceDiscovery builds these objects. Rather than hand-rebuilding ApiService / ApiRole from readServiceConfig, PCA should obtain the model inputs through the same code path discovery uses (the shared component introduced here), so every field a generator may read is populated identically. A unit test that runs each registered generator against PCA-reconstructed inputs would have caught this and would guard against regressions as new generators land.
There was a problem hiding this comment.
This new branch treats getCurrentServiceConfiguration() == null as "the service is now invalid / disabled" and forces re-discovery. But that method also returns null on any ApiException - network blip, auth failure, transient 5xx:
} catch (ApiException e) {
log.clouderaManagerConfigurationAPIError(e);
}
return currentConfig; // still null on API errorSo a null return is ambiguous: it can mean either "config genuinely produced no model" or "couldn't reach CM". hasConfigChanged's else branch (prior config present, currentConfig == null) can't tell them apart and unconditionally sets configHasChanged = true, logs serviceDisabled, and triggers a full re-discovery.
Failure path: CM API has a transient outage while computing the current config for a service that has a valid prior baseline → getCurrentServiceConfiguration catches the ApiException and returns null → hasConfigChanged forces re-discovery → repeats every polling cycle until CM recovers.
Regression: previously an API-error null was a harmless no-op (configHasChanged stayed false). This PR turns a transient CM error into a repeating full-cluster rediscovery storm — the same failure class KNOX-2900 set out to eliminate.
Fix direction: distinguish "no model produced" from "CM unreachable". Options: let the ApiException propagate (or rethrow) instead of collapsing it to null, or return a tri-state / Optional so the caller can skip the change decision on API errors and only treat a genuinely empty-but-successful result as "service invalid".

KNOX-2900 - Re-enable service-based discovery filter
What changes were proposed in this pull request?
PollingConfigurationAnalyzer changes:
How was this patch tested?
Still needs to be tested.
Integration Tests