KNOX-2900: KNOX-2899 followup - reenable service based discovery filter by bonampak · Pull Request #1395 · apache/knox · GitHub
Skip to content

KNOX-2900: KNOX-2899 followup - reenable service based discovery filter - #1395

Draft
bonampak wants to merge 6 commits into
apache:masterfrom
bonampak:KNOX-2900
Draft

bonampak wants to merge 6 commits into
apache:masterfrom
bonampak:KNOX-2900

Conversation

@bonampak

Copy link
Copy Markdown
Contributor

KNOX-2900 - Re-enable service-based discovery filter

What changes were proposed in this pull request?

PollingConfigurationAnalyzer changes:

  • Ignore excluded service types in CM config monitor event relevance
  • Ignore excluded role types in scale events
  • Prevent rediscovery of services that remained in an invalid config state (use the same workflow for service configuration data discovery as ClouderaManagerServiceDiscovery)
  • Re-enable per-descriptor service filtering in CM discovery (cluster configuration cache data is merged after each topology discovery)

How was this patch tested?

Still needs to be tested.

Integration Tests

# Conflicts:
#	gateway-discovery-cm/src/main/java/org/apache/knox/gateway/topology/discovery/cm/ServiceModelGeneratorsHolder.java
#	gateway-discovery-cm/src/test/java/org/apache/knox/gateway/topology/discovery/cm/ServiceModelGeneratorsHolderTest.java
@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown

…figurationAnalyzer

getReferencedServiceTypes and clusterReferencesExist swallowed descriptor
parse/read errors silently and only caught IOException. A malformed or
unreadable descriptor could silently stop rediscovery or discard a live
cluster's cached configuration.

Both now catch broadly and log the offending file. getReferencedServiceTypes skips the unreadable descriptor and keeps filtering on the descriptors it could read (an unparseable descriptor can't be regenerated until it is fixed, so it can't be the target of a useful rediscovery now).
clusterReferencesExist instead assumes a reference remains when a descriptor can't be read, so an unreadable descriptor never causes the monitored cluster's baseline to be discarded.
…de when determining referenced services, since SimpleDescriptorHandler never regenerates them (using the gateway-site override list, not the descriptor's own read-only field).

Both now catch broadly and log the offending file. getReferencedServiceTypes skips the unreadable descriptor and keeps filtering on the descriptors it could read (an unparseable descriptor can't be regenerated until it is fixed, so it can't be the target of a useful rediscovery now).
clusterReferencesExist instead assumes a reference remains when a descriptor can't be read, so an unreadable descriptor never causes the monitored cluster's baseline to be discarded.

@smolnar82 smolnar82 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found 2 blockers, and the following efficiency issues:

1. every descriptor is parsed twice per polling cycle

File: PollingConfigurationAnalyzer.java (getReferencedServiceTypes() / clusterReferencesExist())

clusterReferencesExist() and getReferencedServiceTypes() each independently loop over ts.getDescriptors() and re-parse every descriptor file via SimpleDescriptorFactory.parse() (around lines 423 and 602). With N descriptors on disk, each polling cycle now parses every file twice per cluster — doubling filesystem reads + JSON parsing — and duplicates near-identical IOException handling.

Fix direction: merge the two into a single pass over ts.getDescriptors() that returns both "does any descriptor reference this (address, cluster)" and "the set of referenced service names", so each descriptor is parsed once.

2. CORE_SETTINGS is re-fetched from CM once per relevant service type

File: PollingConfigurationAnalyzer.java (getCoreSettingsConfig())

getCoreSettingsConfig() issues a fresh readServices(clusterName, summary) plus a readServiceConfig() round-trip every time getCurrentServiceConfiguration() runs — i.e. once per distinct relevant service type per polling cycle. Cluster discovery, by contrast, computes CORE_SETTINGS once for the whole cluster (coreSettingsConfig(...)).

Failure path: a polling cycle with events for three service types (e.g. HDFS, HIVE_ON_TEZ, YARN) makes three separate readServices() + readServiceConfig() calls just to re-derive the same CORE_SETTINGS value, adding avoidable CM API load on every interval.

Fix direction: compute CORE_SETTINGS once per polling pass and reuse it across all services handled in that pass (cache it for the duration of the cycle).

3.: role-type exclusion config is cached for the analyzer's lifetime while service-type exclusion is read fresh

File: PollingConfigurationAnalyzer.java (isExcludedRoleType() / isExcludedServiceType())

isExcludedRoleType() lazily builds and then permanently caches its TypeNameFilter (roleTypeFilter) from gatewayConfig on first use, whereas the adjacent isExcludedServiceType() re-reads gatewayConfig.getClouderaManagerServiceDiscoveryExcludedServiceTypes() fresh on every call.

Failure path: an operator updates gateway.cloudera.manager.service.discovery.excluded.role.types on a running gateway (without restarting the CM configuration monitor thread). isExcludedServiceType honors an equivalent excluded-service-types change immediately, but isExcludedRoleType keeps using the stale filter built at first invocation and silently ignores the update.

Fix direction: pick one policy for both. Either read both config values fresh per call, or resolve both once at construction — but don't split the behavior between the two sibling checks introduced in this PR.

roleConfigs.put(role, configList);
}
currentConfig = new ServiceConfigurationModel(svcConfig, roleConfigs);
final ApiService apiService = new ApiService().name(service).type(serviceType);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To reuse the discovery workflow, getCurrentServiceConfiguration() reconstructs the ApiService / ApiRole / config objects it feeds into ServiceModelFactory.generateServiceModels(). But this reconstruction doesn't populate the same fields that real discovery (ClouderaManagerServiceDiscovery.discoverService) hands to generators. Any ServiceModelGenerator that dereferences a field the reconstruction leaves null throws an NPE inside generateService(). This is not an edge case — it hits the majority of generators:

  • role.getHostRef().getHostname() — used by 34 of 51 generators (OozieServiceModelGenerator, SolrServiceModelGenerator, HBase, Impala, and most others). NPEs if the reconstructed role's hostRef (or its hostname) isn't populated exactly as discovery populates it.
  • service.getClusterRef().getClusterName() — used by YarnUIServiceModelGenerator and JobHistoryUIServiceModelGenerator. NPEs because the synthetic ApiService sets only name / type, no clusterRef.
  • Any other field a given generator's handles() / generateService() reads that the reconstruction omits.

Failure path & blast radius: a start/restart/scale event for almost any discoverable service (YARN, Oozie, Solr, HBase, …) → hasConfigChangedgetCurrentServiceConfigurationServiceModelFactory.generateServiceModels → the generator's generateService()NPE. The NPE is not an ApiException, so the method's own try/catch doesn't catch it; it propagates to monitorClusterConfigurationChanges' outer catch(Exception), aborting the entire polling cycle for all clusters. Because the triggering event is never marked processed, the same NPE recurs every polling interval — PCA is effectively dead for that gateway.

Root cause / fix direction: the reconstruction in getCurrentServiceConfiguration() diverges from how ClouderaManagerServiceDiscovery builds these objects. Rather than hand-rebuilding ApiService / ApiRole from readServiceConfig, PCA should obtain the model inputs through the same code path discovery uses (the shared component introduced here), so every field a generator may read is populated identically. A unit test that runs each registered generator against PCA-reconstructed inputs would have caught this and would guard against regressions as new generators land.

Comment on lines +367 to +371

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new branch treats getCurrentServiceConfiguration() == null as "the service is now invalid / disabled" and forces re-discovery. But that method also returns null on any ApiException - network blip, auth failure, transient 5xx:

} catch (ApiException e) {
  log.clouderaManagerConfigurationAPIError(e);
}
return currentConfig; // still null on API error

So a null return is ambiguous: it can mean either "config genuinely produced no model" or "couldn't reach CM". hasConfigChanged's else branch (prior config present, currentConfig == null) can't tell them apart and unconditionally sets configHasChanged = true, logs serviceDisabled, and triggers a full re-discovery.

Failure path: CM API has a transient outage while computing the current config for a service that has a valid prior baseline → getCurrentServiceConfiguration catches the ApiException and returns nullhasConfigChanged forces re-discovery → repeats every polling cycle until CM recovers.

Regression: previously an API-error null was a harmless no-op (configHasChanged stayed false). This PR turns a transient CM error into a repeating full-cluster rediscovery storm — the same failure class KNOX-2900 set out to eliminate.

Fix direction: distinguish "no model produced" from "CM unreachable". Options: let the ApiException propagate (or rethrow) instead of collapsing it to null, or return a tri-state / Optional so the caller can skip the change decision on API errors and only treat a genuinely empty-but-successful result as "service invalid".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants