hawkbit/docs/clustering.md at master · eclipse-hawkbit/hawkbit · GitHub
Skip to content

clustering.md

Latest commit

 

History

History
134 lines (93 loc) · 5.45 KB

clustering.md

File metadata and controls

134 lines (93 loc) · 5.45 KB
Raw

Clustering

hawkBit is able to run in a cluster with some constraints. This guide provides insights into the basic concepts and how to setup your own cluster. You can find additional information in the hawkBit runtime’s README.

Big picture

Clustering Diagram

Events

Event communication between nodes is based on Spring Cloud Stream.
There are different binder implementations available. The hawkbit Update Server uses RabbitMQ binder. Every node gets its own queue to receive cluster events, the default payload is JSON.

Remote events are controlled by hawkbit.events.remote.enabled (default true). The update server (monolith) disables them, together with RabbitMQ, in its default configuration (hawkbit.events.remote.enabled=false and RabbitAutoConfiguration excluded) - enable both when running it in a cluster (see Base Setup).

Clustering Diagram

Event Channel Types in Spring Cloud Stream

Remote events in hawkBit are distributed through two distinct types of channels:

Fanout Event Channel
  • Every service instance listening to fanoutEventChannel receives a copy of every message, regardless of instance count.
  • Common for events that should be processed by each consumer independently
    • In-memory cache updates
    • Internal state propagation
    • Logging or auditing
  • Not recommended for scenarios where only one consumer should process an event (see serviceEventChannel for that).

Note: Every instance bound to this channel will get its own copy of the message.

Service Event Channel

The serviceEventChannel is used to ensure exclusive consumption of events across service instances. Only one instance per consumer group receives and processes each message, which is critical for non-idempotent or resource-sensitive operations.

  • Only one instance in a consumer group receives each message.
  • Ideal for external integrations, third-party API calls, or any task that must not be duplicated.
  • Load-balanced across instances within the same group.
Optional Protostuff for Spring cloud stream

The micro-service instances are configured to communicate via Spring Cloud Stream. Optionally, you could use Protostuff based message payload serialization for improved performance.

Note: If Protostuff is enabled it shall be enabled on all microservices!

Add/Uncomment to/in your application.properties :

spring.cloud.stream.default.content-type=application/binary+protostuff

Add to your pom.xml :

<dependency>
    <groupId>io.protostuff</groupId>
    <artifactId>protostuff-core</artifactId>
</dependency>
<dependency>
    <groupId>io.protostuff</groupId>
    <artifactId>protostuff-runtime</artifactId>
</dependency>

Caching

Every node is maintaining its own caches independent of the other nodes.
So there is no globally shared/synchronized cache instance within the cluster.

In order to keep nodes in sync, a TTL (time to live) can be set for all caches to ensure that after some time the cache is refreshed from the database.
The TTL is set by the property hawkbit.cache.ttl (default 10s, the monolith sets 0s - no expiration, since it is expected to run as a single instance).
Caches are Caffeine caches and could be fine-tuned with a Caffeine spec via hawkbit.cache.spec (all caches, default expireAfterWrite=${hawkbit.cache.ttl}) or hawkbit.cache.[<tenant>.]<name>.spec (cache / tenant specific).
See TenantAwareCacheManager.

Schedulers

hawkBit has multiple schedulers which run after a defined period of time.
They are disabled by default and enabled only on the nodes which explicitly set the properties (as the mgmt server and the monolith do in their default configuration):

  • hawkbit.autoassign.scheduler.enabled
  • hawkbit.rollout.scheduler.enabled
  • hawkbit.autocleanup.scheduler.enabled
  • hawkbit.tenant.hard-delete.scheduler.enabled

All enabled schedulers run on every node they are enabled on.

This has to be kept in mind e.g. if the scheduler executes critical code which has to be executed only once.

Known constraints

Denial-of-Service (DoS) filter

hawkBit owns the feature of guarding itself from DoS attacks: a DoS filter.
It reduces the maximum number of requests per second which can be configured for read and write requests.

This mechanism is only working for every node separately, i.e. in a cluster environment the worst-case behaviour would be that the maximum number of requests per second will be increased to its product if every request is handled by a different node.

The same constraint exists with the validator to check if a user tried too many logins within a defined period of time.
See DosFilter.