[KYUUBI #7759] Reduce the connection limit count when session creation fails by Chaho12 · Pull Request #7760 · apache/kyuubi · GitHub
Skip to content

[KYUUBI #7759] Reduce the connection limit count when session creation fails - #7760

Open
Chaho12 wants to merge 1 commit into
apache:masterfrom
Chaho12:session-limiter-leak
Open

Chaho12 wants to merge 1 commit into
apache:masterfrom
Chaho12:session-limiter-leak

Conversation

@Chaho12

@Chaho12 Chaho12 commented Sep 22, 2026 •

Copy link
Copy Markdown

Why are the changes needed?

KyuubiSessionManager.openSession takes a count from the session limiter before the session is
opened, and closeSession is the only place that gives one back.

SessionManager.openSession calls createSession outside the try whose catch calls
closeSession, and AbstractSession.normalizedConf is a val, so a session whose conf fails
validation dies in the constructor and is never registered in handleToSession. closeSession
never runs for it, and the count is kept for good.

Once as many opens have failed as kyuubi.server.limit.connections.per.user allows, every further
open from that user is rejected by the limiter although the user holds no session, until the
server restarts. kyuubi.server.limit.connections.per.ipaddress leaks the same way.

Ordinary client configuration reaches this on a server that follows the documented
kyuubi.session.conf.restrict.list guidance: spark.sql.extensions is in the example restrict
list in the session configuration docs,
and a client that keeps sending it locks its own account out. We hit this in production - three
server nodes each rejected 20 opens for the restricted key, after which every open from that user
failed with the connection-limit message while kyuubi_connection_opened stayed 0, and a rolling
restart was the only way to recover.

Affects master and 1.11.1. Batch sessions are not affected: openBatchSession takes its count
after the session is built and inside the try that closes it.

close #7759

How was this patch tested?

Added KyuubiSessionManagerLimiterSuite, which sets kyuubi.server.limit.connections.per.user=2
and a restrict list, then opens limit + 2 sessions carrying the restricted key and asserts every
attempt is still rejected by the restrict list rather than by the limiter.

build/mvn test -pl kyuubi-server -am -Dtest=none \
    -DwildcardSuites=org.apache.kyuubi.session.KyuubiSessionManagerLimiterSuite

Passes with the change; with the change reverted it fails on the third attempt with
"Connection limit per user reached (user: user limit: 2)" did not contain "spark.sql.extensions is a restrict key", which is the production symptom.

Was this patch assisted by generative AI tooling?

Assisted-by: Claude Opus 5

@Chaho12 Chaho12 changed the title [KYUUBI ##7759] Reduce the connection limit count when session creation fails [KYUUBI #7759] Reduce the connection limit count when session creation fails Sep 22, 2026
@Chaho12
Chaho12 force-pushed the session-limiter-leak branch from e1dbf13 to ebbc2c9 Compare September 22, 2026 09:13
@wForget

This comment was marked as outdated.

@Chaho12

Chaho12 commented Sep 30, 2026

Copy link
Copy Markdown
Author

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused lifecycle fix correctly avoids double-decrementing and is covered by a regression test.

0 open findings

What changed in this PR

Fixes #7759 by releasing connection-limit capacity when interactive session construction fails before registration.

Changes:

  • Decrements the session limiter on constructor failure.
  • Adds a regression test covering repeated restricted-configuration failures.
File Description
KyuubiSessionManager.scala Restores limiter counts after session-construction failures.
KyuubiSessionManagerLimiterSuite.scala Verifies failed attempts do not exhaust the per-user limit.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@wForget wForget left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Chaho12

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Session limiter count is not returned when session creation fails, locking the user out

4 participants