Connections V2: force-directed pivot + multi-field support · Issue #4084 · arkime/arkime · GitHub
Skip to content

Connections V2: force-directed pivot + multi-field support #4084

Description

@31453

Connections V2 — an evolution of the Connections SPIGraph that adds two capabilities on top of today's static bipartite view:

  1. Force-directed pivot (multi-hop): a new SPIGraph type where you expand nodes in place — click a node, pull in its connections, and the graph grows hop by hop (link-analysis / Maltego style).
  2. Multi-field / heterogeneous nodes: support multiple field types (host, ip, port, …) as node types in one graph, not just a single srcField → dstField pair.

Both reuse and extend the existing Connections model (configurable src/dst fields). Part of the Arkime 7 UI epic (#3926). Large.

Consolidates #147. #147 (awick, 2013) asked for two things on connections: (1) multi-field / heterogeneous node types (host, ip, port, …), and (2) a → b → c multi-hop chains "like Kibana Graph." Connections V2 is the single vehicle for both, so #147 is closed in favor of this issue. Phase 1 delivers the multi-hop pivot (accretive expand-in-place); the multi-field / heterogeneous half is Phase 2 (see Phasing / Out of scope).


Decisions (settled — Phase 1 pivot)

Area Decision
What it is A new SPIGraph graph type (pivot), alongside connections/heatmap/etc. — not a session-detail tool or standalone view.
Pivot interaction Expand in place (accretive). Acting on a node fetches that entity's neighbors and merges them into the existing graph (dedupe nodes/edges). The graph grows across multiple hops.
Node/edge model Same as Connections — homogeneous nodes from a configured srcField + dstField pair; edges = sessions between them. Reuses /api/connections directly. (Heterogeneous nodes are Phase 2.)
Code reuse Extract a shared force-graph engine from ConnectionsGraph.vue (~1,700 lines of D3); both Connections and Pivot render through it. Pivot adds the expand/merge/prune layer.
Performance ceiling Hard node cap (default ~500, configurable) + auto-prune. Expansions past the cap prune the lowest-weight / least-recently-touched nodes (pinned + just-expanded nodes protected).

How it relates to Connections (today)

Connections is already a force-directed SPIGraph type: self-contained ConnectionsGraph.vue, owns its /api/connections fetch, draws a bipartite graph of srcField → dstField (default source.ip → destination.ip, saved in user settings). Controls: session-count cap (length), minConn, weight metric, baselineDate compare, force strength, zoom, text size, lock, export PNG. Node values are clickable via the standard SessionField menu.

The gaps Connections V2 fills: Connections only ever shows src↔dst for the current query — no expansion — and only a single src/dst field pair. Its only node action is "hide node." Connections V2 keeps the same node/edge model but adds expand-in-place (Phase 1) and multiple heterogeneous node types (Phase 2).


The expand mechanic (Phase 1 crux)

  1. Initial graph = the current search's srcField → dstField connections — identical to the Connections type, rendered via the shared engine.
  2. Expand a node X → call /api/connections filtered on X (X.exp == value), same src/dst field pair, current time range.
    • This is what lets the graph reach entities not in the original query — that's the pivot.
    • (Open question: whether expand is "free" — X.exp == value alone — or AND-ed with the base expression. Proposed default: free pivot within the active time range.)
  3. Merge the response into the live graph:
    • /api/connections returns { nodes, links } where link source/target are array indices. The merge layer must remap those indices to global node ids and dedupe by node id (combine weights/sessions) and by edge (combine values).
    • Mark X as expanded (so it's visually distinct and not trivially re-expanded).
  4. Prune if over the cap (below).

Shared force-graph engine (the refactor)

Extract from ConnectionsGraph.vue into a reusable engine (ForceGraph.vue / useForceGraph composable):

Moves into the engine (shared): D3 force simulation (link/charge/center/collision), SVG node+link render, drag, zoom/pan, text-size, lock/unlock, export PNG, Node/Link popups + legend. Takes nodes[] + links[] + render config; emits node/link interactions.

Stays type-specific:

  • Connections: src/dst field controls, minConn, weight, baselineDate compare, its /api/connections load.
  • Pivot: expand → merge → prune, expanded-node marking, pin-to-protect, the Expand action added to NodePopup.

⚠️ Safety net asymmetry. api-connections.t snapshot-tests the /api/connections backend, which Phase 1 Pivot reuses unchanged — so the backend stays covered. But the engine extraction is frontend D3 with no automated tests → Connections must be visually re-verified after the refactor (it's a behavior-preserving change to a proven 1,700-line file).


Performance: hard cap + prune

  • Global cap on total nodes (default ~500; configurable via a control and/or user setting).
  • When an expansion would exceed the cap, prune unpinned, non-just-expanded nodes by lowest weight / least-recently-touched until under the cap.
  • Protect: pinned/locked nodes (reuse the existing lock concept) and the nodes from the current expansion.
  • Feedback: show "showing N of M" and surface what was pruned (so removal isn't silent — see open questions).
  • Per-expansion volume is still bounded by the Connections length cap before the global prune runs.

SPIGraph registration

  • Add 'pivot' to graphTypeOptions in Spigraph.vue (['default','heatmap','pie','table','treemap','sankey','connections','pivot']).
  • i18n spigraph.graphType-pivot (e.g. "Pivot") across all 11 locales.
  • Dispatch like Connections: render <arkime-pivot-graph> when spiGraphType === 'pivot'; it owns its own data so SPIGraph's periodic refresh is stopped (add the matching branch to changeSpiGraphType + the connections-style guards in the load/route watchers).
  • Reuse the Connections src/dst field controls (Pivot uses the same field pair), via the existing controls-anchor pattern.

Components (Phase 1)

  • New viewer/.../connections/ForceGraph.vue (+ optional composable) — extracted shared D3 engine.
  • Refactor ConnectionsGraph.vue — render via ForceGraph; keep its connections-specific controls/data.
  • New viewer/.../spigraph/PivotGraph.vue — feeds ForceGraph; owns expand/merge/prune + pin state.
  • Change NodePopup.vue — add an Expand action (emit expand); Connections can hide it, Pivot wires it.
  • Change Spigraph.vue — register + dispatch the pivot type.
  • i18n — graphType-pivot + any pivot-specific control strings.

Phasing

Phase 1 — multi-hop pivot (expand-in-place):

  1. Extract the shared engine from ConnectionsGraph; refactor Connections onto it; visually verify Connections is unchanged (backend api-connections.t still green).
  2. Register the pivot type; render the initial connections graph through the shared engine.
  3. Expand-in-place — Expand action → /api/connections filtered on the node → merge (index remap + dedupe) → mark expanded.
  4. Cap + prune + pinning + pruned-node feedback.
  5. Polish — expanded/pinned visual states, expand-scope behavior, UI review, i18n.

Phase 2 — multi-field / heterogeneous nodes (the other half of #147):
6. Allow multiple field types (host, ip, port, …) as node types in one graph, so a node can be a host, an IP, a port, etc. — and edges connect across types (e.g. ip → port, host → ip).
7. This resurfaces the connections backend work #147 flags: the current /api/connections is callback-based and single-pair; heterogeneous/multi-field support likely needs the callbacks → async/await rewrite of /api/connections (which Phase 1 deliberately dodges by reusing the endpoint as-is).
8. Heterogeneous session nodes (a session as a first-class node type) fold in here.


Out of scope for Phase 1

  • Heterogeneous / cross-field pivot and session nodes — deferred to Phase 2 above (the multi-field half of support multi fields on connections tab #147). Reviving it resurfaces the /api/connections callbacks→async/await rewrite, which Phase 1 dodges by reusing the endpoint as-is.
  • Shareable / URL-persisted pivot sessions (the accreted graph state isn't encoded in the query).
  • A new backend endpoint for Phase 1 — /api/connections is reused as-is (Phase 2 may extend it).

Remaining open questions (small)

  • Expand scope: free pivot (X.exp == value alone) vs. AND-ed with the current base expression. (Proposed: free, within the active time range.)
  • Expand trigger: double-click node, a NodePopup "Expand" button, or both.
  • Merge dedupe semantics: sum vs. max for node weight/sessions when a node reappears; how to combine duplicate edge values.
  • Prune metric + notice: lowest-weight vs. least-recently-touched (vs. combined), and whether to show a "pruned N nodes" notice so removal isn't silent.
  • Default cap + where configured: the ~500 number, and whether it's a graph control, a user setting, or both.
  • Multi-field model (Phase 2): how heterogeneous node types + cross-type edges are configured and queried, and the shape of the /api/connections rewrite.

Activity

  1. added this to the 7.0 milestone on Jun 30, 2026
  2. self-assigned this
    on Jun 30, 2026
  3. added theissue type on Jun 30, 2026
  4. 31453 commented on Jun 30, 2026

    @31453
    CollaboratorAuthor

    This is the modern implementation of the long-standing request in #147 ("a → b → c like Kibana Graph"). v1 here delivers that multi-hop half via accretive expand-in-place.

    The multi-field / heterogeneous-node half of #147 (host/ip/port as different node types) is intentionally out of scope for v1 — it's tracked as phase 2 in #147. The deferred "cross-field pivot" option in the scope above is that work.

  5. 31453 commented on Jun 30, 2026

    @31453
    CollaboratorAuthor

    Notes from Andy: rewrite a new connections v2. rewrite the api to not include the ntop.

  6. changed the title [-]Force-directed pivot graph[/-] [+]Connections V2: force-directed pivot + multi-field support[/+] on Jul 1, 2026
  7. hiSandog commented on Jul 4, 2026

    @hiSandog

    The pruning and merge rules seem like the highest-risk UX part of Phase 1. It may be worth tracking hop depth, last-expanded time, and pinned state separately in the graph model so auto-prune can be predictable: never remove pinned nodes, avoid removing the node just expanded, and make it visible when older low-weight nodes were pruned due to the cap. That would make multi-hop exploration feel much less surprising when the graph approaches the default node ceiling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

  • Status
    Todo

Relationships

None yet

Development

No branches or pull requests

Issue actions