unified: Add Swift node type schema generator · microsoft/codeql@c6d4f8d · GitHub
Skip to content

Commit c6d4f8d

Browse files
tausbnCopilot
tausbn
and
Copilot
committed
unified: Add Swift node type schema generator
Adds the SwiftSyntax-based schemagen tool and a manual wrapper for regenerating swift_node_types.yml after dependency updates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 9567f1a commit c6d4f8d

9 files changed

Lines changed: 311 additions & 2 deletions

File tree

‎unified/AGENTS.md‎

Lines changed: 4 additions & 1 deletion

‎unified/extractor/src/languages/swift/adapter.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,9 @@ fn parse_range(node: &Value) -> Option<Range> {
261261
})
262262
}
263263

264-
/// The authoritative swift-syntax input node-types schema.
264+
/// The authoritative swift-syntax input node-types schema, generated from
265+
/// swift-syntax by `swift-syntax-rs/schemagen` (run
266+
/// `unified/scripts/regenerate-node-types.sh` to refresh it).
265267
/// [`json_to_ast`] seeds every parse with the schema built from this,
266268
/// pre-registering every input kind and field so rule matching never references
267269
/// a name absent from a given file's tree.

‎unified/extractor/swift_node_types.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# GENERATED from swift-syntax by unified/swift-syntax-rs/schemagen.
2+
# Do not edit; run unified/scripts/regenerate-node-types.sh instead.
13
supertypes:
24
decl:
35
- accessorDecl

‎unified/scripts/regenerate-node-types.sh‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/bin/bash
2+
# Regenerate `extractor/swift_node_types.yml`, the schema describing the shape
3+
# of the trees produced by `swift_syntax_rs::parse_to_json`, from swift-syntax
4+
# itself.
5+
#
6+
# Run this after changing the pinned swift-syntax version, and review the diff:
7+
# a new or renamed node kind generally means the mapping in
8+
# `extractor/src/languages/swift/swift.rs` needs attention too.
9+
#
10+
# This needs a local Swift toolchain (see `swift-syntax-rs/.swift-version` for
11+
# the pinned version). The schema it derives from lives in `SyntaxSupport`, a
12+
# target of swift-syntax's separate `CodeGeneration` package: it is not a
13+
# product of swift-syntax, and Bazel's swift-syntax module does not export its
14+
# sources, so there is no way to depend on it directly.
15+
set -euo pipefail
16+
IFS=$'\n\t'
17+
18+
root=$(cd "$(dirname "$0")/.." && pwd)
19+
swift_syntax_rs_dir="$root/swift-syntax-rs"
20+
ffi_dir="$swift_syntax_rs_dir/swift"
21+
schemagen_dir="$swift_syntax_rs_dir/schemagen"
22+
output="$root/extractor/swift_node_types.yml"
23+
24+
if ! command -v swift >/dev/null 2>&1; then
25+
echo "error: Swift is required; install the version pinned in $swift_syntax_rs_dir/.swift-version." >&2
26+
exit 1
27+
fi
28+
29+
# Codespaces sets `safe.bareRepository=explicit` through environment-based Git
30+
# configuration, which prevents SwiftPM from using its cached bare dependency
31+
# repositories. Relax only that injected setting, and only for Swift
32+
# subprocesses, as `swift-syntax-rs/build.rs` does for local Cargo builds.
33+
run_swift() {
34+
if [[ ${GIT_CONFIG_KEY_0:-} == "safe.bareRepository" ]]; then
35+
GIT_CONFIG_VALUE_0=all swift "$@"
36+
else
37+
swift "$@"
38+
fi
39+
}
40+
41+
# `schemagen` takes swift-syntax as a path dependency on this checkout, so that
42+
# the schema describes exactly the version the parser links.
43+
echo "Resolving swift-syntax..." >&2
44+
(
45+
cd "$ffi_dir"
46+
run_swift package resolve >&2
47+
)
48+
checkout="$ffi_dir/.build/checkouts/swift-syntax"
49+
syntax_support="$checkout/CodeGeneration/Sources/SyntaxSupport"
50+
if [[ ! -d $syntax_support ]]; then
51+
echo "error: $syntax_support not found after resolving swift-syntax." >&2
52+
exit 1
53+
fi
54+
55+
# Refresh rather than merge, so that sources deleted upstream do not linger.
56+
rm -rf "$schemagen_dir/Sources/SyntaxSupport"
57+
cp -R "$syntax_support" "$schemagen_dir/Sources/SyntaxSupport"
58+
59+
echo "Generating $output..." >&2
60+
# Generate to a temporary file first: redirecting straight into `$output` would
61+
# truncate the existing schema before the build has even run, leaving nothing
62+
# behind if it fails.
63+
tmp=$(mktemp)
64+
trap 'rm -f "$tmp"' EXIT
65+
(
66+
cd "$schemagen_dir"
67+
run_swift run schemagen
68+
) > "$tmp"
69+
if [[ ! -s $tmp ]]; then
70+
echo "error: schemagen produced no output; $output left unchanged." >&2
71+
exit 1
72+
fi
73+
mv "$tmp" "$output"
74+
chmod 644 "$output"
75+
echo "Regenerated $output" >&2

‎unified/swift-syntax-rs/README.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,20 @@ cargo test
150150

151151
The first build compiles `swift-syntax` and can take several minutes.
152152

153+
## Regenerating the extractor node types
154+
155+
After updating the pinned swift-syntax version, regenerate the unified
156+
extractor's input schema:
157+
158+
```sh
159+
../scripts/regenerate-node-types.sh
160+
```
161+
162+
The script uses swift-syntax's authoritative `SyntaxSupport` definitions and
163+
requires the local Swift toolchain pinned by [`.swift-version`](.swift-version).
164+
Review the resulting `extractor/swift_node_types.yml` diff alongside the Swift
165+
mapping rules. See [`schemagen/README.md`](schemagen/README.md) for details.
166+
153167
## Building with Bazel (CI)
154168

155169
CI builds this crate hermetically with Bazel. A Swift toolchain is downloaded

‎unified/swift-syntax-rs/schemagen/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
/.build
2+
# Copied from swift-syntax's CodeGeneration package by
3+
# `unified/scripts/regenerate-node-types.sh`; not ours to vendor.
4+
/Sources/SyntaxSupport

‎unified/swift-syntax-rs/schemagen/Package.swift‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
// swift-tools-version:5.9
2+
import PackageDescription
3+
4+
// `schemagen` regenerates `unified/extractor/swift_node_types.yml`, the input
5+
// schema describing the shape of the trees produced by
6+
// `swift_syntax_rs::parse_to_json`. Run it through
7+
// `unified/scripts/regenerate-node-types.sh`, which stages the sources this
8+
// package needs; see `README.md` for the details.
9+
//
10+
// The tools version is deliberately older than the FFI package's: it selects
11+
// the Swift 5 language mode, and `SyntaxSupport` (see below) is not clean under
12+
// Swift 6 strict concurrency because its node tables are non-Sendable globals.
13+
let package = Package(
14+
name: "schemagen",
15+
platforms: [
16+
// Matches the FFI package: swift-syntax 603 requires macOS 10.15.
17+
.macOS(.v10_15),
18+
],
19+
dependencies: [
20+
// Deliberately a path dependency on the checkout the neighbouring FFI
21+
// package resolved, rather than a second URL/exact pin: the schema has
22+
// to describe the very swift-syntax that the parser links, and a
23+
// single pin cannot drift from itself.
24+
.package(name: "swift-syntax", path: "../swift/.build/checkouts/swift-syntax"),
25+
],
26+
targets: [
27+
// `SyntaxSupport` is a target of swift-syntax's separate
28+
// `CodeGeneration` package, not a product of swift-syntax itself, so it
29+
// cannot be depended on directly. The regeneration script copies its
30+
// sources here (the directory is git-ignored) and this target builds
31+
// them as if they were our own.
32+
.target(
33+
name: "SyntaxSupport",
34+
dependencies: [
35+
.product(name: "SwiftSyntax", package: "swift-syntax"),
36+
.product(name: "SwiftSyntaxBuilder", package: "swift-syntax"),
37+
]
38+
),
39+
.executableTarget(name: "schemagen", dependencies: ["SyntaxSupport"]),
40+
]
41+
)

‎unified/swift-syntax-rs/schemagen/README.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# schemagen
2+
3+
Generates [`unified/extractor/swift_node_types.yml`][schema], the schema that
4+
describes the shape of the trees produced by `swift_syntax_rs::parse_to_json`.
5+
The extractor seeds every parse with it, so rule matching never refers to a
6+
node kind or field that swift-syntax can produce but the schema does not know.
7+
8+
Run it through the script, which stages the sources described below:
9+
10+
```console
11+
$ unified/scripts/regenerate-node-types.sh
12+
```
13+
14+
Do this after changing the pinned swift-syntax version, and read the resulting
15+
diff: a new or renamed node kind usually means the mapping in
16+
[`swift.rs`][mapping] needs attention too.
17+
18+
This requires the local Swift toolchain pinned by
19+
[`.swift-version`](../.swift-version).
20+
21+
## Why the sources are copied in
22+
23+
The schema is derived from `SyntaxSupport`, the module that describes
24+
swift-syntax's own syntax tree. This is the same description swift-syntax
25+
generates itself from, and is therefore authoritative in a way that observing
26+
parser output never would be. The runtime `SwiftSyntax` module is not a
27+
substitute: its `SyntaxNodeStructure` exposes layout as key paths, without the
28+
field names, optionality, and base-kind relationships this schema records.
29+
30+
`SyntaxSupport` is awkward to depend on, though. It is a target of
31+
`CodeGeneration`, a package inside the swift-syntax repository that is
32+
separate from swift-syntax itself, and it is not one of that package's
33+
products. SwiftPM can only depend on products, and Bazel's swift-syntax module
34+
does not export the `CodeGeneration` sources, so neither build system can
35+
reach it directly.
36+
37+
The regeneration script therefore copies those sources out of the resolved
38+
swift-syntax checkout into `Sources/SyntaxSupport`, where this package builds
39+
them as its own. That directory is git-ignored and refreshed on every run, so
40+
it always matches the pinned version rather than drifting as a stale vendored
41+
copy would.
42+
43+
For the same reason this package takes swift-syntax as a path dependency on the
44+
checkout the neighbouring FFI package resolved, rather than declaring a second
45+
pinned dependency of its own. The schema must describe exactly the
46+
swift-syntax version linked by `swift-syntax-rs`.
47+
48+
## What is filtered out
49+
50+
The schema describes the JSON the extractor's adapter receives, not
51+
swift-syntax's tree verbatim, so `main.swift` mirrors what
52+
[`adapter.rs`][adapter] does:
53+
54+
- Abstract base kinds become `supertypes:` entries rather than node kinds.
55+
- Collection nodes are dropped, and a collection-typed child is recorded as
56+
its element kinds, because the adapter elides collections into JSON arrays.
57+
- `unexpectedBeforeX` and `unexpectedBetweenXAndY` error-recovery children are
58+
dropped; no rule matches them. This filters on the child name:
59+
`unexpectedCodeDecl` is a real node kind and is retained.
60+
- Token-typed children become the synthetic `_token` kind. Only the varying
61+
token kinds whose text is not implied by the kind, listed in
62+
`VARYING_TOKEN_KINDS` in `adapter.rs`, are emitted as kinds of their own.
63+
Fixed tokens are anonymous and keyed by their text, so no rule can name
64+
them.
65+
66+
Setting `EMIT_SUPERTYPES=0` omits the `supertypes:` section, which can be useful
67+
when diffing two versions for kind and field changes alone.
68+
69+
[schema]: ../../extractor/swift_node_types.yml
70+
[mapping]: ../../extractor/src/languages/swift/swift.rs
71+
[adapter]: ../../extractor/src/languages/swift/adapter.rs

‎unified/swift-syntax-rs/schemagen/Sources/schemagen/main.swift‎

Lines changed: 97 additions & 0 deletions

0 commit comments

Comments
 (0)