iframe-proxy

Otzie2023 · 2026-08-22T13:56:42Z

Support non-ASCII identifiers following UAX #31

Fixes #3736. Closes #737 as a side effect (see below).

Identifiers are currently limited to [A-Za-z_$][A-Za-z0-9_]*; anything else is
rejected by the lexer without a message. This adds an opt-in identifier syntax
based on Unicode Standard Annex #31, behind --enable=unicode-identifiers.

The character set

ID_Start    := (XID_Start + { U+0024, U+005F }) & Identifier_Status=Allowed
ID_Continue := XID_Continue & Identifier_Status=Allowed

XID_Start/XID_Continue is the UAX #31 default, and the same grammar C++23
(P1949R7) and Rust 1.53 use. U+0024 DOLLAR SIGN has to be added explicitly
because it is Sc, not XID, and OpenSCAD needs it for $fn, $fa and the
rest; U+005F LOW LINE is added for compatibility with the current syntax.

The intersection with Identifier_Status=Allowed, the general security profile
of UTS #39, is the part that goes beyond C++ and Rust's grammar. It addresses
the objection raised in this issue directly: plain XID does not keep invisible
characters out, because variation selectors U+FE00–FE0F are Mn and therefore
in XID_Continue. The UTS #39 profile removes them, along with obsolete
scripts, technical symbols, and canonical duplicates.

Emoji, ZWJ/ZWNJ, and mathematical operators (∑ ∏ ∫ ∂ ∇ √ ∞ °) are outside the
set. The Greek letters people usually reach for — π, Δ, Σ, Ω, θ, λ, φ — are in
XID_Start already, so most of the "special math symbols" wish from the 2021
comment is covered without a bespoke allowlist.

Normalisation

Identifiers are normalised to NFC in the lexer, and validated after
normalisation. P1949 instead makes non-NFC source ill-formed; normalising seemed
the better fit here, because ä as U+00E4 and as U+0061 U+0308 are
indistinguishable in the editor and in a diff, and most OpenSCAD files are
edited in the bundled editor rather than in a toolchain that can enforce NFC on
save. Rust made the same call.

Validating the normalised form also accepts the canonical singletons: U+2126
OHM SIGN, U+212A KELVIN SIGN and U+212B ANGSTROM SIGN are excluded from the UTS
#39 profile precisely because NFC folds them into U+03A9, U+004B and U+00C5, so
after normalisation they are ordinary identifier characters and Ω typed either
way is one variable. Roughly 1,000 code points fall into that category,
including the composition exclusions in Devanagari, Bengali and Gurmukhi that
several input methods emit by default.

The generator verifies that the profile is closed under NFC before writing the
tables, so normalising an accepted identifier can never turn it into a rejected
one.

g_utf8_normalize() comes from glib, which is already a required dependency.

The lexer tables

The 2021 discussion stalled on the scanner tables blowing up. That happens when
the code point ranges are encoded as UTF-8 byte patterns in the rules — Unicode
16 has 767 XID_Start and 1400 XID_Continue ranges, which expand to hundreds
of byte alternatives each. Measured against this tree with flex 2.6.4:

The last row does not compile with flex's defaults at all (Definition value for {XIDS} too long), and after splitting the pattern across 61 definitions it uses
12,502 of the 13,000 default NFA slots.

This PR keeps Unicode out of the DFA. The rules match any non-ASCII sequence
using a variant of the {UNICODE} macro that was already in lexer.l, and the
classification happens in the action as a binary search over a generated table:

U2ID    \xc2[\x80-\x9f\xa1-\xbf]|[\xc3-\xdf]{U}
U3ID    [\xe0-\xee]{U}{U}|\xef[\x80-\xba\xbc-\xbf]{U}|\xef\xbb[\x80-\xbe]
UNICODEID {U2ID}|{U3ID}|{U4}{U}{U}{U}

The exclusions carved out of {UNICODE} are U+00A0 and U+FEFF, which lexer.l
treats as whitespace; without that, a<NBSP>= 1 would lex as a single token and
nbsp-utf8-test.scad would break.

The ASCII rule {IDSTART}{IDREST}* is kept unchanged and placed before the new
one. Both match a pure-ASCII identifier equally long, so flex resolves the tie
in favour of the earlier rule and ASCII input never reaches the new code path —
no runtime check needed. 0x1F, 1a and the deprecated digit-leading form are
unaffected.

Because {UNICODEID} is a coarse approximation, it also accepts overlong forms,
surrogates and values above U+10FFFF; the action validates the encoding with
g_utf8_validate() before normalising. Malformed input now reports why it was
rejected instead of failing silently, which is an improvement even for people
who never write a non-ASCII identifier.

The blanket {UNICODE} reject rule in INITIAL becomes unreachable and is
removed — flex confirms this with rule cannot be matched if it is left in.

Tables

src/core/UnicodeIdentifierTables.h is generated and checked in, the way GCC
does with ucnid.h, so there is no build-time dependency on the UCD.
scripts/generate-unicode-identifier-tables.py regenerates it from
DerivedCoreProperties.txt and IdentifierStatus.txt; it refuses to run if the
two files are from different Unicode versions. Current tables: Unicode 16.0.0,
305 start ranges and 380 continue ranges, 5,480 bytes. ASCII is deliberately not
tabulated.

Editor highlighting

src/gui/ScadLexer.cc needs a matching change. The editor uses a separate
lexertl-based lexer whose identifier rules are ASCII-only, so a non-ASCII byte
splits an identifier into several tokens with different styles:

VARIABLE   <geh>   TEXT <\xc3> TEXT <\xa4>  VARIABLE <use>
SPECIALVAR <$wandst> TEXT <\xc3> TEXT <\xa4> VARIABLE <rke>

The second line is the visible one: the tail of $wandstärke falls back from
the special-variable colour to the ordinary variable colour. The rules now
accept any byte outside ASCII:

rules_.push("[a-zA-Z0-9_\\x80-\\xff]+", evariable);
rules_.push("[$][a-zA-Z0-9_\\x80-\\xff]+", especialVariable);

That is deliberately coarser than the grammar in lexer.l — the editor lexer
only drives highlighting, and the parser reports the code points that are not
actually allowed. One cosmetic consequence: a<NBSP>= 1 now highlights as a
single identifier token, though it still parses as two.

Other code paths

I traced the other places that produce identifier names:

-D/--D is appended to the source text (openscad.cc) and goes through the
normal lexer, so it is covered.
Customizer parameter names come from assignment->getName(), i.e. from the
AST, so they are covered too.
customizer/comment_lexer.l only lexes annotation bodies; the parameter name
is matched by source location, so it needs no change.
ParameterSet::readFile() uses parameter names as JSON keys and could see a
non-NFC key from an older file or an external tool. Not addressed here, since
it cannot happen until non-ASCII identifiers are actually in use — happy to
add it to this PR if you'd prefer.

Identifiers are plain std::string through Lookup, Assignment and
Context::lookup_variable, so normalising at lex time means nothing downstream
changes.

Issue #737

µm = 0.001 * mm; from 2014 now produces
Parser error: Identifier cannot start with U+00B5 instead of failing silently.
U+00B5 MICRO SIGN is Restricted in UTS #39 because it is confusable with
U+03BC GREEK SMALL LETTER MU and NFC does not fold the two. μm with the Greek
letter works. This is also the worked example in Rust's uncommon_codepoints
lint.

Not included

Confusable and mixed-script detection (UTS Infobox #39 §4/§5). Worth having eventually
as a warning, along the lines of Rust's confusable_idents and
mixed_script_confusables, but it is separable and should not gate the
grammar change.
The deprecated digit-leading identifier form stays ASCII-only.

Feature flag

Everything is behind Feature::ExperimentalUnicodeIdentifiers. With the flag
off, the only change in behaviour is that a non-ASCII identifier now says
Non-ASCII identifiers are experimental, enable them with --enable=unicode-identifiers instead of failing without a message. Dropping the
gate later is a one-line change.

Tests

src/core/UnicodeIdentifier_test.cc — 33 Catch2 assertions covering the
accepted set, NFC folding in both directions, the canonical singletons, the
rejected classes, and malformed UTF-8.
tests/data/scad/misc/unicode-identifiers.scad — echo test with
--enable=unicode-identifiers, including a variable defined precomposed and
read decomposed.
tests/data/scad/misc/unicode-identifiers-fail.scad — the #737 case, as an
expected parse failure.

Built and run against this tree on Linux with GCC 15.2, Qt 6.10.2,
QScintilla 2.14.1, CGAL 6.1.1, Boost 1.90

Identifiers were limited to [A-Za-z_$][A-Za-z0-9_]*, and anything else was rejected by the lexer without a message. This adds an opt-in identifier syntax behind --enable=unicode-identifiers: ID_Start := (XID_Start + { U+0024, U+005F }) & Identifier_Status=Allowed ID_Continue := XID_Continue & Identifier_Status=Allowed XID_Start/XID_Continue is the UAX #31 default that C++23 and Rust use. The intersection with the UTS #39 general security profile removes code points that are invisible, obsolete, or canonical duplicates; plain XID does not cover this, since variation selectors are Mn and therefore XID_Continue. Identifiers are normalised to NFC before they are validated, so that canonically equivalent spellings name the same variable. Validating the normalised form also accepts the canonical singletons U+2126, U+212A and U+212B, which the UTS #39 profile excludes precisely because NFC folds them. Encoding the code point ranges as UTF-8 byte patterns in the flex rules grows the scanner tables by more than an order of magnitude, so the rules match any non-ASCII sequence and the classification happens in the action, as a binary search over a generated table. The tables are generated from the UCD by scripts/generate-unicode-identifier-tables.py and checked in, so there is no build-time dependency on the Unicode data files. The editor's lexertl rules are widened to match, otherwise a non-ASCII byte splits an identifier into differently styled tokens. Fixes #3736.

t-paul

Very nice 🎉. Thanks for the clean solution. Basing this on the official Unicode config should make it future proof.

Identifiers reaching the AST are NFC, because the lexer normalises them. Parameter set files and the -P command line argument do not go through the lexer and carry whatever normalisation their author used, so comparing the two can fail on canonically equivalent spellings. Nothing reports it: ParameterObjects::importValues() falls back to the design default, and openscad.cc applies no set at all, both without a message. An export driven by -p/-P can therefore produce the wrong dimensions and still exit zero. This cannot happen with a file OpenSCAD wrote itself, since those keys come from the AST. It needs a parameter set produced elsewhere - a generator script, a hand edit, or text taken from a macOS source, where the filesystem hands out NFD. Add normalize_utf8_nfc() to core/str_utf8_wrapper.h, next to the other UTF-8 helpers, using the g_utf8_normalize() glib already provides. ASCII returns without conversion or allocation, which covers every existing parameter set. A string containing a NUL byte is passed through untouched, because g_utf8_normalize() returns a NUL terminated buffer and normalising one would truncate it, which could collapse two distinct names into one lookup key. ParameterSet::setName() normalises rather than each of its callers, so a set name is NFC whether it came from a file or from the customizer. Parameter names are map keys with no such choke point, so they are normalised where ParameterSets::readFile() reads them, and openscad.cc normalises the -P argument before comparing. Also add the <ios> include UnicodeIdentifier.cc needs for std::hex and std::uppercase, which include-cleaner flags. Follow-up to #6968.

* Rename CGALWorker to GeometryWorker The class has not been CGAL-specific for some time: it evaluates the tree through whichever backend is selected, and Manifold is the default. The name now points at the wrong one. The member in MainWindow is renamed to match -- leaving it as cgalworker would have kept the stale name in the place it is actually read. Also initialises three members that were left indeterminate: GeometryWorker's tree pointer, and MainWindow's compileErrors and compileWarnings. Each is assigned before use today, so this changes no behaviour; it just stops that being something a reader has to verify. No functional change. No test expectations were touched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Normalise parameter set names to NFC Identifiers reaching the AST are NFC, because the lexer normalises them. Parameter set files and the -P command line argument do not go through the lexer and carry whatever normalisation their author used, so comparing the two can fail on canonically equivalent spellings. Nothing reports it: ParameterObjects::importValues() falls back to the design default, and openscad.cc applies no set at all, both without a message. An export driven by -p/-P can therefore produce the wrong dimensions and still exit zero. This cannot happen with a file OpenSCAD wrote itself, since those keys come from the AST. It needs a parameter set produced elsewhere - a generator script, a hand edit, or text taken from a macOS source, where the filesystem hands out NFD. Add normalize_utf8_nfc() to core/str_utf8_wrapper.h, next to the other UTF-8 helpers, using the g_utf8_normalize() glib already provides. ASCII returns without conversion or allocation, which covers every existing parameter set. A string containing a NUL byte is passed through untouched, because g_utf8_normalize() returns a NUL terminated buffer and normalising one would truncate it, which could collapse two distinct names into one lookup key. ParameterSet::setName() normalises rather than each of its callers, so a set name is NFC whether it came from a file or from the customizer. Parameter names are map keys with no such choke point, so they are normalised where ParameterSets::readFile() reads them, and openscad.cc normalises the -P argument before comparing. Also add the <ios> include UnicodeIdentifier.cc needs for std::hex and std::uppercase, which include-cleaner flags. Follow-up to openscad#6968. * Extract MainWindow::resetCompileMessageCounts() (openscad#6994) Preparation for running geometry evaluation in a separate process, where a second caller resets these counters on the path that receives a worker's result. Extracting it now keeps that later change to the one line that needs it. * Bump github/codeql-action in the github-actions group (openscad#7002) Bumps the github-actions group with 1 update: [github/codeql-action](https://github.com/github/codeql-action). Updates `github/codeql-action` from 4.37.3 to 4.37.9 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.37.3...v4.37.9) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.9 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update German translation (openscad#7006) * Make sure boost headers are correctly installed (openscad#7003) * Add an "experimental, may change or be removed without notice" warning to Preferences/Features. Fixes openscad#7008. * Default to UTF-8. * Quiet unsolicited info level log message from Qt multimedia plugin. * Update translation template (closes openscad#7007). * Stop marking internal values as translatable (openscad#7024) Some strings extracted into locale/openscad.pot are not user interface text. Translating them changes what gets written to the settings or what is compared against them, and nothing warns a translator about it. * Update translation template for openscad#7024 changes. (openscad#7026) * Rename Preferences "Features" tab to "Experimental". (openscad#7020) * Fix use-after-free when a MainWindow is destroyed (openscad#7005) Closing a window crashes in MainWindow::onEditorDockVisibilityChanged(): MainWindow::~MainWindow QWidget::~QWidget -> QObjectPrivate::deleteChildren Dock::~Dock -> QDockWidget::~QDockWidget QWidget::setParent -> QWidgetPrivate::hide_helper visibilityChanged -> MainWindow::onEditorDockVisibilityChanged MainWindow::updateExportActions MainWindow::formatIdentifierToAction exportMap.find() <- already destroyed The docks are children of the MainWindow, so ~QWidget deletes them, and deleting a QDockWidget reparents and hides it, which emits visibilityChanged and runs the window's event filter. Both land back in MainWindow after its own members have been destroyed: member destruction happens right after the ~MainWindow body, while Qt only severs connections in ~QObject, which runs last. The slot then reads a destroyed exportMap and a dangling activeEditor; MainWindow::eventFilter() reads a destroyed rubberBandManager. Disconnect and un-filter the docks in the destructor body, while the object is still whole. Claude-Session: https://claude.ai/code/session_01Wm1jGSWf57JvbSJ6i6gxTQ Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Jordan Brown <31949071+jordanbrown0@users.noreply.github.com> Co-authored-by: Marius Kintel <marius@kintel.net> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: whosawhatsis <whosawhatsis@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Torsten Paul <Torsten.Paul@gmx.de> Co-authored-by: otzie2023 <otzelot2021@outlook.de> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ótzï fón GlüϷgor <141450226+Otzie2023@users.noreply.github.com> Co-authored-by: Marius Kintel <marius@kintel.net> Co-authored-by: Jordan Brown <31949071+jordanbrown0@users.noreply.github.com> Co-authored-by: Daniel Jalkut <jalkut@red-sweater.com> Co-authored-by: Guenther Sohler <guenther.sohler@photeon.com>

Restore the Unicode lexer validation and NFC normalization from openscad#6968 and normalize -P parameter-set names as in openscad#7000. Keep unicode-identifiers available in non-experimental builds, enabled by default, and allow an explicit --disable=unicode-identifiers or Features preference opt-out. Preserve existing saved preferences and accept the existing --enable flag. Disable requests take precedence over enable requests. Restore integration coverage for Unicode identifiers, security-profile rejection, NFC/NFD parameter names and set selection, and opt-out behavior.

Otzie2023 and others added 2 commits August 22, 2026 15:42

Merge branch 'openscad:master' into feat/unicode-identifiers

9770347

Otzie2023 mentioned this pull request Aug 24, 2026

International native characters in the names of variables, functions, and modules. #3736

Closed

Otzie2023 added 3 commits August 27, 2026 21:23

Merge branch 'openscad:master' into feat/unicode-identifiers

732ba3d

Merge branch 'openscad:master' into feat/unicode-identifiers

e3ed03e

Merge branch 'openscad:master' into feat/unicode-identifiers

557ad86

t-paul approved these changes Aug 30, 2026

View reviewed changes

t-paul merged commit 6e12e27 into openscad:master Aug 30, 2026
9 checks passed

Otzie2023 mentioned this pull request Aug 31, 2026

Normalise parameter set names to NFC #7000

Merged

Otzie2023 mentioned this pull request Oct 8, 2026

Enable non-ASCII identifiers by default. #7101

Open

	NFA states	DFA states	Table entries
current lexer	437	198	1,755
this PR	479	212	2,073
XID encoded in the rules	12,502	1,510	62,843

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

Support non-ASCII identifiers following UAX #31 - #6968

Support non-ASCII identifiers following UAX #31#6968
t-paul merged 5 commits into
openscad:masterfrom
Otzie2023:feat/unicode-identifiers

Otzie2023 commented Aug 22, 2026

Uh oh!

t-paul left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

Otzie2023 commented Aug 22, 2026

Support non-ASCII identifiers following UAX #31

The character set

Normalisation

The lexer tables

Tables

Editor highlighting

Other code paths

Issue #737

Not included

Feature flag

Tests

Uh oh!

t-paul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants