feat: a leaner `@shikijs/primitive` package by antfu · Pull Request #1251 · shikijs/shiki · GitHub
Skip to content

feat: a leaner @shikijs/primitive package#1251

Merged
antfu merged 4 commits intomainfrom
feat/tokenizer
Feb 26, 2026
Merged

feat: a leaner @shikijs/primitive package#1251
antfu merged 4 commits intomainfrom
feat/tokenizer

Conversation

@antfu
Copy link
Copy Markdown
Member

@antfu antfu commented Feb 25, 2026 •
edited
Loading

This PR introduces a learner @shikijs/primitive package peeled from @shikijs/core, which contains the core primitives without advanced features like html-serilizer/transformer/ansi, etc.

The @shikijs/core remanins unchanged

/cc @farnabaz

@netlify
Copy link
Copy Markdown

netlify Bot commented Feb 25, 2026 •
edited
Loading

@netlify
Copy link
Copy Markdown

netlify Bot commented Feb 25, 2026 •
edited
Loading

❌ Deploy Preview for shiki-next failed.

Name Link
🔨 Latest commit c0ea816
🔍 Latest deploy log https://app.netlify.com/projects/shiki-next/deploys/69a029d1d6065c0008808e8a

@antfu antfu changed the title feat: a leaner @shikijs/tokenizer package feat: a leaner @shikijs/primitive package Feb 26, 2026
@antfu antfu requested a review from Copilot February 26, 2026 11:11
@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 26, 2026 •
edited
Loading

Codecov Report

❌ Patch coverage is 96.15385% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.41%. Comparing base (cdca994) to head (c0ea816).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...ges/primitive/src/highlight/code-to-tokens-base.ts 97.74% 3 Missing ⚠️
packages/core/src/constructors/highlighter.ts 88.88% 1 Missing ⚠️
packages/core/src/highlight/code-to-hast.ts 66.66% 1 Missing ⚠️
packages/core/src/highlight/code-to-html.ts 66.66% 1 Missing ⚠️
packages/primitive/src/constructors/async.ts 87.50% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1251      +/-   ##
==========================================
+ Coverage   89.03%   89.41%   +0.38%     
==========================================
  Files          76       77       +1     
  Lines        3439     3431       -8     
  Branches      994      990       -4     
==========================================
+ Hits         3062     3068       +6     
+ Misses        338      327      -11     
+ Partials       39       36       -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@antfu antfu merged commit 05a77f1 into main Feb 26, 2026
13 of 18 checks passed
Copilot AI reviewed Feb 26, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new @shikijs/primitive package that extracts core tokenization primitives from @shikijs/core, creating a leaner package focused on TextMate grammar-based tokenization without advanced features like HTML serialization, transformers, and ANSI support.

Changes:

  • Created new @shikijs/primitive package with core tokenization functionality extracted from @shikijs/core
  • Renamed ShikiInternal interface/type to ShikiPrimitive throughout the codebase
  • Renamed syncThemesTokenization to alignThemesTokenization for better clarity
  • Removed deprecation warning system (enableDeprecationWarnings, warnDeprecated) from the codebase
  • Maintained backward compatibility by exporting deprecated function name aliases (createShikiInternal, createShikiInternalSync)

Reviewed changes

Copilot reviewed 38 out of 48 changed files in this pull request and generated no comments.

Show a summary per file
File Description
packages/primitive/package.json New package configuration for the primitive tokenizer package
packages/primitive/src/**/*.ts Core tokenization implementation moved from @shikijs/core, including tokenizers, grammar state, TextMate registry, and utility functions
packages/types/src/highlighter.ts Interface renamed from ShikiInternal to ShikiPrimitive
packages/core/package.json Added dependency on @shikijs/primitive
packages/core/src/index.ts Re-exports primitive package functions including deprecated aliases for backward compatibility
packages/core/src/highlight/code-to-tokens-base.ts Simplified to wrap primitive's implementation and add ANSI language support
packages/core/src/constructors/highlighter.ts Updated to use primitive package's functions
packages/core/src/warn.ts Removed - deprecation warning system deleted
test/exports/**/*.yaml Updated export snapshots reflecting new API surface
packages/shiki/test/themes.test.ts Updated to import and use renamed alignThemesTokenization function
packages/monaco/src/index.ts Updated type references from ShikiInternal to ShikiPrimitive
Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

This was referenced Feb 27, 2026
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
This was referenced Feb 28, 2026
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@antfu