feat(postgres): replace pg-hstore with inline hstore parser by WikiRik · Pull Request #18151 · sequelize/sequelize · GitHub
Skip to content

feat(postgres): replace pg-hstore with inline hstore parser - #18151

Merged
WikiRik merged 9 commits into
mainfrom
WikiRik/pg-hstore
Apr 7, 2026
Merged

feat(postgres): replace pg-hstore with inline hstore parser#18151
WikiRik merged 9 commits into
mainfrom
WikiRik/pg-hstore

Conversation

@WikiRik

@WikiRik WikiRik commented Mar 11, 2026

Copy link
Copy Markdown
Member

Pull Request Checklist

  • Have you added new tests to prevent regressions?
  • If a documentation update is necessary, have you opened a PR to the documentation repository?
  • Did you update the typescript typings accordingly (if applicable)?
  • Does the description below contain a link to an existing issue (Closes #[issue]) or a description of the issue you are solving?
  • Does the name of your PR follow our conventions?

Description of Changes

pg-hstore has been unmaintained since 2021 and lacked TypeScript types (requiring @ts-expect-error). The implementation was ~70 lines with a single underscore dependency used only for _.defaults.

Replace with a purpose-built TypeScript implementation that:

  • Drops the pg-hstore and underscore dependencies
  • Adds full type safety (no @ts-expect-error)
  • Uses Object.create(null) to prevent prototype pollution (fixes skipped test)
  • Removes unused callback API and hardcodes sanitize: true

Expand hstore unit tests with cases from the original pg-hstore test suite: multi-value objects, embedded JSON strings, URLs ending with backslash, newlines/carriage returns, prototype injection protection, and idempotency.

Summary by CodeRabbit

  • Tests

    • Added integration tests to guard against prototype-pollution in JSON/HSTORE round-trips and expanded HSTORE coverage (empty/null values, escaping, special characters, idempotent roundtrips).
  • Chores

    • Removed the external HSTORE dependency and replaced it with an internal HSTORE implementation.
  • Refactor

    • Standardized plain-object creation across packages for more consistent internal behavior.

@coderabbitai

coderabbitai Bot commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

Comment thread packages/postgres/src/_internal/hstore.ts Fixed
@WikiRik
WikiRik marked this pull request as ready for review March 11, 2026 19:57
@WikiRik
WikiRik requested a review from a team as a code owner March 11, 2026 19:57
@WikiRik
WikiRik requested review from ephys and sdepold March 11, 2026 19:57

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
packages/postgres/src/_internal/hstore.ts (1)

34-38: Consider renaming to avoid shadowing the global unescape.

While the global unescape is deprecated, shadowing it can cause confusion and is flagged by Biome's noShadowRestrictedNames rule. A more specific name like unescapeHstoreValue would be clearer.

Suggested rename
-function unescape(value: string): string {
+function unescapeHstoreValue(value: string): string {
   return value
     .slice(1, -1) // strip surrounding quotes
     .replaceAll(/\\(["\\])/g, '$1'); // unescape \" and \\ in one pass
 }

Then update the usage on line 52:

-    result[unescape(rawKey)] = rawValue === 'NULL' ? null : unescape(rawValue);
+    result[unescapeHstoreValue(rawKey)] = rawValue === 'NULL' ? null : unescapeHstoreValue(rawValue);
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/postgres/src/_internal/hstore.ts` around lines 34 - 38, Rename the
local function unescape to a clearer non-shadowing name (e.g.,
unescapeHstoreValue) and update every local call site in this module that
currently invokes unescape (for example the usage inside the hstore parsing
logic) to call the new name; keep the implementation identical but change the
function identifier and any references so you no longer shadow the global
deprecated unescape.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/core/test/integration/data-types/data-types.test.ts`:
- Around line 1916-1926: The test is not creating an own "__proto__" property
because object literal __proto__ is treated specially; update the payloads
passed to testSimpleInOut (the input and the expected object) to use computed
property syntax so the "__proto__" key is an actual own property (e.g., use
['__proto__'] as the property name) when constructing the objects passed to
testSimpleInOut for vars.User and 'attr', and leave the subsequent expect(({} as
any).__proto__).to.equal(Object.prototype) check unchanged.
- Around line 1857-1868: The test is not actually creating an own "__proto__"
property because `{ __proto__: 'polluted' }` sets the prototype; update the test
that calls testSimpleInOut(vars.User, 'jsonObject', ...) to pass an object with
an actual own property by using computed property syntax (e.g., create the input
and expected objects as something equivalent to const polluted = {
['__proto__']: 'polluted' } and pass polluted for both input and expected), so
the round-trip truly tests prototype pollution protection while keeping the
final assertion that ({} as any).__proto__ equals Object.prototype; modify the
call sites referring to that literal in this test accordingly (testSimpleInOut
and its input/expected values).

---

Nitpick comments:
In `@packages/postgres/src/_internal/hstore.ts`:
- Around line 34-38: Rename the local function unescape to a clearer
non-shadowing name (e.g., unescapeHstoreValue) and update every local call site
in this module that currently invokes unescape (for example the usage inside the
hstore parsing logic) to call the new name; keep the implementation identical
but change the function identifier and any references so you no longer shadow
the global deprecated unescape.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e9769d0a-13b5-4e12-9342-c017e8c89030

📥 Commits

Reviewing files that changed from the base of the PR and between 81a2032 and 1f289a5.

⛔ Files ignored due to path filters (1)
  • yarn.lock is excluded by !**/yarn.lock, !**/*.lock
📒 Files selected for processing (4)
  • packages/core/test/integration/data-types/data-types.test.ts
  • packages/postgres/package.json
  • packages/postgres/src/_internal/hstore.test.ts
  • packages/postgres/src/_internal/hstore.ts
💤 Files with no reviewable changes (1)
  • packages/postgres/package.json

Comment thread packages/core/test/integration/data-types/data-types.test.ts
Comment thread packages/core/test/integration/data-types/data-types.test.ts Outdated
@WikiRik
WikiRik requested a review from SippieCup March 18, 2026 07:27
Rik Smale and others added 6 commits April 6, 2026 19:56
pg-hstore has been unmaintained since 2021 and lacked TypeScript types
(requiring @ts-expect-error). The implementation was ~70 lines with a
single `underscore` dependency used only for _.defaults.

Replace with a purpose-built TypeScript implementation that:
- Drops the pg-hstore and underscore dependencies
- Adds full type safety (no @ts-expect-error)
- Uses Object.create(null) to prevent prototype pollution (fixes skipped test)
- Removes unused callback API and hardcodes sanitize: true

Expand hstore unit tests with cases from the original pg-hstore test suite:
multi-value objects, embedded JSON strings, URLs ending with backslash,
newlines/carriage returns, prototype injection protection, and idempotency.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The hstore format only escapes backslashes and double quotes.
Single quotes have no special meaning in hstore — they are literal
characters that pass through unchanged.

The previous implementation inherited the `''`-escaping from pg-hstore,
which was designed for SQL string literal interpolation (old style).
In Sequelize, the dialect's escapeString/bind parameters handle SQL-level
quoting separately, so pre-escaping single quotes in the hstore layer
caused data corruption: a stored value of `it's` would round-trip as
`it''s`.

Also add:
- Empty-string key tests for both stringify and parse
- Empty-string key in the roundtrip fixture

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- sanitize(): two chained replaces → single pass with character class /[\\"]/g
- unescape(): three chained replaces → slice(1,-1) + single pass /\\(["\\])/g
- HSTORE_PAIR_REGEX: remove unnecessary capture group and backreference,
  use standard quoted-string-with-escapes pattern NULL|"(?:[^"\\]|\\.)*"
  (non-backtracking, no backreference overhead)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…otype pollution integration tests

- sanitize(): two chained replaces → single-pass /[\\"]/g with replaceAll
- unescape(): three chained replaces → slice(1,-1) + single-pass /\\(["\\])/g
- HSTORE_PAIR_REGEX: remove capture group/backreference, use standard
  quoted-string-with-escapes pattern NULL|"(?:[^"\\]|\\.)*"
- Add integration test for prototype pollution via hstore round-trip
- Add integration test for prototype pollution via JSON/JSONB round-trip

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@WikiRik
WikiRik force-pushed the WikiRik/pg-hstore branch from 1f289a5 to c542318 Compare April 6, 2026 18:04
@WikiRik
WikiRik requested review from ephys and removed request for ephys April 6, 2026 18:06

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/postgres/src/_internal/hstore.ts`:
- Around line 12-18: The valueToString function currently only sanitizes when
typeof value === 'string', so non-string inputs (number/boolean/object) are
stringified with String(value) and not escaped; change valueToString to handle
null explicitly and to run sanitize on the stringified result for all non-null
inputs (i.e., call sanitize(String(value)) for numbers/booleans/objects) so
objects with custom toString() are properly escaped; keep the sanitize call for
string inputs but unify logic so sanitize is always applied before returning a
non-null string.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 37a2c432-baf6-4001-8464-f73816f89109

📥 Commits

Reviewing files that changed from the base of the PR and between 1f289a5 and c542318.

⛔ Files ignored due to path filters (1)
  • yarn.lock is excluded by !**/yarn.lock, !**/*.lock
📒 Files selected for processing (4)
  • packages/core/test/integration/data-types/data-types.test.ts
  • packages/postgres/package.json
  • packages/postgres/src/_internal/hstore.test.ts
  • packages/postgres/src/_internal/hstore.ts
💤 Files with no reviewable changes (1)
  • packages/postgres/package.json
✅ Files skipped from review due to trivial changes (1)
  • packages/core/test/integration/data-types/data-types.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/postgres/src/_internal/hstore.test.ts

Comment thread packages/postgres/src/_internal/hstore.ts
Comment thread packages/core/test/integration/data-types/data-types.test.ts Outdated
Comment thread packages/postgres/src/_internal/hstore.ts Outdated
Rik Smale added 2 commits April 6, 2026 21:19
Some have additional type issues so I'm not converting those yet
Inspect the round-tripped JSON value directly so the test verifies __proto__ remains an own property instead of becoming the returned object's prototype.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/postgres/src/_internal/hstore.ts (1)

49-55: Consider adding a bounds check for robustness.

The loop assumes matches come in pairs. While PostgreSQL output should always be well-formed, a defensive check could prevent silent data loss if the regex ever matches an odd number of tokens due to malformed input.

💡 Optional defensive check
   for (let i = 0; i < matches.length; i += 2) {
     const rawKey = matches[i];
     const rawValue = matches[i + 1];
-    if (rawKey && rawValue) {
+    if (rawKey && rawValue !== undefined) {
       result[unescapeHstoreValue(rawKey)] =
         rawValue === 'NULL' ? null : unescapeHstoreValue(rawValue);
+    } else if (rawKey && rawValue === undefined) {
+      // Malformed hstore: odd number of tokens
+      throw new Error(`Malformed hstore value: unpaired key "${rawKey}"`);
     }
   }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/postgres/src/_internal/hstore.ts` around lines 49 - 55, The loop
iterating over matches (for (let i = 0; i < matches.length; i += 2)) assumes
pairs and can drop a trailing unmatched token; add a defensive bounds check
inside the loop (or adjust the loop condition) to ensure matches[i + 1] exists
before using rawValue, and skip or log the incomplete pair so
unescapeHstoreValue and assignment to result only run on complete key/value
pairs.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@packages/postgres/src/_internal/hstore.ts`:
- Around line 49-55: The loop iterating over matches (for (let i = 0; i <
matches.length; i += 2)) assumes pairs and can drop a trailing unmatched token;
add a defensive bounds check inside the loop (or adjust the loop condition) to
ensure matches[i + 1] exists before using rawValue, and skip or log the
incomplete pair so unescapeHstoreValue and assignment to result only run on
complete key/value pairs.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 619b480e-6fce-43e1-9867-e121ec6d5599

📥 Commits

Reviewing files that changed from the base of the PR and between c542318 and ab4fe42.

📒 Files selected for processing (19)
  • packages/core/src/abstract-dialect/query-generator.js
  • packages/core/src/model-definition.ts
  • packages/core/src/model.js
  • packages/core/src/utils/format.ts
  • packages/core/src/utils/object.ts
  • packages/core/src/utils/sql.ts
  • packages/core/test/integration/data-types/data-types.test.ts
  • packages/core/test/support.ts
  • packages/db2/src/query-generator.js
  • packages/ibmi/src/query-generator.js
  • packages/ibmi/src/query.js
  • packages/mssql/src/query-generator.js
  • packages/oracle/src/query-generator.js
  • packages/oracle/src/query.js
  • packages/postgres/src/_internal/hstore.ts
  • packages/postgres/src/query.js
  • packages/snowflake/src/query.js
  • packages/sqlite3/src/query-generator.js
  • packages/sqlite3/src/query.js
✅ Files skipped from review due to trivial changes (1)
  • packages/core/src/model-definition.ts

@ephys ephys left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming tests pass

@WikiRik
WikiRik enabled auto-merge (squash) April 6, 2026 19:55
@WikiRik
WikiRik merged commit 639c646 into main Apr 7, 2026
147 of 148 checks passed
@WikiRik
WikiRik deleted the WikiRik/pg-hstore branch April 7, 2026 11:00
@coderabbitai coderabbitai Bot mentioned this pull request Apr 7, 2026
5 tasks
papandreou added a commit to papandreou/sequelize that referenced this pull request Apr 15, 2026
* main: (200 commits)
  meta: update actions/upload-artifact action to v7.0.1 (sequelize#18212)
  meta: update sequelize AUTHORS (sequelize#18208)
  feat(cli): add migration generate, run, status & undo (sequelize#18193)
  feat(core): Drop the ability to specify the dialect as a string (sequelize#18204)
  feat: update some dialect adapters (sequelize#18189)
  feat(postgres): replace pg-hstore with inline hstore parser (sequelize#18151)
  feat(core): add `sql.random`, improve `Order` typing (sequelize#18203)
  meta: use Sequelize Bot for drafting PRs (sequelize#18201)
  meta: automatically mark unfinished PRs as draft (sequelize#18197)
  meta: update dependency @oclif/test to v4.1.18 (sequelize#18198)
  fix(core): replace @typeparam with @template in JSDoc (sequelize#18094)
  feat(core): Add support for UUID v7 (sequelize#17832)
  meta: update sequelize AUTHORS (sequelize#18190)
  meta: update dependency @oclif/plugin-help to ^6.2.43 (sequelize#18195)
  meta: update dependency esbuild to v0.28.0 (sequelize#18192)
  meta: update dependency @oclif/plugin-help to ^6.2.42 (sequelize#18191)
  meta: update dependency @oclif/core to ^4.10.5 (sequelize#18187)
  meta: update sequelize AUTHORS (sequelize#18158)
  feat: add Node 24 support, drop Node 18 (sequelize#18185)
  feat(core): add support for readonly attribute arrays (sequelize#18186)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants