fix: Handle YAML parsing errors and missing entity_type in markdown files by phernandez · Pull Request #368 · basicmachines-co/basic-memory · GitHub
Skip to content

fix: Handle YAML parsing errors and missing entity_type in markdown files - #368

Merged
phernandez merged 1 commit into
mainfrom
fix/issue-184-185-markdown-parsing-errors
Oct 16, 2025
Merged

fix: Handle YAML parsing errors and missing entity_type in markdown files#368
phernandez merged 1 commit into
mainfrom
fix/issue-184-185-markdown-parsing-errors

Conversation

@phernandez

Copy link
Copy Markdown
Member

Issues Fixed

Fixes basicmachines-co/basic-memory-cloud#184
Fixes basicmachines-co/basic-memory-cloud#185

Issue #184: NOT NULL constraint on entity_type

Problem: Sync was failing with database constraint violations when markdown files were missing the type field in frontmatter or had it explicitly set to null.

Error from production (41 occurrences in 24 hours):

NOT NULL constraint failed: entity.entity_type
Failed to upsert entity for Archive/articles published/The Invisible Weight of Mental Habits.md

Root cause: Files without type field or with type: null resulted in None being passed to the database, violating the NOT NULL constraint.

Issue #185: YAML parsing errors causing sync failures

Problem: Files with malformed YAML frontmatter caused sync to fail completely, leading to:

  • 12+ minute sync operations (for just 2 files - normally processes thousands in seconds)
  • Infinite retry loops with circuit breaker
  • Keepalive ping failures during extended operations
  • Complete blocking of sync for entire knowledge base

Error from production:

Failed to sync file: path=family/mom/_archive/group-chat-texts-00601-01200-paste.md
error=block sequence entries are not allowed in this context 
in "<unicode string>", line 5, column 7

Root cause: Malformed YAML syntax (e.g., unexpected list syntax) caused python-frontmatter to raise YAMLError, which wasn't being caught, causing the entire sync to fail.

Solution

Changes to src/basic_memory/markdown/entity_parser.py

  1. Added YAML error handling (Issue fix: enhance character conflict detection and error handling for sync operations #185):

    • Wrapped frontmatter.loads() in try/catch for yaml.YAMLError
    • On parsing failure: log warning with file context, treat as plain markdown
    • Allows sync to continue processing other files
  2. Enhanced entity_type defaults (Issue fix: preserve permalink when editing notes without frontmatter permalink #184):

    • Explicitly check for None values (not just missing keys)
    • Always ensure entity_type is set to "note" when missing or null
    • Handles both type: (missing) and type: null (explicit null) cases

New comprehensive tests

Created tests/markdown/test_entity_parser_error_handling.py with 7 tests:

Impact

Robustness

  • Files with YAML errors no longer block entire sync operation
  • Graceful degradation: treat malformed files as plain markdown
  • Better error messages with file context for debugging

Performance

Data Integrity

  • All files get synced with appropriate defaults
  • No more silent failures or missing content in database
  • Users can find and fix problematic files from logs

Code Quality

  • Markdown parsing module now has 100% test coverage
  • All 43 markdown tests pass (7 new, 36 existing)
  • No regressions in existing functionality

Testing

$ uv run pytest tests/markdown/ -v
================================ 43 passed in 8.49s ================================

# Coverage for markdown module
src/basic_memory/markdown/entity_parser.py         68      0   100%
src/basic_memory/markdown/markdown_processor.py    49      0   100%
src/basic_memory/markdown/plugins.py              119      0   100%
src/basic_memory/markdown/schemas.py               43      0   100%

Related

🤖 Generated with Claude Code

…iles (#184, #185)

## Issues Fixed

### Issue #184: NOT NULL constraint on entity_type
**Problem**: Sync was failing with database constraint violations when
markdown files were missing the `type` field in frontmatter or had it
explicitly set to `null`.

**Error**:
```
NOT NULL constraint failed: entity.entity_type
```

**Solution**: Added explicit check for None/null entity_type values and
always provide "note" as the default type.

### Issue #185: YAML parsing errors causing sync failures
**Problem**: Files with malformed YAML frontmatter caused sync to fail
completely, leading to:
- 12+ minute sync operations (for just 2 files)
- Infinite retry loops
- Keepalive ping failures
- Complete blocking of sync for entire knowledge base

**Error**:
```
block sequence entries are not allowed in this context
in "<unicode string>", line 5, column 7
```

**Solution**: Wrapped YAML parsing in try/catch block to gracefully
handle malformed frontmatter. When YAML parsing fails:
- Log warning with file path and error details
- Treat file as plain markdown without frontmatter
- Continue sync with remaining files

## Changes

**src/basic_memory/markdown/entity_parser.py**:
- Added `yaml` and `logger` imports
- Wrapped `frontmatter.loads()` in try/catch for YAMLError
- Enhanced entity_type handling to catch None values explicitly
- Added warning logs for failed YAML parsing

**tests/markdown/test_entity_parser_error_handling.py** (new):
- 7 comprehensive tests covering both issues
- Tests for malformed YAML, invalid YAML, missing type, null type
- Tests ensure valid files still parse correctly

## Impact

- **Robustness**: Files with YAML errors no longer block entire sync
- **Performance**: Eliminates 12+ minute sync hangs from single bad files
- **Data integrity**: All files get synced with appropriate defaults
- **User experience**: Better error messages, no more silent failures
- **Coverage**: Markdown parsing now at 100% test coverage

## Testing

- All 43 markdown tests pass (7 new tests added)
- All existing tests pass with no regressions
- Entity parser now has 100% code coverage

Signed-off-by: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@github-actions

github-actions Bot commented Oct 16, 2025

Copy link
Copy Markdown
Contributor

@phernandez
phernandez merged commit d1431bd into main Oct 16, 2025
16 checks passed
@phernandez
phernandez deleted the fix/issue-184-185-markdown-parsing-errors branch October 16, 2025 18:19
claude Bot pushed a commit that referenced this pull request Oct 17, 2025
Fixes #378

- Changed error handling in update_frontmatter() to match PR #368 pattern
- YAML parsing errors now logged as WARNING instead of ERROR
- Files with malformed frontmatter treated as plain markdown
- Added test for malformed YAML case (KB: title format)

This resolves 1,112 errors/3hrs in production logs from files with
colons in titles that confuse the YAML parser.

Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
phernandez added a commit that referenced this pull request Oct 17, 2025
The update_frontmatter() function was logging ERROR (level 17) for malformed
YAML frontmatter instead of handling it gracefully. This caused 1,112+ errors
per 3 hours in production from files with titles like "KB: Something" where
the colon breaks YAML parsing.

This fix applies the same error handling pattern from PR #368's entity_parser.py:
- Catch ParseError and yaml.YAMLError when parsing frontmatter
- Log as WARNING (level 13) instead of ERROR
- Treat file as having no frontmatter and proceed with update
- Only log ERROR for actual file operation failures

Files with malformed frontmatter now get updated successfully with valid
frontmatter instead of spamming error logs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant