Fix SQLAlchemy Performance Issues by RustyBower · Pull Request #1051 · scoringengine/scoringengine · GitHub
Skip to content

Fix SQLAlchemy Performance Issues#1051

Merged
RustyBower merged 13 commits into
masterfrom
claude/fix-performance-issues-J4FHc
Jan 20, 2026
Merged

Fix SQLAlchemy Performance Issues#1051
RustyBower merged 13 commits into
masterfrom
claude/fix-performance-issues-J4FHc

Conversation

@RustyBower

Copy link
Copy Markdown
Collaborator

This commit addresses the most critical performance issues found in the codebase:

  1. Remove NullPool - Switched from NullPool to proper connection pooling

    • NullPool was creating a new database connection for every operation
    • Flask-SQLAlchemy now uses proper connection pooling with pool_pre_ping
  2. Remove commit-before-query monkeypatch - Eliminated the performance-killing workaround

    • The monkeypatch was committing before EVERY query, preventing batch operations
    • This was added to work around stale data issues with improper session management
  3. Implement proper session lifecycle management via Flask-SQLAlchemy

    • Sessions are now automatically managed per-request in web views
    • Fresh session created at request start, cleaned up at request end
    • Solves the stale data issue properly without performance penalty
  4. Add application context support for engine and worker processes

    • bin/engine, bin/setup, and bin/web now create app contexts
    • Allows Flask-SQLAlchemy to work properly outside of request context

Changes:

  • scoring_engine/db.py: Replaced raw SQLAlchemy with Flask-SQLAlchemy
  • scoring_engine/models/base.py: Updated to use db.Model
  • scoring_engine/web/init.py: Initialize Flask-SQLAlchemy with proper config
  • All views, models, engine files: Changed from 'session' to 'db.session'
  • bin/* scripts: Added app context support
  • tests/*: Updated to use new Flask-SQLAlchemy API

Performance improvements:

  • Connection pooling now enabled (was disabled with NullPool)
  • No more commit before every query (massive improvement)
  • Proper transaction batching now possible
  • Session lifecycle properly managed per-request

This migration maintains backward compatibility while fixing the root cause of the caching/stale data issues that led to the monkeypatch workaround.

@RustyBower RustyBower changed the title Find and fix performance issues Fix SQLAlchemy Performance Issues Dec 30, 2025
@RustyBower RustyBower force-pushed the claude/fix-performance-issues-J4FHc branch from b2f9fb2 to cf49032 Compare December 30, 2025 03:18
This commit addresses the most critical performance issues found in the codebase:

1. **Remove NullPool** - Switched from NullPool to proper connection pooling
   - NullPool was creating a new database connection for every operation
   - Flask-SQLAlchemy now uses proper connection pooling with pool_pre_ping

2. **Remove commit-before-query monkeypatch** - Eliminated the performance-killing workaround
   - The monkeypatch was committing before EVERY query, preventing batch operations
   - This was added to work around stale data issues with improper session management

3. **Implement proper session lifecycle management** via Flask-SQLAlchemy
   - Sessions are now automatically managed per-request in web views
   - Fresh session created at request start, cleaned up at request end
   - Solves the stale data issue properly without performance penalty

4. **Add application context support** for engine and worker processes
   - bin/engine, bin/setup, and bin/web now create app contexts
   - Allows Flask-SQLAlchemy to work properly outside of request context

Changes:
- scoring_engine/db.py: Replaced raw SQLAlchemy with Flask-SQLAlchemy
- scoring_engine/models/base.py: Updated to use db.Model
- scoring_engine/web/__init__.py: Initialize Flask-SQLAlchemy with proper config
- All views, models, engine files: Changed from 'session' to 'db.session'
- bin/* scripts: Added app context support
- tests/*: Updated to use new Flask-SQLAlchemy API

Performance improvements:
- Connection pooling now enabled (was disabled with NullPool)
- No more commit before every query (massive improvement)
- Proper transaction batching now possible
- Session lifecycle properly managed per-request

This migration maintains backward compatibility while fixing the root cause
of the caching/stale data issues that led to the monkeypatch workaround.
This commit fixes "Working outside of application context" errors that
occur when using Flask-SQLAlchemy with pytest.

Problem:
- Flask-SQLAlchemy requires an active application context for all database operations
- Models with class methods (@staticmethod) that query the database need app context
- Tests were failing with "RuntimeError: Working outside of application context"

Solution:
1. Added session-scoped app context fixture in tests/conftest.py
   - Provides app context for test collection (parametrize decorators)
   - Ensures db operations during test collection work properly

2. Updated tests/scoring_engine/unit_test.py
   - UnitTest now creates its own Flask app and context in setup_method
   - Sets self.session = db.session for backward compatibility
   - Properly cleans up context in teardown_method

3. Fixed tests/scoring_engine/web/web_test.py
   - WebTest now reuses the app from UnitTest instead of creating a duplicate
   - Avoids nested app contexts that cause issues
   - Still creates test_client for HTTP testing

Files changed:
- tests/conftest.py: Added session-scoped app_context fixture
- tests/scoring_engine/unit_test.py: Create app/context for each test
- tests/scoring_engine/web/web_test.py: Reuse parent app instead of creating new one

This ensures all database operations in tests (including model class methods
like Team.get_all_blue_teams(), Round.get_last_round_num(), etc.) have the
required Flask application context.
This commit fixes three categories of test failures:

1. **Engine missing db attribute**
   - Added `self.db = db` to Engine.__init__ for backward compatibility
   - Engine code uses `self.db.session` throughout, so this reference is needed
   - File: scoring_engine/engine/engine.py

2. **Duplicate app context in TestEngine**
   - Removed duplicate app context creation in TestEngine.setup_method()
   - UnitTest parent class already creates app context
   - Creating a second context caused "popped wrong app context" errors
   - File: tests/scoring_engine/engine/test_engine.py

3. **Competition.save() signature mismatch**
   - Updated test to call `competition.save()` instead of `competition.save(self.session)`
   - Competition.save() no longer takes a session parameter (uses db.session internally)
   - File: tests/scoring_engine/test_competition.py

All three issues stemmed from the Flask-SQLAlchemy migration where:
- db.session is now managed by Flask-SQLAlchemy
- Application contexts are required for all database operations
- Session is no longer passed around as a parameter

These fixes maintain backward compatibility while completing the migration.
The integration-get-round target was calling Round.get_last_round_num()
without an application context, causing a RuntimeError.

With Flask-SQLAlchemy, all database operations require an active app
context. Updated the command to create an app and push context before
calling the model method.

This fixes the error:
RuntimeError: Working outside of application context.
The integration tests were calling model methods and database queries at
module import time (before app context was available):

1. ensure_integration_data() was called at module level (line 16)
2. @pytest.mark.parametrize decorators called db queries directly (lines 29, 36, 42)

This caused 'RuntimeError: Working outside of application context' during
test collection.

Solution:
1. Created tests/integration/conftest.py with setup_integration_db fixture
   - Calls ensure_integration_data() after app context is created
   - Uses autouse=True and depends on app_context fixture

2. Replaced @pytest.mark.parametrize with pytest_generate_tests hook
   - pytest_generate_tests runs after fixtures are set up
   - App context is available when generating test parameters
   - Dynamically creates parametrized tests for blue_team, service, check

This ensures all database operations during test collection happen
inside an application context.
@RustyBower RustyBower force-pushed the claude/fix-performance-issues-J4FHc branch from 2b69709 to 531367e Compare December 30, 2025 20:35
- Changed session.query() to db.session.query() in check_result_for_round()
- Changed session.query() to db.session.query() in last_ten_checks property
- These were missed during the Flask-SQLAlchemy migration
The pytest_generate_tests hook runs during collection phase, before
fixtures are executed. Need to create app context explicitly for
database queries during test parametrization.
The pytest_generate_tests hook runs during collection and needs to query
the database for test parametrization. Call ensure_integration_data() to
initialize the database before running queries.
The last_check_result() method was accessing self.checks[-1] which
relies on lazy loading. In integration tests, service objects created
during collection are in a different session than during execution,
causing lazy loading to fail.

Changed to use an explicit query to avoid session issues.
Some test classes pop the context in their teardown before calling
super(), which causes IndexError when the parent tries to pop again.
Make teardown_method defensive by checking if context has tokens
before popping.
Removed autouse=True from session-scoped app_context fixture.
Tests like test_config_loader_env.py don't use Flask/SQLAlchemy and
don't need an app context. Integration tests still get the context
via their setup_integration_db fixture dependency. UnitTest-based
tests create their own context in setup_method.
- test_concurrent_users.py: Python script to test concurrent user sessions
- test_simple_concurrent.sh: Simple bash script for quick testing
- TESTING_SESSION_ISOLATION.md: Comprehensive testing guide

These tools help verify the Flask-SQLAlchemy migration properly
isolates sessions between users and prevents data contamination.
This test verifies that Flask-SQLAlchemy properly isolates sessions
between requests and prevents stale data issues.

Test proves:
- Each request gets a fresh session with empty identity map
- External updates are visible in subsequent requests
- No stale cached objects persist between requests

This demonstrates the root cause fix: Flask-SQLAlchemy scopes sessions
to app context (per-request) instead of using a global session,
eliminating the need for the commit-before-query monkeypatch.
@RustyBower RustyBower merged commit fb1d118 into master Jan 20, 2026
6 checks passed
RustyBower pushed a commit that referenced this pull request Jan 22, 2026
This commit addresses two critical performance issues:

1. Remove lazy="joined" from Team model relationships (services, users, inject)
   - Changed to lazy="select" to prevent over-fetching data
   - Previously, every Team query loaded ALL related services, users, and injects
   - Now relationships are only loaded when explicitly accessed
   - Significantly reduces database load and memory usage

2. Fix redundant database query in /api/team/<team_id>/services endpoint
   - Use pre-loaded checks instead of calling service.last_ten_checks property
   - The checks were already loaded via subqueryload(Service.checks)
   - Eliminates N+1 query problem (was making extra DB query per service)
   - For 50 services, this saves 50 database queries per API call

These changes build on top of PR #1051 which fixed the underlying session
management and connection pooling issues.

Performance Impact:
- Reduces database queries by 50+ per API request
- Eliminates over-fetching of unused relationship data
- Lower memory usage when querying Teams
- Better scalability under load
@RustyBower RustyBower deleted the claude/fix-performance-issues-J4FHc branch January 23, 2026 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants