{{ message }}
Fix SQLAlchemy Performance Issues#1051
Merged
Merged
Conversation
b2f9fb2 to
cf49032
Compare
This commit addresses the most critical performance issues found in the codebase: 1. **Remove NullPool** - Switched from NullPool to proper connection pooling - NullPool was creating a new database connection for every operation - Flask-SQLAlchemy now uses proper connection pooling with pool_pre_ping 2. **Remove commit-before-query monkeypatch** - Eliminated the performance-killing workaround - The monkeypatch was committing before EVERY query, preventing batch operations - This was added to work around stale data issues with improper session management 3. **Implement proper session lifecycle management** via Flask-SQLAlchemy - Sessions are now automatically managed per-request in web views - Fresh session created at request start, cleaned up at request end - Solves the stale data issue properly without performance penalty 4. **Add application context support** for engine and worker processes - bin/engine, bin/setup, and bin/web now create app contexts - Allows Flask-SQLAlchemy to work properly outside of request context Changes: - scoring_engine/db.py: Replaced raw SQLAlchemy with Flask-SQLAlchemy - scoring_engine/models/base.py: Updated to use db.Model - scoring_engine/web/__init__.py: Initialize Flask-SQLAlchemy with proper config - All views, models, engine files: Changed from 'session' to 'db.session' - bin/* scripts: Added app context support - tests/*: Updated to use new Flask-SQLAlchemy API Performance improvements: - Connection pooling now enabled (was disabled with NullPool) - No more commit before every query (massive improvement) - Proper transaction batching now possible - Session lifecycle properly managed per-request This migration maintains backward compatibility while fixing the root cause of the caching/stale data issues that led to the monkeypatch workaround.
This commit fixes "Working outside of application context" errors that occur when using Flask-SQLAlchemy with pytest. Problem: - Flask-SQLAlchemy requires an active application context for all database operations - Models with class methods (@staticmethod) that query the database need app context - Tests were failing with "RuntimeError: Working outside of application context" Solution: 1. Added session-scoped app context fixture in tests/conftest.py - Provides app context for test collection (parametrize decorators) - Ensures db operations during test collection work properly 2. Updated tests/scoring_engine/unit_test.py - UnitTest now creates its own Flask app and context in setup_method - Sets self.session = db.session for backward compatibility - Properly cleans up context in teardown_method 3. Fixed tests/scoring_engine/web/web_test.py - WebTest now reuses the app from UnitTest instead of creating a duplicate - Avoids nested app contexts that cause issues - Still creates test_client for HTTP testing Files changed: - tests/conftest.py: Added session-scoped app_context fixture - tests/scoring_engine/unit_test.py: Create app/context for each test - tests/scoring_engine/web/web_test.py: Reuse parent app instead of creating new one This ensures all database operations in tests (including model class methods like Team.get_all_blue_teams(), Round.get_last_round_num(), etc.) have the required Flask application context.
This commit fixes three categories of test failures: 1. **Engine missing db attribute** - Added `self.db = db` to Engine.__init__ for backward compatibility - Engine code uses `self.db.session` throughout, so this reference is needed - File: scoring_engine/engine/engine.py 2. **Duplicate app context in TestEngine** - Removed duplicate app context creation in TestEngine.setup_method() - UnitTest parent class already creates app context - Creating a second context caused "popped wrong app context" errors - File: tests/scoring_engine/engine/test_engine.py 3. **Competition.save() signature mismatch** - Updated test to call `competition.save()` instead of `competition.save(self.session)` - Competition.save() no longer takes a session parameter (uses db.session internally) - File: tests/scoring_engine/test_competition.py All three issues stemmed from the Flask-SQLAlchemy migration where: - db.session is now managed by Flask-SQLAlchemy - Application contexts are required for all database operations - Session is no longer passed around as a parameter These fixes maintain backward compatibility while completing the migration.
The integration-get-round target was calling Round.get_last_round_num() without an application context, causing a RuntimeError. With Flask-SQLAlchemy, all database operations require an active app context. Updated the command to create an app and push context before calling the model method. This fixes the error: RuntimeError: Working outside of application context.
The integration tests were calling model methods and database queries at module import time (before app context was available): 1. ensure_integration_data() was called at module level (line 16) 2. @pytest.mark.parametrize decorators called db queries directly (lines 29, 36, 42) This caused 'RuntimeError: Working outside of application context' during test collection. Solution: 1. Created tests/integration/conftest.py with setup_integration_db fixture - Calls ensure_integration_data() after app context is created - Uses autouse=True and depends on app_context fixture 2. Replaced @pytest.mark.parametrize with pytest_generate_tests hook - pytest_generate_tests runs after fixtures are set up - App context is available when generating test parameters - Dynamically creates parametrized tests for blue_team, service, check This ensures all database operations during test collection happen inside an application context.
2b69709 to
531367e
Compare
- Changed session.query() to db.session.query() in check_result_for_round() - Changed session.query() to db.session.query() in last_ten_checks property - These were missed during the Flask-SQLAlchemy migration
The pytest_generate_tests hook runs during collection phase, before fixtures are executed. Need to create app context explicitly for database queries during test parametrization.
The pytest_generate_tests hook runs during collection and needs to query the database for test parametrization. Call ensure_integration_data() to initialize the database before running queries.
The last_check_result() method was accessing self.checks[-1] which relies on lazy loading. In integration tests, service objects created during collection are in a different session than during execution, causing lazy loading to fail. Changed to use an explicit query to avoid session issues.
Some test classes pop the context in their teardown before calling super(), which causes IndexError when the parent tries to pop again. Make teardown_method defensive by checking if context has tokens before popping.
Removed autouse=True from session-scoped app_context fixture. Tests like test_config_loader_env.py don't use Flask/SQLAlchemy and don't need an app context. Integration tests still get the context via their setup_integration_db fixture dependency. UnitTest-based tests create their own context in setup_method.
- test_concurrent_users.py: Python script to test concurrent user sessions - test_simple_concurrent.sh: Simple bash script for quick testing - TESTING_SESSION_ISOLATION.md: Comprehensive testing guide These tools help verify the Flask-SQLAlchemy migration properly isolates sessions between users and prevents data contamination.
This test verifies that Flask-SQLAlchemy properly isolates sessions between requests and prevents stale data issues. Test proves: - Each request gets a fresh session with empty identity map - External updates are visible in subsequent requests - No stale cached objects persist between requests This demonstrates the root cause fix: Flask-SQLAlchemy scopes sessions to app context (per-request) instead of using a global session, eliminating the need for the commit-before-query monkeypatch.
RustyBower
pushed a commit
that referenced
this pull request
Jan 22, 2026
This commit addresses two critical performance issues: 1. Remove lazy="joined" from Team model relationships (services, users, inject) - Changed to lazy="select" to prevent over-fetching data - Previously, every Team query loaded ALL related services, users, and injects - Now relationships are only loaded when explicitly accessed - Significantly reduces database load and memory usage 2. Fix redundant database query in /api/team/<team_id>/services endpoint - Use pre-loaded checks instead of calling service.last_ten_checks property - The checks were already loaded via subqueryload(Service.checks) - Eliminates N+1 query problem (was making extra DB query per service) - For 50 services, this saves 50 database queries per API call These changes build on top of PR #1051 which fixed the underlying session management and connection pooling issues. Performance Impact: - Reduces database queries by 50+ per API request - Eliminates over-fetching of unused relationship data - Lower memory usage when querying Teams - Better scalability under load
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

This commit addresses the most critical performance issues found in the codebase:
Remove NullPool - Switched from NullPool to proper connection pooling
Remove commit-before-query monkeypatch - Eliminated the performance-killing workaround
Implement proper session lifecycle management via Flask-SQLAlchemy
Add application context support for engine and worker processes
Changes:
Performance improvements:
This migration maintains backward compatibility while fixing the root cause of the caching/stale data issues that led to the monkeypatch workaround.