fix: Bedrock reranker credential chain fallback and eval script compatibility with LTM backend split by ssvaddiparthy · Pull Request #1585 · MemMachine/MemMachine · GitHub
Skip to content

fix: Bedrock reranker credential chain fallback and eval script compatibility with LTM backend split - #1585

Open
ssvaddiparthy wants to merge 1 commit into
MemMachine:mainfrom
ssvaddiparthy:fix/aws-eval-reranker
Open

fix: Bedrock reranker credential chain fallback and eval script compatibility with LTM backend split#1585
ssvaddiparthy wants to merge 1 commit into
MemMachine:mainfrom
ssvaddiparthy:fix/aws-eval-reranker

Conversation

@ssvaddiparthy

Copy link
Copy Markdown

Purpose of the change

Fix two issues that prevent MemMachine evaluation benchmarks from running out of the box:

  1. the Bedrock reranker requires explicit AWS credentials in the config YAML, blocking SSO/profile/login-based auth
  2. the eval helper uses the old LongTermMemoryParams class which was replaced by a backend-discriminated union in Wire EventMemory as backend for LongTermMemory #1395.

Description

Bedrock credential chain fix (reranker_conf.py + reranker_manager.py):

AmazonBedrockRerankerConf previously defined its own aws_access_key_id and aws_secret_access_key fields as required (Field(...)), meaning any config using the Bedrock reranker had to include explicit credentials — SSO, AWS CLI login, and profile-based auth couldn't work. This PR replaces the hand-rolled fields with AWSCredentialsMixin (which defaults to None with env-var fallback) and updates RerankerManager._build_amazon_bedrock_reranker to only pass credentials to boto3.client() when they're actually set. When omitted, boto3 falls through to its default credential chain (~/.aws/credentials, ~/.aws/config, SSO, instance metadata).

Eval script fix (agent_utils.py):

PR #1395 ("Wire EventMemory as backend for LongTermMemory") changed LongTermMemoryParams from a single concrete class to a Discriminator-based union of DeclarativeBackendParams | EventBackendParams. The evaluation helper init_memmachine_params still used the old LongTermMemoryParams, which no longer accepts the fields directly. This PR updates it to use DeclarativeBackendParams(backend="declarative", ...).

Tests (test_reranker_conf.py):

  • Added test_valid_amazon_bedrock_reranker_conf_without_explicit_credentials — verifies the config accepts no AWS credentials when env vars are unset
  • Added test_valid_amazon_bedrock_reranker_conf_uses_env_credentials — verifies env-var resolution populates credentials correctly
  • Removed test_missing_required_field_in_bedrock_reranker — this tested the old required-field behavior which no longer applies

Fixes/Closes

None

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g., code style improvements, linting)
  • Documentation update
  • Project Maintenance (updates to build scripts, CI, etc., that do not affect the main project)
  • Security (improves security without changing functionality)

How Has This Been Tested?

  • Unit Test
  • Test Script

Lint (all changed files):

uv run ruff check evaluation/utils/agent_utils.py \
  packages/server/src/memmachine_server/common/configuration/reranker_conf.py \
  packages/server/server_tests/memmachine_server/common/configuration/test_reranker_conf.py \
  packages/server/server_tests/memmachine_server/common/resource_manager/test_reranker_manager.py

Unit tests (config + manager):

uv run pytest \
  packages/server/server_tests/memmachine_server/common/configuration/test_reranker_conf.py \
  packages/server/server_tests/memmachine_server/common/resource_manager/test_reranker_manager.py

End-to-end eval (LoCoMo benchmark, retrieval_agent mode):

Tested the "no keys in config" path using a personal AWS account via boto3's default credential chain. The Bedrock reranker (amazon.rerank-v1:0, us-west-2) resolved credentials automatically and completed successfully — no AccessDeniedException, no ValidationError. All 10 conversation groups processed with the retrieval agent routing queries through MemMachineAgent, SplitQueryAgent, and ChainOfQueryAgent as expected.

./run_test.sh locomo exp1 search retrieval_agent

@ssvaddiparthy
ssvaddiparthy force-pushed the fix/aws-eval-reranker branch 3 times, most recently from 66178bb to 09bbdf7 Compare September 6, 2026 06:30
…pdate eval scripts for LongTermMemory backend split

- Make AmazonBedrockRerankerConf inherit AWSCredentialsMixin so AWS credentials
  are optional with env-var fallback, enabling SSO/profile/login-based auth
- Update RerankerManager to only pass explicit credentials to boto3 when set,
  falling back to the default credential chain otherwise
- Fix agent_utils.py to use DeclarativeBackendParams after MemMachine#1395 changed
  LongTermMemoryParams to a backend-discriminated union
- Add tests for no-creds and env-creds paths, remove obsolete required-field test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant