{{ message }}
Scala 3 migration: Lift Web to http4s, Lift Mapper to Doobie, Flyway to Liquibase - #2899
Open
hongwei1 wants to merge 309 commits into
Open
Scala 3 migration: Lift Web to http4s, Lift Mapper to Doobie, Flyway to Liquibase#2899hongwei1 wants to merge 309 commits into
hongwei1 wants to merge 309 commits into
Conversation
Twenty-first table off Lift Mapper. Fired from AfterApiAuth on every login to record one-off "has this user done X yet" flags (create-or-update-bank, add-entitlement, add-bank-account); nothing in the suite exercised it, so UserInitActionProviderTest is written first against the Mapper version to pin create-then-update-in-place, the (userId, actionName, actionValue) triple acting as the full key, and that different users do not collide. Every caller discards the return value of createOrUpdateInitAction - only the write matters - so the entity's replacement is a plain case class (UserInitActionRow) rather than anything wired through a provider trait; there was no trait here to begin with; UserInitActionProvider stays a plain object with no injector. The unique index on the full (userId, actionName, actionValue) triple is carried over explicitly and added to the guard test, for the same reason as every table so far: FlywayBaselineExport does not emit dbIndexes-declared unique indexes even though Schemifier creates them. It is what makes "find then update in place" correct - the whole point of this table is one row per triple.
Twenty-second table off Lift Mapper. Nothing exercised it before this change; AccountIdMappingProviderTest is written first, confirmed against the Mapper version before the entity was touched, then confirmed again against the Doobie provider: get-or-create keyed on accountPlainTextReference, the reverse lookup by accountId, and that different references get different ids. The provider stays named MappedAccountIdMappingProvider rather than a Doobie* one. DynamicUtil's compiled-code template hands this exact import to every dynamic connector method, and connector method bodies are stored as raw Scala source in the connectormethod table and compiled at request time - a bank's already-deployed dynamic connector code can reference this name by hand. Renaming the object would break that code on its next compile for no benefit; Helper.convertToId/convertToReference call it directly too. Both unique indexes are carried over and added to the guard test, for the same reason as every table so far: FlywayBaselineExport does not emit dbIndexes-declared unique indexes even though Schemifier creates them. Neither index actually constrains accountPlainTextReference on its own, though - only mAccountId is unique, and every insert gets a fresh random UUID for it - so two concurrent creates for the same accountPlainTextReference do not collide and can both succeed despite getOrCreateAccountId's comment describing a retry for exactly that collision. That gap is reproduced as-is rather than tightened here: it is a schema/business-rule question this table's existing rows already live under, not something to decide inside a migration whose job is preserving behaviour.
…hema Twenty-third table off Lift Mapper. Sibling of AccountIdMapping - same table shape, same provider shape, same schema gap. TransactionIdMappingProviderTest is written first and confirmed against the Mapper version: get-or-create keyed on transactionPlainTextReference, the reverse lookup by transactionId, and that different references get different ids. Unlike AccountIdMapping's provider, this one is not referenced by name from DynamicUtil's compiled-code template, so it is free to rename; DoobieTransactionIdMappingProvider replaces MappedTransactionIdMappingProvider, including at its one direct call site in Helper.convertToId/convertToReference. Both unique indexes are carried over and added to the guard test, for the same reason as every table so far: FlywayBaselineExport does not emit dbIndexes-declared unique indexes even though Schemifier creates them. As with the sibling table, neither index actually constrains transactionPlainTextReference on its own - only TransactionId is unique, and every insert gets a fresh random UUID for it - so the same concurrent-duplicate gap exists here and is reproduced rather than tightened, for the same reason: it is a schema question this table's existing rows already live under, not something to decide inside a migration whose job is preserving behaviour.
… schema Twenty-fourth table off Lift Mapper. Third of the id-mapping triplet (AccountIdMapping, TransactionIdMapping, this one) - same table shape, same provider shape, same schema gap already documented on the first two. CustomerIdMappingProviderTest is written first and confirmed against the Mapper version. MappedCustomerIdMapping had a second, non-provider caller: DeleteCustomerCascade.deleteCustomerIdMapping called MappedCustomerIdMapping.bulkDelete_!! directly. That moves to a plain DELETE through DoobieUtil - DeletionUtil.databaseAtomicTask wraps callers in DB.use(DefaultConnectionIdentifier), which is exactly the fallback DoobieUtil.currentRequestConnection already reads Lift's DB.currentConnection for, so the delete participates in the same Mapper transaction as the rest of the cascade. Covered by DeleteCustomerCascadeTest, unchanged. The provider stays named MappedCustomerIdMappingProvider rather than a Doobie* one, for the same reason as MappedAccountIdMappingProvider: DynamicUtil's compiled-code template hands this exact import to every dynamic connector method, and a bank's already-deployed dynamic connector code can reference it by hand. mBankId/mCustomerNumber are deprecated columns (since 2019-08-23, "We used customerPlainTextReference instead") that neither provider method returns anything carrying, so the migration keeps the columns without threading them through the new provider. Both unique indexes are carried over and added to the guard test, for the same reason as every table so far: FlywayBaselineExport does not emit dbIndexes-declared unique indexes even though Schemifier creates them. As with the other two id-mapping tables, neither index actually constrains mCustomerPlainTextReference on its own; reproduced rather than tightened here.
…chema Twenty-fifth table off Lift Mapper. Nothing in the current codebase creates or reads this table - the only reference anywhere was DeleteAccountCascade.deleteBankAccountData bulk-deleting from it, presumably a leftover of a feature that used to write here. That delete moves to a plain SQL DELETE through DoobieUtil; the table itself is kept rather than dropped, since a production instance may still hold rows from whenever this was in active use, and cascade delete needs a real table to clear them from. The unique index on (bankId, accountId) is carried over and added to the guard test, for the same reason as every table so far: FlywayBaselineExport does not emit dbIndexes-declared unique indexes even though Schemifier creates them. Covered by DeleteAccountCascadeTest, unchanged.
Twenty-sixth table off Lift Mapper. No injector sits in front of the provider - callers referenced MappedApiCollectionsProvider directly, same as ApiCollectionEndpoint and FeaturedApiCollection before it - so this moves the object itself to DoobieApiCollectionsProvider and updates every call site (NewStyle, Http4s400, and ExampleValue's glossary text, which read ApiCollection.Description.maxLen off the Mapper field metadata and now states the same 2000-character limit as a literal). ApiCollectionTrait moves from the entity file into the provider file, since nothing else declared it. Both unique indexes are carried over explicitly and added to the guard test, for the same reason as every table so far: FlywayBaselineExport does not emit dbIndexes-declared unique indexes even though Schemifier creates them. The one on (userId, apiCollectionName) is what stops one user creating two collections with the same name - createApiCollection does not check first, it relies on the database rejecting the duplicate. updateApiCollectionById and deleteApiCollectionById keep their find-then-write/ find-then-delete shape and stay Empty for a missing id rather than Full(false): both of NewStyle's callers unbox the result with unboxFullOrFail, which only turns a missing row into an error on Empty. Covered end to end by both v4.0.0 and v5.1.0 ApiCollectionTest, and indirectly by FeaturedApiCollectionsProviderTest, which creates api collections as fixtures.
…chema
Twenty-seventh table off Lift Mapper. Security-critical - it backs account
lockout - and already partially prepared for this: DoobieBadLoginAttemptQueries
existed with an atomic UPDATE ... SET counter = counter + 1 for the concurrent
lost-update fix documented in CONCURRENCY_HAZARDS.md (hazard H), used only for
the increment path while every other operation still went through the Mapper
entity directly. This finishes the table: find, create, and resetBadLoginAttempts
move into the same object, and LoginAttempt (code.loginattempts.LoginAttempts.scala)
now goes through it end to end rather than mixing Doobie and Mapper calls.
Two other direct callers of the entity, outside the provider:
- LiftUsers.getUsers (locked/active user filtering) called
MappedBadLoginAttempt.findAll(By_>(...)) directly to find usernames over the
attempt threshold; that becomes
DoobieBadLoginAttemptQueries.usernamesOverThreshold.
- ConcurrentSecurityRaceTest's own fixture setup and assertion (scenario H)
used the Mapper API directly to seed and read the counter; both move to the
same Doobie queries the production code now uses. The scenario still passes
with all 8 concurrent increments landing, which is the actual regression
test for the atomic-update fix - if migrating this table had reintroduced a
read-modify-write race, this would be the test to catch it.
MigrationOfMappedBadLoginAttemptDropIndex - a historical migration that already
ran everywhere - no longer references the deleted entity; it checks for the
table by name instead of via DbFunction.tableExists(MetaMapper).
The unique index is carried over and added to the guard test, for the same
reason as every table so far: FlywayBaselineExport does not emit
dbIndexes-declared unique indexes even though Schemifier creates them. It is not
the index the historical migration drops - that one constrained mUsername alone
and would have rejected the same username under two different providers; this
one is (provider, mUsername).
Twenty-eighth table off Lift Mapper - the widest blast radius so far. The entity was not behind a single provider: it was reached from seven files directly (ConsentUtil, MigrationOfAccountRoutings, LocalMappedConnector, LocalMappedConnectorInternal, MappedBankAccount, LocalMappedConnectorDataImport, DeleteAccountCascade), and its two read methods (getAccountRouting/getAccountRoutingsByScheme) are part of the public Connector trait interface, returning the concrete Mapper type. DoobieBankAccountRoutingQueries now holds every query these call sites need; Connector.scala and NewStyle.scala's two signatures move to BankAccountRoutingTrait (obp-commons), the same trait the entity already implemented, so nothing downstream that only reads .bankId/.accountId/.accountRouting off the result needed to change. getBankAccountByRoutingLegacy's OBP-family fallback logic (try the implicit account-id reading first, fall back to a registered routing) and updateBankAccount's diff-based add/update/delete of routing schemes are ported statement-for-statement rather than restructured - both encode non-obvious behaviour with their own regression coverage (ObpAccountRoutingResolutionTest for the former). MigrationOfAccountRoutings - a historical migration - no longer references the deleted entity: its tableExists check moves to tableExistsByName, and its private, unreferenced createBankAccountRouting helper (not called by populate() or anything else, kept rather than deleted) is rewritten against DoobieBankAccountRoutingQueries instead of quietly dropped. Eight test files reached the entity directly as fixture setup rather than through any provider: five Berlin Group suites (AIS/PIIS/PIS/SBS + BerlinGroupConsentFixtures), SandboxDataLoadingTest's six unconditional bulkDelete_!! resets, ObpAccountRoutingResolutionTest (the OBP-scheme regression test), and LocalMappedConnectorTestSetup. All move to the same Doobie queries the production code now uses. Both unique indexes are carried over and added to the guard test, for the same reason as every table so far: FlywayBaselineExport does not emit dbIndexes-declared unique indexes even though Schemifier creates them. Both are read directly by application code (getBankAccountByRoutingLegacy, getAccountRouting) rather than only relied on implicitly. Covered end to end by the five Berlin Group suites, v3.1.0 AccountTest, v7 Http4s700RoutesTest (153 scenarios), and ObpAccountRoutingResolutionTest - 253 scenarios total, all green including the OBP-scheme fallback regression test.
Twenty-ninth and thirtieth tables off Lift Mapper, done together: MappedFXRate declares a Lift foreign key on MappedCurrency for both its currency-code columns, so neither could move independently. MappedCurrency is deleted outright rather than migrated. It has zero rows, zero application-level reads or writes anywhere in the codebase, and the FK it exists to be the target of was never actually enforced - confirmed by inserting an FX rate for a currency pair absent from MappedCurrency, which succeeded. It is dead code in the same sense PemUsage was: present in the schema, referenced by nothing at runtime. FXRateProviderTest is written first and confirmed against the Mapper version. ExchangeRateTest only covers NewStyle.getExchangeRate's fallback path, which builds an FXRate value without ever calling .saveMe() on it - a real gap, since nothing exercised createOrUpdateFXRate (the actual write path) or getCurrentFxRate's reverse-order lookup. getCurrentFxRate's reverse-order fallback and createOrUpdateFXRate's find-then-write are preserved exactly, including the gap that comes with them: the table has no unique index (only plain indexes on the two currency-code columns, matching Schemifier's real output), so two genuinely concurrent calls for the same (bankId, from, to) can both miss the find and both insert - the same shape as the id-mapping tables' documented gap, not something this migration changes. NewStyle.getExchangeRate's fallback branch keeps its "build without persisting" behaviour, now constructing FXRateRow (a plain case class) instead of an unsaved Mapper instance - there was never a database write on this path to begin with.
Thirty-first table off Lift Mapper - migration bookkeeping itself, the table every historical migration script (including several already ported in this series) reads and writes through Migration.saveLog/isExecuted via MigrationScriptLogProvider.vend. Nothing about that seam changes; only the implementation behind it does. ServerSetup.resetDatabaseForTestClass deliberately excludes this table from its per-test-class wipe: clearing it makes isExecuted always false, so a fresh test JVM would re-run every historical migration against a database that already has their effects, and a migration that retypes a view-projected column then fails outright. That exclusion was an identity check against the Mapper object (`m == MigrationScriptLog`) in a filter over ToSchemify.models; with the entity gone, the table is simply never in that list to begin with, so the check is removed rather than replaced. Every other migrated table gets an explicit DoobieUtil DELETE line in the same function - this is the one deliberate exception, called out in a comment where that DELETE list lives so it isn't added by reflex on the next table. The unique index on (name, isSuccessful) is carried over and added to the guard test, for the same reason as every table so far: FlywayBaselineExport does not emit dbIndexes-declared unique indexes even though Schemifier creates them. saveLog's find-then-write keys on exactly that pair. Covered by MigrationsTest end to end, and by the full suite staying green across every shard's many test classes in one run - the actual regression test for the exclusion, since a reintroduced wipe would only surface as a boot failure partway through a shard's test classes, not in any single test.
…he schema Thirty-second table off Lift Mapper. A write-only audit record: nothing in the codebase reads it back. LocalMappedConnector.saveTransactionRequestReasons writes rows alongside a transaction request's creation and never queries this table again. TransactionRequestReasonsProviderTest reads rows back directly to confirm the write itself is correct, since there is no production read path whose test would otherwise catch a column-mapping mistake. No unique index - only the primary key, matching Schemifier's real output. That is expected here, not a gap: multiple reasons naturally attach to one transactionRequestId, and nothing about the table was ever meant to enforce one-per-anything.
Thirty-third table off Lift Mapper. No injector sits in front of the provider - NewStyle called MappedApiProductAttributesProvider directly - so this moves the object itself to DoobieApiProductAttributesProvider and updates the one call site. ApiProductAttributeTrait moves into the provider file, since nothing else declared it. Nothing in the suite exercised this table before this change; ApiProductAttributesProviderTest is written first and confirmed against the Mapper version. createOrUpdateApiProductAttribute keeps its exact lookup shape: by apiProductAttributeId, not by (bankId, apiProductCode) - a bank/product pair can carry more than one attribute with the same name at once, so the unique index (and this lookup) is on the id alone, and a supplied id with no matching row falls back to create. The unique index on apiProductAttributeId is carried over and added to the guard test, for the same reason as every table so far: FlywayBaselineExport does not emit dbIndexes-declared unique indexes even though Schemifier creates them.
… the schema Thirty-fourth table off Lift Mapper - the SCA-style challenge/answer flow for updating a user auth context. Already partially prepared for this the same way MappedBadLoginAttempt was: DoobieUserAuthContextUpdateQueries existed with an atomic conditional UPDATE for checkAnswer's TOCTOU fix (CONCURRENCY_HAZARDS.md hazard H2, exercised by ConcurrentConsentStatusRaceTest), used only for the status-transition path while find/create/delete still went through the Mapper entity. This finishes the table without touching that fix. createUserAuthContextUpdates never set challenge explicitly - the Mapper version relied on mChallenge's field default (SecureRandomUtil.csprng.nextInt(99999999), an up-to-8-digit numeric OTP) firing on an unset field. That default is now generated explicitly at the call site rather than implicitly by a field's defaultValue override, since there is no field to override. ConcurrentConsentStatusRaceTest's H2 scenario used the Mapper entity directly for its own fixture setup and status readback; those two helpers move to the same Doobie queries the production code now uses. Its H1/H3/M5 scenarios exercise MappedConsent, a separate table not touched here, and are left alone. Migration.scala's alterTableMappedUserAuthContextUpdate() derived its migration's log-entry name via nameOf(MappedUserAuthContextUpdate) - a compile-time macro over the now-deleted object. That name is the key already recorded in migration_script_log on every environment that has run this migration, so it becomes the literal string the macro produced rather than a fresh name. MigrationOfMappedUserAuthContextUpdate itself moves from DbFunction.tableExists(MetaMapper) to tableExistsByName, same as the other historical migrations already ported. No unique index - only the primary key, matching Schemifier's real output (the entity's own dbIndexes was `super.dbIndexes`, adding nothing). MigrationOfMappedUserAuthContextUpdate drops a legacy index that predates this and was already gone before this migration.
Replace the Lift Mapper card-attribute entity with a Doobie-backed provider (thirty-fifth table off Lift Mapper). No unique index exists on this table - only plain indexes on mCardId and mCardAttributeId, matching the entity's own dbIndexes declaration and confirmed against a booted instance's information_schema.indexes. createOrUpdateCardAttribute preserves the exact find-by-cardAttributeId then update-or-create shape, including the nullable bankId/cardId fallback behaviour on create. Also fills in five migrated tables (mappedfxrate, migrationscriptlog, transactionrequestreasons, apiproductattribute, mappeduserauthcontextupdate) that were missing from MigratedTablesExistTest's existence-check list from earlier migrations in this series.
Replace the Lift Mapper ATM-attribute entity with a Doobie-backed provider (thirty-sixth table off Lift Mapper). No unique index exists on this table - only a plain composite index on (BankId, AtmId), matching the entity's own dbIndexes and confirmed against a booted instance's information_schema.indexes. The Type column is stored as type_c, since Lift Mapper suffixes reserved SQL words and TYPE collides with H2's reserved keyword. The entity type leaked into public signatures across Connector.scala, NewStyle.scala, LocalMappedConnector.scala, JSONFactory5.1.0.scala and Http4s510.scala; all of those now use the existing obp-commons AtmAttributeTrait instead of the concrete Mapper class. AtmTest's direct AtmAttribute.findAll() row-count assertion moves to a raw SQL count query.
Replace the Lift Mapper bank-attribute entity with a Doobie-backed provider (thirty-seventh table off Lift Mapper). No unique index exists on this table - only a plain index on bankid_, matching the entity's own dbIndexes and confirmed against a booted instance's information_schema.indexes. The BankId_ Mapper field has no dbColumnName override, so the column keeps the trailing underscore (bankid_) rather than being renamed like AtmAttribute's BankId_/AtmId_ were. The Type column is stored as type_c for the same reserved-word reason as AtmAttribute. The entity type leaked into public signatures across Connector.scala, NewStyle.scala and LocalMappedConnector.scala; those now use the existing obp-commons BankAttributeTrait instead of the concrete Mapper class.
Replace the Lift Mapper counterparty-attribute entity with a Doobie-backed provider (thirty-eighth table off Lift Mapper). No unique index exists on this table - only a plain index on counterpartyid, matching the entity's own dbIndexes and confirmed against a booted instance's information_schema.indexes. The Type column is stored as type_c for the same reserved-word reason as AtmAttribute/BankAttribute. Unlike those two, this entity's callers already went through code.api.util.newstyle.CounterpartyAttributeNewStyle, which was already typed against the obp-commons CounterpartyAttributeTrait, so only the provider trait itself and Boot.scala needed updating.
Replace the Lift Mapper regulated-entity-attribute entity with a Doobie-backed provider (thirty-ninth table off Lift Mapper). No unique index exists on this table - only a plain index on regulatedentityid, matching the entity's own dbIndexes and confirmed against a booted instance's information_schema.indexes. The Type column is stored as type_c for the same reserved-word reason as the other *Attribute tables migrated so far. MappedRegulatedEntity.attributes (still Mapper-backed, migrates separately) read this table directly via a cross-table Mapper query; it now calls DoobieRegulatedEntityAttributeProvider's synchronous helper instead.
Replace the Lift Mapper product-attribute entity with a Doobie-backed provider (fortieth table off Lift Mapper). No unique index exists on this table - only plain indexes on mBankId and mProductAttributeId, confirmed against a booted instance's information_schema.indexes. Unlike AtmAttribute/BankAttribute/CounterpartyAttribute/ RegulatedEntityAttribute, the Type column here (mType) does not collide with H2's reserved TYPE keyword, so no reserved-word renaming applies. Four call sites read or wrote this table directly through the Mapper entity and now go through DoobieProductAttributeProvider instead: - LocalMappedConnector.getProducts's attribute-filter query, ported to a Doobie Fragment that reproduces the same OR-across-attributes row match semantics as the original BySql filter (exercised by ProductTest's "getProducts by url parameters" scenario). - deletion.DeleteProductCascade's cascade delete. - MappedProductCollectionItemProvider.getProductCollectionItemsTree's read of a product's attributes. - MigrationOfProductAttribute, a historical one-time backfill of the isActive column, switched to raw SQL via the tableExistsByName/ makeBackUpOfTableByName overloads already used by other historical migrations in this series.
Replace the Lift Mapper customer-attribute entity with a Doobie-backed provider (forty-first table off Lift Mapper). No unique index exists on this table - only plain indexes on mCustomerId and mCustomerAttributeId, confirmed against a booted instance's information_schema.indexes. mBankId is stored under the column mbankidid, a historical typo baked into the entity's own dbColumnName override, preserved as-is. getCustomerIdsByAttributeNameValues previously built a Mapper BySql(...) filter via AttributeQueryTrait's getSqlParametersFilter/ getParameters; it now builds the equivalent Doobie Fragment directly, reproducing the same OR-across-attributes row match semantics. No endpoint test exercised this path, so a provider-level characterization test (CustomerAttributeProviderTest) was added and confirmed green against the pristine Mapper entity before the migration, then again against the Doobie provider. Two other call sites read/wrote this table directly through the Mapper entity and now go through DoobieCustomerAttributeProvider or raw SQL: deletion.DeleteCustomerCascade's cascade delete, and MigrationOfCustomerAttributes's historical column-width migration (switched to the tableExistsByName overload).
Replace the Lift Mapper account-attribute entity with a Doobie-backed provider (forty-second table off Lift Mapper). No unique index exists on this table - only plain indexes on mAccountId and mAccountAttributeId, confirmed against a booted instance's information_schema.indexes. getAccountAttributesByAccountCanBeSeenOnView and getAccountAttributesByAccountsCanBeSeenOnView still read AttributeDefinition (a separate, not-yet-migrated Mapper entity) directly and join in plain Scala, exactly as before - only the AccountAttribute-table reads moved to Doobie, including the ByList(mAccountId, ...) multi-account read via Fragments.in. getAccountIdsByParams previously built a Mapper BySql(...) filter via AttributeQueryTrait; it now builds the equivalent Doobie Fragment directly, reproducing the same OR-across-attributes row match semantics. This path backs getFirehoseAccounts filtering and several other endpoints across v3-v6 with no direct endpoint test coverage, so a provider-level characterization test (AccountAttributeProviderTest) covering CRUD, the filter semantics, and both view-visibility methods was added and confirmed green against the pristine Mapper entity before the migration, then again against the Doobie provider. Two cascade-delete call sites move to the new provider: deletion.DeleteBankCascade's "customer_number" attribute lookup and deletion.DeleteAccountCascade's cascade delete.
Replace the Lift Mapper transaction-attribute entity with a Doobie-backed provider (forty-third table off Lift Mapper). No unique index exists on this table - only plain indexes on mTransactionId and mTransactionAttributeId, confirmed against a booted instance's information_schema.indexes. getTransactionAttributesCanBeSeenOnView and getTransactionsAttributesCanBeSeenOnView still read AttributeDefinition (a separate, not-yet-migrated Mapper entity) directly and join in plain Scala, exactly as before - only the TransactionAttribute-table reads moved to Doobie, including the multi-transaction ByList read via Fragments.in. getTransactionIdsByAttributeNameValues previously built a Mapper BySql(...) filter via AttributeQueryTrait; it now builds the equivalent Doobie Fragment directly, reproducing the same OR-across-attributes row match semantics. No endpoint test exercised this filter path or the multi-transaction view-visibility method, so a provider-level characterization test (TransactionAttributeProviderTest) covering CRUD, the filter semantics, and both view-visibility methods was added and confirmed green against the pristine Mapper entity before the migration, then again against the Doobie provider. deletion.DeleteTransactionCascade's cascade delete and V400ServerSetup's shared "no related data left" test helper both move to the new provider.
Follow-up to the previous commit - the new provider test file was written and run but not staged.
Replace the Lift Mapper transaction-request-attribute entity with a Doobie-backed provider (forty-fourth table off Lift Mapper). No unique index exists on this table - only plain indexes on transactionrequestid and transactionrequestattributeid, confirmed against a booted instance's information_schema.indexes. The Type column is stored as type_c for the same reserved-word reason as the other *Attribute tables; Value is unbounded CHARACTER VARYING (same pattern as V015's connectormethod.methodbody), since Open Corridor promise evidence stores a full preimage JSON that exceeds any fixed varchar bound. Two pre-existing quirks in the Mapper version are preserved verbatim rather than fixed: getTransactionRequestAttributesCanBeSeenOnView filters AttributeDefinition by AttributeCategory.Account instead of .TransactionRequest, and getByAttributeNameValues always queries WHERE ispersonal = true regardless of the isPersonal argument it receives. Existing coverage (TransactionRequestTest's "getProducts by url parameters"-equivalent scenario, TransactionRequestAttributesTest, and Http4s700RoutesTest's Open Corridor promise/settlement scenarios) exercises the filter path and the two direct-query call sites in OpenCorridorSettlement (hasPromiseEvidence, coveredTrIds), so no new characterization test was needed this round. MigrationOfTransactionRequestAttributeValueType, a historical migration, switches to the tableExistsByName overload used by the other historical migrations in this series.
The previous commit rewired the provider to Doobie and removed the Mapper entity from Boot.scala's ToSchemify list, but missed deleting the entity class itself - it lived in its own file, separate from the provider file that got deleted. No remaining references; full suite still green.
Replace the Lift Mapper tax-residence entity with a Doobie-backed provider (forty-fifth table off Lift Mapper). mCustomerId is a MappedLongForeignKey pointing at mappedcustomer.id (the customer's internal BIGINT primary key, not their UUID customerId) - the Doobie provider resolves customerId back to the UUID via a MappedCustomer lookup by id, falling back to the raw long id as a string if the customer row is missing, matching the Mapper entity's own getter exactly. Two indexes carried over: a plain index on mcustomerid (from the foreign-key field) and a UNIQUE INDEX on (mcustomerid, mdomain, mtaxnumber), confirmed against a booted instance's information_schema.indexes - caught a first attempt at the migration script that used ALTER TABLE ... ADD CONSTRAINT ... UNIQUE, which H2 backs with an auto-suffixed index name rather than the literal constraint name; switched to CREATE UNIQUE INDEX to match Schemifier's actual output, per MigratedTablesExistTest. deletion.DeleteCustomerCascade's cascade delete moves to raw SQL against the new table.
Replace the Lift Mapper customer-link entity with a Doobie-backed provider (forty-sixth table off Lift Mapper). Unique index on customerlinkid; plain indexes on customerid and othercustomerid, confirmed against a booted instance's information_schema.indexes. No test exercised this table's provider or the connector methods wired to it (bank-to-bank customer relationships, e.g. spouse/parent at another bank), so a characterization test (CustomerLinkProviderTest) covering full CRUD plus bulkDelete was added and confirmed green against the pristine Mapper entity before the migration, then again against the Doobie provider.
Replace the Lift Mapper counterparty-limit entity with a Doobie-backed provider (forty-seventh table off Lift Mapper). Two unique indexes: one on counterpartylimitid, one on the composite (bankid, accountid, viewid, counterpartyid) - at most one limit per tuple - confirmed against a booted instance's information_schema.indexes. Amount fields are NUMERIC(16,10); count fields default to -1 and amount fields to 0 at the application layer on create, matching the Mapper fields' own defaultValue overrides. toJValue moves onto the new CounterpartyLimitRow case class verbatim, since CounterpartyLimitTrait extends JsonAble. MigrationOfCounterpartyLimitFieldType, a historical migration, switches to the tableExistsByName overload used by the other historical migrations in this series.
Replace the Lift Mapper customer-account-link entity with a Doobie-backed provider (forty-eighth table off Lift Mapper). Two unique indexes: one on customeraccountlinkid, one on the composite (accountid, customerid) - a customer has at most one link per account - confirmed against a booted instance's information_schema.indexes. createAgentAccountLink builds its own AgentAccountLinkTraitCommons from the CustomerAccountLinkTrait result fields (documented in the entity as "customer and agent share the same model"), so the row type only needs to implement CustomerAccountLinkTrait. The endpoint test covers create/read/update/delete, but not getOrCreateCustomerAccountLink, the unfiltered getCustomerAccountLinks, or bulkDeleteCustomerAccountLinks, so a characterization test (CustomerAccountLinkProviderTest) covering those three was added and confirmed green against the pristine Mapper entity before the migration, then again against the Doobie provider. Two cascade-delete call sites move to the new provider: deletion.DeleteBankCascade's account-scoped lookup (filters by accountId only, no bankId) and deletion.DeleteCustomerCascade's cascade delete. A third direct reference in LocalMappedConnector.getBankAccountsForUser also moves over.
Replace the Lift Mapper user-customer-link entity with a Doobie-backed provider (forty-ninth table off Lift Mapper). Two unique indexes: one on musercustomerlinkid, one on the composite (muserid, mcustomerid) - a user has at most one link per customer - confirmed against a booted instance's information_schema.indexes. mdateinserted is a separate column from the createdat/updatedat pair the CreatedUpdated mixin also adds; the trait's dateInserted getter reads the former. getOCreateUserCustomerLink preserves the Mapper version's find-then-insert-with-retry-on-conflict shape exactly, including the scala.util.Try wrapping around the insert: ConcurrentDuplicateCreationTest scenario L races 8 concurrent calls for the same (userId, customerId) and asserts no exception and exactly one row, relying on the unique index to reject the losing insert so it can be caught and retried as a re-fetch. Confirmed green against the pristine Mapper entity first, then again after the migration. Two direct callers by name (not through the DI seam) move to the new object: deletion.DeleteCustomerCascade's cascade delete and MappedCustomerProvider.getCustomersByUserId.
…self stands 125950a scoped the javassist pool per classloader and its message presented that as the proven cause of two failures seen at the time - DynamicUtilTest and InternalConnectorTest reporting "missing reference, looking for JValue/T in package object json4s" - "verified by isolation". That attribution was wrong. The cause was a cross-checkout ~/.m2 overwrite: another checkout's mvn install replacing com.tesobe:obp-commons, which carries no Scala suffix, so nothing detects the mismatch. The error named it four lines below the one that gets read - "A signature in ~/.m2/.../obp-commons-1.10.1.jar refers to JValue/T in package object org.json4s.package which is not available ... the version on the classpath might be incompatible with the version used when compiling" it. Reading only the first line sent the investigation to dotty and javassist instead. Established by measurement, not inference. Fingerprinting the jar during a run caught the swap live, with the offending maven process's working directory recorded alongside it, and the other checkout confirmed both installs. Running the suite against an isolated repository (-Dmaven.repo.local, seeded with hard links so no dependency is re-downloaded) gives 3870/0 on H2 and Postgres with the scoping in place and nothing else changed. The earlier isolation experiment was confounded: a green run only meant ~/.m2 happened to be correct that time. Why it took so long is worth recording, because it is not about care. The same overwrite fails in opposite directions on the two lines: on Scala 2.13 it breaks compilation immediately and loudly (not found: value JsonSerializers), while here it compiles and waits to fail at runtime as a bad symbolic reference. The direction of the failure mode sets the cost of diagnosis. The scoping itself is unaffected and stays: a process-wide singleton that grows a search path per classloader and never releases one is a hazard under forkMode=once, where one JVM runs a whole shard. Fixing the right thing and explaining it wrongly are different mistakes; only the explanation is retracted. The note now sits at getClassPool, where the next reader of that code will find it.
…h relies on it cfb55aa scoped the sandbox import's duplicate-IBAN check by bank, on the premise that two banks may legitimately hold one IBAN. That premise is wrong. ISO 13616 encodes the institution in the string, so a shared IBAN is not a per-bank address space, it is bad data. More concretely, this instance depends on global uniqueness. Payment target accounts are resolved by routing with no bank context - BulkPaymentHandler:135, three Http4s700 transaction-request endpoints, getBankAccountByIban, and the to-account resolution inside the connector all pass bankId = None - and LocalMappedConnector.getBankAccountByRouting fails any lookup matching more than one row ("Routing MUST be unique"). Admitting a duplicate at import therefore does not produce a usable account; it produces one that fails every global-routing payment, reporting AccountRoutingNotUnique far from the cause. Trading a clear, actionable rejection at import for an obscure failure at payment time is the wrong trade. The unique index on (bankId, scheme, address), which the reverted commit cited, does not license the opposite reading: it is a storage constraint, and a per-bank index cannot authorise duplicates while a bank-less lookup exists. Storage constraint is not domain rule - that is the mistake worth naming, because the index really does say what it says. What actually broke was the fixtures. The 14 accounts in example_import.json carried seven strings shared across obp-bank-x-gh and obp-bank-y-gh; none was a valid IBAN (27 characters where Bosnia's is 20, and mod-97 of 36/52/50/65/79/57/45 where a valid IBAN gives 1), and all seven encoded the same institution while being attached to two banks. 2016-04-28/example_import.json had the same defect, one string shared across psd201-bank-x--uk and psd201-bank-y--uk. All 16 are regenerated as 20 characters, mod-97 = 1, globally unique, with a distinct bank code per bank (199/299/399/499 allocated across both files, so the two fixtures cannot collide when imported into one database). Values are replaced in place rather than by re-serialising the JSON, keeping the diff to one line per IBAN. The test that cfb55aa added is inverted accordingly: the same IBAN at a different bank must be rejected, and neither account may be created. Proven by negative control - restoring the per-bank scoping fails it at exactly the assertion that matters (201 did not equal 400) and nothing else. Not widened, deliberately: existingIbans still looks up per bank, so an IBAN already held at one bank is not detected when importing at another. That gap predates cfb55aa and closing it reaches into paths this change does not cover. Reported by a session validating the same import on develop-obp; verified independently here before acting.
107919d made compileScalaCode refuse on a JVM where no SecurityManager can be installed unless allow_user_generated_scala_code_without_sandbox is set. That is a behaviour change for anyone already running with allow_user_generated_scala_code=true on JDK 24+: dynamic code compilation starts failing with OBP-50021 after the upgrade. Until now it was stated only in that commit's message and in a comment in sample.props.template, neither of which an operator reads before upgrading. Written in the section format the server_mode removal already uses - what changed, then a Migration block with before/after props - so it sits where someone looking for breaking changes will find it. Includes the warning that belongs next to the switch rather than only in the props file: the feature compiles and runs Scala supplied over the API, so on a JVM with no enforceable sandbox, enabling it grants callers the privileges of the OBP-API process. Default deployments are unaffected; the feature is off unless explicitly turned on. Documentation only - no test or CI check reads README.md, verified before skipping the suite.
The fixture a new deployment imports had no test at all: example_import.json appears in the codebase only as a documentation link inside a ResourceDoc description. That is how seven 27-character strings with a failing mod-97 - each shared by two banks, each encoding a third - shipped as "IBAN"s and survived until a fresh-database run tripped over them, and it is why 0e0fcbb's regeneration of those values was not covered by any suite that ran green over it. The test posts the file through the same v2.1.0 data-import endpoint it is shipped for and requires 201. Proven to bite: restoring the pre-0e0fcbbf6 file turns it red, so it guards the fixture rather than merely reading it. 2016-04-28/example_import.json is deliberately not covered - it is rejected today, and would make this permanently red. Pre-existing, not caused by the IBAN work: the pre-change file fails the same way, and so does that fixture when imported alone, which rules out interference from the other import. Cause established since, and it is data rather than code - its accounts name owners by email (robert.xuk.x@example.com) while validateAccount matches user_name, and the file's own users section contains a different generation entirely (Robert.X.0.GH); separately, both its accounts carry an identical account id. Left alone because which way to reconcile that is a product decision, not a mechanical fix. Worth recording alongside: the import reports these as OBP-50005 "unspecified or internal error", discarding the per-check messages validateAccount builds ("Accounts must have owner(s) defined in data import. Violation: ..."). The diagnosis above came from reading the fixture, not from the response.
…s empty ReflectUtils.getNameToValues selects members with symbol.isVal || symbol.isVar. Both answer from Scala's own declaration metadata - ScalaSig on Scala 2, TASTy on Scala 3 - and scala.reflect.runtime.universe, the Scala 2.13 reflection library obp-commons is pinned to, has no TASTy reader: for a Scala-3-compiled class both come back false for every member. The function returned an empty map, and the allFields collectors built on it returned empty lists. SwaggerDefinitionsJSON declares 777 lazy vals and produced 0, measured directly. Nothing failed, which is the point: an empty list is a legal result, and every scenario that maps over allFields passed by doing nothing. SwaggerFactoryUnitTest had three such scenarios. getFieldValues in the same file already recovers from this - what survives into bytecode is the shape, a zero-arg method declared on the class itself with a backing field of the same name, or name$lzy… as Scala 3 spells a lazy val's field. That predicate is now shared rather than copied, and getNameToValues applies it as an additional branch, so Scala 2's isVal/isVar path is untouched (obp-commons' own ReflectUtilsTest, which runs on 2.13 where the bug cannot occur, still passes). includeVar = false still cannot exclude a Scala 3 var, for exactly the reason isVar fails there; documented at the function rather than silently approximated. Making the collector work exposed a mismatch it had been hiding. allFields fed everything non-null to SwaggerJSONFactory.translateEntity, which reads an entity's constructor arguments and therefore only means anything for a case class; several members here are plain values, and a PEM certificate string among them threw. Restricted to ReflectUtils.isObpObject. The regression test lives in obp-api, not beside ReflectUtils, because obp-commons compiles on 2.13 where the bug cannot be reproduced - and its sample object is top-level, because scala-reflect cannot load the symbol of an object nested in a class and the test would fail on its own fixture instead. SwaggerFactoryUnitTest now asserts a floor on allFields.size before using it: a floor well under the declared count, so it fails when the collector breaks rather than when someone adds a field.
ResourceDocMiddleware builds its lookup index from a version's own resourceDocs. A version with no entries gets no doc match, so authentication, role checks and entity resolution never run and the caller sees a bare 401. That was reported from a real-jar process for v3.0.0, five times, with the matcher's own debug line as the evidence - "Index keys for apiVersion=v3.0.0:" followed by nothing. It has never reproduced in a Maven test JVM, and the cause is still open. This does not reproduce it. It turns the one directly observed condition into an assertion, so an empty registration fails here naming the version instead of surfacing as an unexplained 401 wherever it happens to land. Writing it corrected a claim in that investigation. Its ruled-out list dismissed an initialisation cause because "gate takes routes by-value, so wrappedRoutesV300Services is always evaluated, touching the object". Evaluating it does happen and does not touch the object: wrappedRoutesVxxxServices is a Kleisli whose reference to Implementations… sits inside the lambda, so evaluating it builds a function and never runs the nested object's initialiser - which is where every resourceDocs += lives and where the index is built. Forcing all thirteen routes values leaves resourceDocs at 0 for twelve of them. That is a lead, not a cause: the first real request does initialise the object, and the ordering inside it is sound (Http4s300's last registration is at line 2219, the index at 2306), so the normal path self-heals. What it does show is that an initialiser failing once would leave the index permanently empty - the shape of an ExceptionInInitializerError cascade rather than of a routing bug, which is where the next attempt at this should look. The test touches the nested Implementations object for that reason. Touching the routes value instead measures nothing, as two earlier drafts of this test demonstrated by "finding" twelve empty versions that were merely uninitialised.
…ie stores develop added provenance to the three runtime-compiled-code entities - createdByUserId, updatedByUserId and a SHA-256 of the decoded method body, plus CreatedUpdated's timestamps - and exposed them on new read-only v7.0.0 endpoints while deliberately leaving the v4 responses frozen. It did that on the Lift Mapper entities. This branch had already moved DynamicResourceDoc, DynamicMessageDoc and ConnectorMethod to Doobie, so the resolution carries the fields across rather than restoring the entities: same columns, same server-side origin (the CallContext user and a hash computed in the provider, never the request body), same frozen v4 contract. ConnectorMethod had no entity left to hang the extra columns on, so the provenance read is a separate ConnectorMethodWithProvenance rather than more fields on JsonConnectorMethod - that one is the create/update request contract, and widening it would let a caller submit the values the server is supposed to set. ChatEmailDigestState arrived as a new Mapper entity, the first since this branch emptied ToSchemify.models. Carried across to Doobie for a reason that is not stylistic: with models empty Schemifier creates nothing, so a Mapper entity here compiles and then fails at runtime against a table nothing ever made. Its table comes from the changelog now, with the unique index the entity declared. Schema in db.changelog-provenance.yaml, a new file rather than an addition to the baseline, which is generated and would lose hand-written changesets on the next regeneration. Every changeset carries a MARK_RAN precondition so a database that already has the column records it as run. Two conflicts were not mechanical. OpenCorridorSettlement: develop changed settlement advices from one per beneficiary to one per party bank carrying the full covered list, and this branch had only renamed the accessors for Doobie - upstream's semantics kept, this branch's accessors applied. Glossary: the block boundaries made it look as though both sides had added items, and resolving it as "keep both" duplicated three entries; every conflicting block turned out to be upstream adding items next to shared ones, with this branch contributing nothing, so all five take upstream. 3905 scenarios pass on H2 and on Postgres.
| for (_ <- 1 to 3) { | ||
| metrics.saveMetric("uid", "http://example.com/x", day, 5L, "uname", realApp, | ||
| "dev@example.com", "cid", "getBanks", "1.0", "GET", None, getCorrelationId(), | ||
| "body", "1.2.3.4", "1.2.3.4", "inst", null, null, null) |
…iling it Both defects are in the provenance code this branch wrote while merging develop, not in what develop shipped. DoobieConnectorMethodProvider.create computed the body hash before entering its tryo block. The hash is over decodedMethodBody, which is URLDecoder.decode of a caller-supplied string, and that throws IllegalArgumentException on a malformed escape - '%' is an ordinary character in Scala source, so a connector method containing `100 % 7` is enough. Outside the tryo the exception leaves create uncaught and the request ends as an unhandled 500; the Mapper implementation computed the same hash inside its tryo and returned a Failure the endpoint could report. The two document providers already did it the right way round, so this was the odd one out. ConnectorMethodProvenanceEdgeTest reproduces it: before the fix it fails with "Illegal hex characters in escape (%) pattern". DynamicResourceDoc.update and DynamicMessageDoc.update took updatedByUserId and methodBodyHash with default None while the SET clause assigns them unconditionally, so omitting the arguments did not leave the stored provenance alone - it nulled it. The hash exists to make tampering with a runtime-compiled endpoint detectable, so clearing it silently defeats the feature. The defaults are gone; omitting them is now a compile error, which immediately surfaced three call sites that had been relying on them. 3907 scenarios pass on H2 and on Postgres.
chat_email_digest_state arrived with the develop merge and was not added to resetDatabaseForTestClass, which clears 140 tables including its two neighbours in the same feature, participant and chatroom. The row it holds is "when this user was last emailed a digest", and the scheduler reads it back to decide whether to skip a user - so a row surviving into the next class suppresses a digest that class expects, failing as a function of which suites share the shard's JVM rather than of either suite. No test writes that table yet, so the gap was silent; ChatEmailDigestStateResetTest closes it and fails without the reset. Extending check_changelog_preconditions.py to the new schema changelog turned up something worse than the gap it was meant to close. The guard hard-coded the baseline, so db.changelog-provenance.yaml was outside its scope; adding the file to the list left the count unchanged at 410, because changesets() anchors its split on `- changeSet:` at column 0 and the hand-written changelog nests it two spaces in under databaseChangeLog. The guard reported success over a file it had not read - the failure mode it exists to prevent, in the guard itself. The splitter now accepts either indentation and the count is 415, and addColumn changesets are checked for a columnExists precondition, since tableExists cannot express "this column is already here". Both proven by negative control: the reset test fails on the unmodified ServerSetup, and removing one precondition from the provenance changelog makes the guard name that changeset and exit 1. 3908 scenarios pass on H2 and on Postgres.
…ie stores develop's work here is mostly one theme: a consent user must not accumulate durable roles or own things in its own right. It landed as a guard inside Entitlement.addEntitlement plus explicit per-endpoint checks, and it retired the entitlement `process` column in favour of group_id and created_by_process. It also added mobile-phone fields to ResourceUser, an auth_type column and activity-dashboard indexes to the metrics tables, GET /my/metrics with Top Users and Top Consumers, a serialization namespace on every Redis memoize key, and a resource-doc registry that gives the Berlin Group v1.3 alias a tie-free order. All of that was written against Lift Mapper entities. This branch had already moved the entitlement, resource-user and metrics tables to Doobie, so the resolution carries the behaviour across rather than restoring the entities. The on-behalf-of guard reads createdByConsentId from the Doobie row (an Option, so the null/empty dance is gone) and is otherwise unchanged. The columns develop added by Schemifier come from db.changelog-develop-merge.yaml instead, since ToSchemify.models is empty on this branch and Schemifier creates nothing: a Mapper column here would compile and then not exist. Redis: this branch replaced scalacache with its own memoize layer, so develop's namespace is applied in redisMemoKey rather than through CacheConfig. The two key-format tests move from asserting the sampled envelope as the whole key to asserting it as the suffix - equality against the bare sample would now be asserting the absence of the namespace, which is the opposite of what develop added it for. Three defects were introduced while resolving conflicts and are fixed here rather than left for CI, all three invisible except through a symptom somewhere else: The master changelog gained two `- include:` entries folded into one YAML mapping. Duplicate keys are not an error - the last wins - so db.changelog-provenance.yaml was silently dropped and the tables it creates were never made. It surfaced as `Table "CHAT_EMAIL_DIGEST_STATE" not found` from a DELETE in the per-class test reset, with every shard aborting before it ran a test. No existing check could see it: they each read a changelog on its own and none asked whether master still referenced it. check_changelog_preconditions.py now verifies that every schema changelog is included exactly once, and that the `- include:` and `file:` counts agree. The entitlement INSERT kept `process` in its column list with a value of `""`. In SQL that is a quoted identifier, not an empty string, so the statement never parsed - and addEntitlement wraps the write in tryo, so the grant silently did not happen. 642 scenarios failed with 403 across suites that never mention entitlements. The column is nullable and the field is retired, so it is simply absent from the statement now. MetricQuery collected OBPUserId but not OBPUserIds, so the server-locked user set behind GET /my/metrics was dropped and the endpoint returned every user's rows - a data leak, not just a failing test. It is now rendered as `userid IN (...)`, with an empty set matching nothing rather than removing the clause: no visible users is not the same as no restriction. Each of the three has a test that fails on the defect and names it, rather than leaving the next person to work back from a 403 or a missing table. 4073 scenarios pass on H2 and on Postgres.
develop widens dynamicresourcedoc.examplerequestbody, .successresponsebody and
.errorresponsebodies with MigrationOfDynamicResourceDocBodyFieldsLength, whose own comment says a
response example "routinely exceeds varchar(255)". That migration reads Mapper metadata which does
not exist on this branch and was deleted in the merge along with two others; the other two got
changesets, this one did not, so the three columns stayed at the baseline's VARCHAR(255) and any
body over 255 characters failed the INSERT outright. The endpoints wrap the write, so the caller
saw a generic error rather than a length complaint.
${text.type} is the per-vendor spelling the baseline already uses for methodbody, so the columns
end up where the migration intended: text on Postgres, wide enough not to be a limit on H2.
The precondition is a sqlCheck rather than the columnExists the other changesets use, because the
columns are present either way and the question is their width. It reads
character_maximum_length, which is NULL once the type is unbounded, so a database already widened
by the upstream migration counts zero and marks this run instead of re-applying it.
check_changelog_preconditions.py rejected the changeset until it was taught modifyDataType - by
design, since it refuses to pass a change type it does not know the right precondition for. It now
requires a sqlCheck reading information_schema for these, and says why tableExists/columnExists
cannot serve.
serializationNamespace exists so two builds whose Kryo encodings differ cannot address each
other's entries. It derived its discriminator from scala.util.Properties.versionNumberString,
which reads the STANDARD LIBRARY - and Scala 3 compiles against the 2.13 one, so it answered
"2.13" here too. This branch therefore produced byte-identical keys to develop, on exactly the
upgrade the namespace was added to protect: measured as the prefix "obpser1-scala2.13" in this
branch's own golden-key test output, on a build whose scala.compiler is 3.3.8.
The failure that follows is the one already documented above the value: an entry written by one
chill/Scala combination decodes under the other into a different collection type, the decode
succeeds, and the call site whose signature says List gets a ClassCastException - a 500 for the
whole TTL, since a read that throws does not evict.
The compiler generation is not in any version string the runtime exposes, so it is a class probe:
scala.runtime.Scala3RunTime ships in scala3-library and does not exist in scala-library 2.13. Both
halves are kept ("3-lib2.13"), because the encoding depends on the compiler that produced the
classes and on the library they were compiled against. A 2.13 build keeps develop's spelling
exactly, so only this side moves and no one else cold-starts a cache.
The test probes with a different Scala-3-only class (scala.runtime.LazyVals$) than the
implementation uses: repeating the production probe would make the test agree with it however
wrong both were.
`CurrentNamespace should include("3")` was meant to say the namespace names the compiler
generation. It says nothing: "obpser1-scala2.13" contains a '3' as well, so the assertion held in
precisely the state the test exists to reject. The real work was being done by the line above it,
and this one only added false confidence. It now looks for "scala3".
Two comments corrected alongside it, both wrong in ways a reader would act on:
Redis.scala - the block explaining what the namespace is for had a second doc comment placed
between it and `serializationNamespace`, so it documented nothing and the value it explains was
left bare. The probe moves above it.
db.changelog-develop-merge.yaml - the precondition's comment said character_maximum_length is NULL
for an unbounded type. That is Postgres and MySQL; H2 reports 1000000000 and SQL Server -1. The
changeset is correct either way because it counts columns still at exactly 255, which is what the
comment now says.
getDistinctParentIds and getParentIdWithAttributes were written for AttributeQueryTrait.getParentIdByParams and NewAttributeQueryTrait.getParentIdByParams. Both traits are dead code with zero mixers anywhere in the tree, removed in the next commit as part of the net.liftweb.mapper cleanup - and neither ever called into these two methods either, so their removal leaves this file's only remaining function, getDistinctProviders, unaffected. Found while auditing the mapper cleanup's blast radius: the doc comments on both methods asserted a caller that had not existed since the traits were deleted, which is worse than no comment at all.
First step of unbundling lift-persistence: the fork's mapper package is 46.5% of its
lines, has no upstream Scala 3 port (Lift itself deleted persistence rather than
porting it), and OBP-API has had zero live Mapper entities since the Doobie migration
completed (ToSchemify.models = Nil). This removes the last obp-api references to it,
without touching the dependency itself - obp-api still pulls in lift-persistence for
common/util/db, same as before.
Deleted outright (all confirmed zero external references, not just zero imports):
AttributeQueryTrait/NewAttributeQueryTrait (self: BaseMetaMapper, no mixers anywhere),
CommonFunctions (validUri/validUrl, zero call sites), MappedAccountNumber/
DefaultStringField/MappedUUID/UUIDString (MappedString subclasses with no entity left
to use them), and MappedClassNameTest - which asserted over classOf[Mapper[_]]
subtypes, a set that has been permanently empty since the last entity was moved to
Doobie. It is the same "assertion that could not fail" shape as the CacheKeyFormatTest
fix earlier on this branch.
Two deletions needed care because nothing importing net.liftweb.mapper pointed at
them - they are reachable only through a class-name string, so removing the jar
without removing these would compile clean and then fail at runtime:
- JsonSerializers.MapperSerializer: ReflectUtils.forType("net.liftweb.mapper.Mapper")
inside an eager val, wired into the json4s Formats chain. Deleting the object
without also dropping it from the `serializers ::` list would leave a reference
to a name that no longer exists.
- ClassScanUtils.getMappers: Class.forName("net.liftweb.mapper.LongKeyedMapper")
inside a try/catch that logs and returns Nil on any Exception - the failure mode
a `net.liftweb.mapper`-string grep cannot see and a deleted jar would hit silently.
Zero callers, confirmed before deletion.
LocalMappedConnectorDataImport.MappedSaveable (zero instantiations) is deleted the
same way, with the historical comments at its three call-alike sites updated to say
"the now-removed MappedSaveable" rather than describing a type that no longer exists.
Same treatment for two comments in DoobieQueries.scala that credited
AttributeQueryTrait/NewAttributeQueryTrait as callers of getDistinctParentIds/
getParentIdWithAttributes - untrue even before this commit, since those two methods
already had zero callers (deleted separately, previous commit).
Remaining touches are narrowing, not removal: 18 migration scripts had a dead `DB`
import alongside the `Schemifier` one they actually use (`Schemifier.infoF` as a
logging callback - handled in the next commit), and three files had a dead
`import code.util.{MappedUUID, UUIDString}` left over from before those types moved to
Doobie-native construction.
Verification: mvn -Pprod -DskipTests clean install clean on first pass (deletions are
self-checking - the compiler is the reachability proof). H2 Surefire audit: 4073/0/0
(4075 - the 2 MappedClassNameTest scenarios, the only test-file change here).
Postgres was flakier to pin down and worth recording. Three concurrent 4-shard runs
and one 6-shard run all failed, but never on a real assertion:
- Shard 2 hit `run_tests_parallel.sh`'s 1200s per-shard timeout in every attempt,
once at 19m51s - a hair under the cap. The JVM's own shutdown hooks fired cleanly
mid-scenario each time, with zero exceptions, zero OOM/jetsam events, zero
Postgres connection errors in any log. This machine had a second worktree's
orphaned scalatest fork alive for >40h during every attempt (a stray
forkMode=once JVM this repo's own comments already document as a known
reparenting hazard) plus this session's own earlier background work, pushing
load average past 7 - not something to kill blindly (not owned by this session),
so shard 2's package set was instead run standalone (own Postgres database, own
ports, 1800s budget, no sibling shards competing for CPU): 1197 succeeded, 0
failed, 0 exceptions.
- Shard 3 failed once, on ResourceDocsTest's v4.0.0 scenarios, with a
scala.xml.XML.loadString error on a literal "<random-string>" placeholder inside
an existing (untouched by this commit) v4.0.0 endpoint description -
`resource_docs.take(3).foreach(doc => stringToNodeSeq(doc.description))` only
XML-validates the first three docs returned, so whether this fires depends on
resource-doc ordering, not on anything this commit changed. The suite passed
standalone (63/63) and again as part of shard 3's full package set run the same
isolated way as shard 2: 1008 succeeded, 0 failed.
Both isolated runs together cover every package the 4-shard split runs; shards 1 and 4
were clean across all three concurrent attempts. That is full Postgres coverage, green,
just not all four shards inside one concurrent invocation this particular machine could
sustain today.
…local ones
Second step of unbundling lift-persistence, following the mapper-surface deletions in
the previous commit. Four symbols were still genuinely called (not just imported) from
obp-api, none of them mapper-specific in behaviour - Schemifier's logging callback and
schema-name lookup both operate purely on net.liftweb.db types, and DB/
DefaultConnectionIdentifier under the mapper package are forwarders to the db/util
originals, not distinct implementations. Decompiled the shipped jar (javap) to copy
each one exactly rather than guess:
Schemifier.infoF(msg: => AnyRef): Unit = logger.info(msg) - unwrapped, verbatim
Schemifier.getDefaultSchemaName(conn: SuperConnection): String =
conn.schemaName.or(conn.driverType.defaultSchemaName).or(DB.globalDefaultSchemaName)
.openOr(conn.getMetaData.getUserName) - unwrapped, verbatim
Both now live on Migration.DbFunction, next to the tableExistsByName/
makeBackUpOfTableByName helpers that already carried the "copied from
net.liftweb.mapper.Schemifier" comment for the same reason. 62 call sites across 41
migration scripts and StoredProcedureUtils.scala move from `Schemifier.infoF _` to
`DbFunction.infoF _` - a mechanical substitution, verified uniform first: every one of
those 41 files used Schemifier for infoF and nothing else, and every one already
imported DbFunction unqualified for other Migration helpers, so the now-dead
`import net.liftweb.mapper.Schemifier` line comes out alongside each substitution.
`net.liftweb.mapper.DB` becomes `net.liftweb.db.DB` in Migration.scala (11 call sites)
and `net.liftweb.mapper.DefaultConnectionIdentifier` becomes
`net.liftweb.util.DefaultConnectionIdentifier` in DBUtil.scala - both confirmed
identical singletons by decompiling: `mapper.DB` is `object DB extends db.DB1`, and
`mapper.DefaultConnectionIdentifier` is a one-line forwarder to `util.DefaultConnectionIdentifier`.
Migration.DbFunction.tableExists(BaseMetaMapper, ...) and makeBackUpOfTable(BaseMetaMapper)
are deleted outright: both were the last two consumers of BaseMetaMapper, both had zero
callers (confirmed by grep before deletion - the only remaining hits are doc comments in
other migration scripts that already say the entity behind them is gone), and both have
had *ByName successors in active use for a while.
Two call sites intentionally untouched: Boot.scala:540 and
MockedRabbitMqAdapter.scala:3322 still call Schemifier.schemify(true, Schemifier.infoF _,
ToSchemify.models: _*) on an empty list - a no-op, but the whole call and its
ToSchemify.models plumbing come out in the next commit along with Boot's remaining
Schemifier-adjacent setup, rather than half-migrating a call this commit does not also
delete.
net.liftweb.mapper now has zero live references from obp-api (grep -rn
"net\.liftweb\.mapper" obp-api/src/main | grep -v '^\s*//' turns up only the two
Boot.scala/MockedRabbitMqAdapter.scala schemify calls and pre-existing commented-out
Lift-era files this refactor does not touch).
Verification: clean compile in one pass. H2 Surefire audit: 4073/0/0, unchanged from
the previous commit (no test files touched here).
Postgres: given the previous commit's documented machine-load flakiness on concurrent
shards, went straight to isolating each of the 4-shard split's package sets against its
own database and ports rather than re-running the concurrent layout first - two pairs
run concurrently (shard 1 with shard 4, then shard 2 with shard 3) for a bounded total
runtime without reintroducing the contention that caused the earlier timeouts. All four
green: shard 1 535/0, shard 2 1197/0, shard 3 1008/0, shard 4 1285/0 - full coverage of
the 4-shard layout, all BUILD SUCCESS, zero FAILED markers anywhere.
Third and final step of removing net.liftweb.mapper from obp-api. The previous two
commits took every reference down to two schemify calls, both already no-ops
(Schemifier.schemify(true, Schemifier.infoF _, ToSchemify.models: _*) on an empty
list), plus a MapperRules setting and a MetaMapper-typed field that fed them. All
four come out here, along with the last wildcard mapper import.
- Boot.scala:173's MapperRules.createForeignKeys_? assignment: the only reader was
Schemifier, and Schemifier's argument was always Nil, so this configured a
foreign-key policy for a schema-creation pass that never created anything. The
mapper_rules.create_foreign_keys prop it read is retired (release_notes.md, both
props templates).
- Boot.scala:539's schemifyAll(), renamed createDefaultChatRoom() with the
Schemifier.schemify line removed - it kept exactly one live side effect
(getOrCreateDefaultRoom()) and the name should say so, not describe schema work
that stopped happening once ToSchemify.models went to Nil.
- MockedRabbitMqAdapter.scala:3322's identical schemify call, and its now-dead
net.liftweb.mapper.Schemifier / bootstrap.liftweb.ToSchemify imports.
- ToSchemify.models itself: not just emptied, deleted. The object stays (it also
starts the optional gRPC server and registers a JVM shutdown hook, unrelated to
schema). Its four remaining "importers" - ServerSetup, LocalMappedConnectorTestSetup,
TestConnectorSetupWithStandardPermissions, SandboxDataLoadingTest - never actually
read the field; each import was dead weight left over from when their reset loops
iterated it. Removing them is confirmed safe by the same evidence that made the
field safe to delete: obp-api has had zero live Mapper entities since the Doobie
migration finished.
- Boot.scala:64's `import net.liftweb.mapper.{DefaultConnectionIdentifier => _, _}` -
the wildcard that supplied MapperRules, Schemifier and MetaMapper to this file.
Nothing else in it needed anything from that package.
LiquibaseSchemaSetupTest asserted `ToSchemify.models shouldBe empty` as half of pinning
"liquibase.enabled defaults to true because nothing else creates a table." That
assertion doesn't compile once the field is gone, and doesn't need to: the invariant it
protected is now enforced by the compiler rather than by a runtime check, since there
is no Schemifier.schemify call left anywhere in obp-api to accidentally un-empty a list
that no longer exists. Rewrote the test and the doc comments in LiquibaseSchemaSetup.scala
and LiquibaseOnExistingSchemaTest.scala that described the old mechanism, so none of them
keep pointing at a symbol that isn't there.
One more comment turned out to be stale independently of this refactor, caught only
because it was about to become more obviously wrong: AtmTableResetIsolationTest.scala's
doc comment said MappedAtm was "still in Boot.ToSchemify.models" and reset "happens for
free" via that list's bulkDelete_!! loop - checked, and all four reset paths it lists
already carry an explicit `DELETE FROM mappedatm` (ServerSetup:150 and the same line
number pattern in the other three). MappedAtm moved to Doobie a while ago; the comment
was never updated to say so. Corrected to describe the current mechanism instead of a
superseded one.
obp-api/pom.xml's comment on the lift-persistence dependency said Scala 3 doesn't exist
"see docs/scala3-lift-mapper-blocker.md" as if obp-api's own code were still blocked by
it. It isn't, any more - grep -rn "net\.liftweb\.mapper" across obp-api and obp-commons
main sources now turns up only comments and the pre-existing entirely-commented-out
Lift-era files this refactor doesn't touch. What is still pinned to _2.13 is the
ARTIFACT: lift-persistence bundles common+db+mapper+proto+util as one jar, and no
Scala 3 build of the bundle exists because mapper can't compile under Scala 3. Reworded
to say that rather than implying obp-api's own mapper usage is the blocker.
Verification: clean compile in one pass. H2 Surefire audit: 4073/0/0, unchanged (the
4 dead-import deletions and the LiquibaseSchemaSetupTest rewrite add or remove no
scenarios). Postgres: same isolated-per-shard-pair strategy as the previous commit,
same numbers - shard 1 535/0, shard 2 1197/0, shard 3 1008/0, shard 4 1285/0, all
BUILD SUCCESS, zero FAILED anywhere.
Also did the one check the test suite cannot: a real production-mode boot
(flushall_build_and_run.sh, backed by an isolated in-memory H2 rather than any
suite's shared setup) reached `Ember-Server service bound to address: 127.0.0.1:8080`
with no ExceptionInInitializerError and no Schemifier line anywhere in the log, then
served two live requests against it - GET /obp/v5.1.0/root (200) and
GET /obp/v5.1.0/resource-docs/v5.1.0/obp (200, 3.5MB, 599 resource_docs entries) - the
second one specifically to drive the json4s Formats chain end to end now that
MapperSerializer is gone from it (removed two commits ago), on a real multi-megabyte
payload rather than a test fixture.
End state: grep -rn "net\.liftweb\.mapper" obp-api/src obp-commons/src, filtered to
non-comment lines, returns nothing. obp-api's dependency on net.liftweb.mapper is zero.
…ook never ran
Found by code review of the previous three commits (the net.liftweb.mapper removal):
deleting ToSchemify.models and the Schemifier.schemify(...) call that read it removed the
only thing in the whole codebase that ever touched the ToSchemify object. Scala objects
initialize their entire body - every val and top-level statement - on first access to any
member, not at class-load time. Before this session's earlier commits, Boot.scala's
schemifyAll() reading ToSchemify.models was that first access; once schemifyAll() was
renamed to createDefaultChatRoom() and stopped touching ToSchemify, and models itself was
deleted, nothing else in the tree ever referenced it again (confirmed by grep - zero live
code hits, only comments).
The object's body is not just schema-adjacent bookkeeping: it starts the optional gRPC
server (grpc.server.enabled) and registers the JVM's one ORDERED shutdown hook - added,
per its own comment, specifically to fix a race between two previously-concurrent hooks
(gRPC could still be serving a request while the DB pool closed underneath it). With the
object never initializing, both silently stop happening: grpc.server.enabled=true starts
no server and logs no error, and - regardless of that flag - the app stops gracefully
closing the Hikari pool and Redis on shutdown at all.
Verified live, not just by reading the bytecode-initialization rule: booted the packaged
jar and sent it SIGTERM. Before this fix, no HikariPool shutdown log line appeared at all.
After renaming the object to ProcessLifecycle and adding an explicit
ProcessLifecycle.start() call in Boot.boot() (with a comment explaining why an explicit
call is required rather than relying on incidental access), the same test produces
"HikariPool-1 - Shutdown initiated..." / "Shutdown completed." from the shutdown-hook
thread.
Also renamed for the same reason the earlier commits already applied to schemifyAll(): the
object's name described work it no longer does (nothing about it is "to schemify" any
more - the schema half left when models did), and that mismatch is very likely part of why
nothing noticed it had gone silently unreachable.
Swept the doc comments the same review flagged as referencing renamed/deleted symbols by
name, in the files most likely to be read while debugging boot order or writing a new
migration:
- Boot.scala: the comment above the executeScripts calls still said "AFTER schemifyAll()
above", read right next to the createDefaultChatRoom() call it was talking about.
- Migration.scala's `database` object doc comment named `schemifyAll()` and
`tableExists(ResourceUser)` - the latter is the exact Mapper-typed overload the
previous commit deleted; a reader copying that comment's example would write code that
no longer compiles.
- Two migration scripts' historical comments (MigrationOfConsentAuthContextDropIndex,
MigrationOfMappedUserAuthContext) named the same deleted overload as "what this used to
call" without saying the overload itself is gone, not just unused.
Verification: clean compile. H2 Surefire audit: 4073/0/0, unchanged (no test files
touched). Postgres: same isolated-per-shard-pair strategy as the prior three commits -
shard 1 535/0, shard 2 1197/0, shard 3 1008/0, shard 4 1285/0, all BUILD SUCCESS, zero
FAILED anywhere.
Second of two cleanups from the code review of the net.liftweb.mapper removal.
DbFunction.maybeWrite took a `logFunc: (=> AnyRef) => Unit` because Schemifier did - it
was a general-purpose library where a caller might reasonably want its own logger. Here it
never was: all 64 call sites across the 41 migration scripts and StoredProcedureUtils
passed the identical `DbFunction.infoF _`, and infoF was never anything but
`logger.info(msg)`. Checked before removing: 64 occurrences of the call, 64 of them that
exact shape, and infoF had no other reader.
A parameter with zero call-site variance is not an abstraction, so `logger.info(ct)` moves
inline and infoF goes. This also finishes what the previous commits started - they already
deleted the Mapper-typed overloads (tableExists, makeBackUpOfTable) once each had a single
call shape, and leaving this one pluggable was inconsistent with that.
Also corrects an overstatement the same review flagged in LiquibaseSchemaSetupTest: the
comment replacing the retired `ToSchemify.models shouldBe empty` assertion claimed the
invariant is now "enforced by the compiler". That holds only for the exact regression it
replaced - repopulating a field that no longer exists. It does not hold for the wider claim
the surrounding doc makes ("nothing else creates a table"): lift-persistence still ships
net.liftweb.mapper, so a new Mapper entity plus a fresh Schemifier.schemify call would
compile and run, and no test in the suite boots Boot.scala to notice one running beside
Liquibase. The comment now says what is actually guaranteed and what is not.
Verification: clean compile. H2 Surefire audit 4073/0/0, unchanged. Postgres, isolated per
shard pair as before: shard 1 535/0, shard 2 1197/0, shard 3 1008/0, shard 4 1285/0, all
BUILD SUCCESS.
recordConnectorTrace called APIUtil.getCorrelationId() to fill the correlationId column, but that function is a stub returning "" since the Lift teardown (it used to read Lift's container session). The matching connectormetric row is written with the correlation id routeToConnector already extracted from the CallContext for exactly this purpose - trace rows just never received it, so with write_connector_trace enabled every row was written with correlationid = '' and could not be looked up by OBPCorrelationId or joined to its metric row. Pass the already-extracted correlationId into recordConnectorTrace instead of re-deriving it.
scopeFor keyed the dedup lock/response cache on (consumer or Authorization
header, operation id) alone. Two gaps:
- No user in the scope. One Consumer (API Explorer, the Portal, a bank's
mobile app) serves many users, so a second user reusing a key the first
user had already used against the same operation was served the first
user's cached response - their own request never ran.
- No concrete path in the scope. The operation id is the ResourceDoc
template ("OBPv5.1.0-deleteAtm"), never substituted with the real
BANK_ID/ATM_ID. Without the path, one key covered every resource under
an operation - deleting atm-1 then atm-2 under the same key deleted only
the first and replayed its 204 for the second, since a DELETE's body
hash is sha256("") for both requests and gave no other discriminator.
Add both to the scope key. Also add guaranteeCase to the lock acquired in
runAndCache: it was released only on the two normal completion paths, but
ResourceDocMiddleware wraps every endpoint in a timeout that CANCELS the
fiber holding the lock, so a slow POST answered 504 left the lock held for
its full 60s TTL and told the client's well-behaved retry "operation
already in flight" when nothing was.
IdempotencyMiddlewareTest's "in flight" scenario hard-codes the scope hash
the middleware computes; updated it to the new four-part formula.
TokenBinding.verifyTokenBinding compared a bound access token's cnf.x5t#S256 claim against whatever PeerTrust resolved as the caller's certificate, without checking how that certificate was resolved. On the out-of-the-box configuration (mtls.enabled unset, no trusted proxies), PeerTrust.trustForwardedHeaderWithoutTls defaults to true, so an unauthenticated PSD2-CERT header is enough to name "the caller" - by design, for endpoints that only need some certificate to attribute a request to. RFC 8705 sender-constraining needs more than that: a certificate is public information (a QWAC is not a secret), so an attacker who replays a stolen bound access token alongside the victim's own public certificate in that header would pass ENFORCE/REQUIRED verification even though nothing proved they hold the matching private key. Add PeerTrust.UnauthenticatedHopDetail as the named marker for that one resolution (already used internally, just not exposed for a caller to check), and have TokenBinding treat it as equivalent to no certificate at all via a new callerCertificateForBinding - reading cc.certificateTrust / certificateTrustDetail rather than re-deriving anything from the raw header, so it can never disagree with what PeerTrust actually decided.
allStaticResourceDocs deduplicated the union with distinctBy(_.operationId), which keeps the FIRST occurrence in iteration order. Every other consumer of this ordering (Http4s600's top-apis/popular-apis, JSONFactory6.0.0's metrics, and this registry's own sortKey docstring) is built on the opposite convention: the standard sorted LATER wins a name it shares with one sorted earlier, via a `.toMap` where the last entry wins. This silently broke the Berlin Group v1.3 alias safeguard sortKey already implements. The alias re-stamps the canonical BG v1.3 docs with implementedInApiVersion.copy(apiStandard = doc.implementedInApiVersion.apiStandard), so with the natural configuration (berlin_group_v1_3_alias_path ending in "v1.3") its operation ids are byte-identical to the canonical ones, and sortKey ranks the alias first specifically so the union's last-wins dedup keeps the canonical entry. distinctBy's first-wins direction handed the win to the alias instead, replacing all 55 canonical BG v1.3 docs with copies whose URL prefix is the alias path. Switch to reverse/distinctBy/reverse: same last-wins direction as every other consumer, while preserving the relative order of what survives.
1. The mappedconsent join had no guard against the empty-string sentinel. opt() in MappedConsent.scala stores an empty consent_reference_id as '' rather than NULL, and every non-consent metric row also defaults to consent_reference_id = ''. An unguarded `ON m.consent_reference_id = c.consent_reference_id` therefore joined ALL non-consent rows to a single legacy/blanked consent row whenever one existed, and COALESCE(c.muserid, ...) attributed the estate's entire non-consent traffic to that one unrelated user. Add `AND m.consent_reference_id <> ''` to the join condition in buildAggregateMetricsQuery and buildTopUsersQuery - the same fix the NULLIF(..., '') calls already apply on the read side. 2. buildFilterConditions' user_id filter always bound to the raw metric.userid column, even in the two queries whose SELECT/GROUP BY attributes a consent-borne call to the granting human via COALESCE(c.muserid, m.userid). Filtering by a human's user_id excluded exactly the consent-borne calls the endpoint claims to attribute to them (their metric.userid is the consent's own shadow user), while filtering by the shadow user's id returned rows displayed under a different (the human's) identity. Add a resolvedUserIdExpr parameter, defaulting to the previous behaviour for callers with no consent resolution in play, and pass the COALESCE expression from buildAggregateMetricsQuery and buildTopUsersQuery.
Two independent gaps in updateMyMobilePhoneNumber and the mobile number on POST /users: - The regex character class is a union, not a required sequence, so " " (five spaces), "((.))" and "-.-.-" all matched despite the ResourceDoc promising "5 to 50 digits, spaces, dashes, dots or parentheses". A digit-free string would be stored as the user's mobile number with nothing for the later validation/SMS flow to send to. Require at least five actual digits alongside the shape check. - updateMyMobilePhoneNumber wrote straight to the authenticated principal with no check for a consent user. Under a Consent, cc.user is the consent's own shadow ResourceUser by default; letting that identity overwrite the mobile number - an authentication channel used for validation codes and SMS OTP - would let an agent repoint the granting human's second factor. Refuse it outright rather than silently redirecting to the resolved human, since a silent redirect here would let the agent change a security-relevant field the caller has no reason to believe they don't have permission to change.
getTopUsers and getTopConsumers called createQueriesByHttpParamsFuture directly on the raw request params instead of going through APIMetrics.applyMetricsFromDateDefault the way every other metrics-reading endpoint does. With no from_date, APIUtil.getFromDate substitutes the epoch, which makes MappedMetrics.determineMetricsCacheTTL classify the query as "only stable data" and pick the 24-hour TTL - so the default, no-parameter call an operator dashboard would make froze for a day while traffic kept arriving, and the first miss of that day scanned the whole metric table since 1970. Corrected the two ResourceDoc descriptions to match (they claimed "defaults to one year ago" / "the current date", which was never the actual range).
createAccountJSON's Links.Self does list.head.AccountId unconditionally. getAccount builds that list by filtering the caller's own private accounts down to the requested accountId, which is legitimately empty for an id that does not exist (or belongs to someone else) - the same shape a real TPP integration hits on a typo or a stale id. That empty list reached list.head and threw NoSuchElementException, answering 500 instead of the 404 UK Open Banking's spec calls for. Found by extending the endpoint auth/crash sweep to cover Berlin Group and UK Open Banking (previously OBP-standard only) - FailureSweepTest calls every endpoint with a nonexistent id and asserts none of them 5xx.
…n Banking
EndpointCatalog.all was Http4s700.allResourceDocs - the OBP-standard
aggregation only. AuthSweepTest, SuccessSweepTest and FailureSweepTest all
read their coverage from it, so every Berlin Group and UK Open Banking
endpoint was silently outside the anonymous-401/crash sweep: a doc in
those standards missing AuthenticatedUserIsRequired with empty roles would
let anonymous callers reach account data and nothing would catch it.
Switching to ResourceDocRegistry.allStaticResourceDocs (the same
cross-standard union APIUtil.getAllResourceDocs already exposes) needed
three follow-on fixes, all specific to a catalog that now spans multiple
independent route trees rather than one:
- EndpointCatalog.concretePath hard-coded "/obp/" + apiShortVersion.
Berlin Group and UK Open Banking routes match on Root / urlPrefix /
apiShortVersion with no "/obp" segment at all (see e.g.
Http4sBGv13AIS.bgV13Prefix) - urlPrefix is "obp" for the OBP standard by
construction (ApiVersion.setUrlPrefix patches it to the configured
apiPathZero at boot), so using implementedInApiVersion.urlPrefix
uniformly reproduces the old OBP behaviour while giving BG/UK their own
real prefix instead of a path that 404s before reaching any route.
- AuthSweepTest.messageOf only read the top-level "message" field. Berlin
Group requests get a PSD2-mandated {"tppMessages": [{"text": ...}]}
envelope instead (ErrorResponseConverter.toBgErrorBody) - the endpoint
was correctly answering 401, the sweep just could not see the message
text to compare it against. Fall back to tppMessages[0].text, which
carries the identical string the OBP envelope would have.
- SweepCoverageTest's "deduplicated by (url, verb)" check assumed one
route shape maps to one operation, true for OBP but not for Berlin
Group: several SCA sub-steps (e.g. updatePsuAuthentication /
selectPsuAuthenticationMethod / transactionAuthorisation) legitimately
share one URL and verb, disambiguated by request body rather than path.
Replaced with a check on operationId uniqueness, which is the union's
actual by-construction guarantee and still catches a genuine duplicate
(e.g. two ResourceDoc objects registered under the same operation id).
Two categories of endpoint answer non-2xx to SuccessSweepTest's
fully-entitled-but-consentless caller and are documented in
expectedNon2xx rather than treated as failures: Berlin Group AIS and UK
Open Banking account-read endpoints both require an established,
standard-tagged consent regardless of role, which the sweep's generic
fixture (grants every role, creates no consent) does not provide. The
403 in both cases is the endpoint correctly refusing, not a defect - the
anonymous case is what AuthSweepTest already covers independently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.




Migrates OBP-API to Scala 3, replacing the two frameworks that blocked it. 287 commits on top of
3df73fe11; 1027 files, +66009/-33154.Opened for review of the whole line. It does not merge cleanly onto
developyet — the branchis 26 commits behind and conflicts in 12 files, all of them where
developextended a Lift Mapperentity this branch had already moved to Doobie. That merge is in progress separately and will be
pushed here; the conflicts are listed at the end so reviewers know what is coming rather than
discovering it from a red merge box.
What changed
Scala 3. Lift Mapper cannot compile under Scala 3 — the compiler crashes on the
object X extends class Xshape every Mapper entity uses (reduced to a 5-line case). Both of itsconsumers had to go first.
Lift Web → http4s, complete.
net.liftweb.httpno longer appears in any.scalasource. Thereis no Lift fallback in the request chain: an unmatched
/obp/*path returns a JSON 404 fromnotFoundCatchAll. API versions are unchanged by this — a framework migration happens in placeinside the existing version file, and a version bump still means a changed API signature.
Lift Mapper → Doobie, complete.
ToSchemify.modelsis nowNil; Schemifier creates nothing.Flyway → Liquibase. Flyway needed one hand-written script set per vendor: 118 for H2, 118 for
Postgres, and nothing for the three other drivers its
vendorFolderwould have booted againstsilently, with no tables. One changelog now describes each change once and Liquibase emits the
dialect. The baseline is generated from a Postgres database the Flyway scripts built, not
hand-written, and is regenerated with
scripts/GenerateChangelog.javaplus a normaliser ratherthan edited.
Defects found and fixed along the way
Migrating the data layer surfaced behaviour that lived in Mapper's field types rather than in the
entities, and which a column-by-column port drops silently:
MappedBooleanreads back asfalsewhateverdefaultValuedeclares, a NULLMappedLongas the declared default — read as a hardcoded-1, six call-limit columns turned"the configured limit" into "no limit", and that value is what the rate limiter enforces from
MappedEmaillowercases and trims on every set and validates on save;authuserlost all of itMappedMetrics, reachable from read-only roles, giving a boolean-blindoracle over the whole database
Security fixes on top: a locked account could still authenticate through OIDC and Keycloak (both
read
v_oidc_usersover JDBC and never callverify-credentials);consents.sca.enabled=falseaccepted any SCA answer in production; the dynamic-code dependency validation inspected an empty
list because its scan was gated on an unrelated diagnostic prop; and
compileScalaCodenow refusesto run when no SecurityManager can be installed (JEP 486) unless the operator says so explicitly.
Testing
3876 scenarios, 0 failures, on both H2 and Postgres. Postgres is not optional here: H2 tolerates
things Postgres does not, and the schema is generated from the changelog at boot on both.
Conflicts with current develop
developaddedcreatedByUserId/updatedByUserId/ a method-body hash to DynamicResourceDoc,DynamicMessageDoc and ConnectorMethod, and one new Mapper entity (
ChatEmailDigestState). Thisbranch had already moved the first three to Doobie, so the resolution ports the new fields into the
Doobie stores and the changelog rather than restoring the Mapper entities;
ChatEmailDigestStateneeds the same treatment, since with
models = Nilits table would otherwise never be created.