feat(simulate): store and show sub-actors and their turns by KarthikAvinashFI · Pull Request #3539 · future-agi/future-agi · GitHub
Skip to content

feat(simulate): store and show sub-actors and their turns - #3539

Draft
KarthikAvinashFI wants to merge 22 commits into
devfrom
feat/TH-8651-sub-actors
Draft

KarthikAvinashFI wants to merge 22 commits into
devfrom
feat/TH-8651-sub-actors

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

1. Why

The harness can now give an environment a cast of sub-actors: people on the caller's side who are neither the agent nor the person it serves, such as a voice near the person, someone who takes the phone, the caller's own phone menu or a second chat participant (companion PR below). The platform has to switch that on per deployment, keep what the harness produces, and show it: which sub-actors a world has, which scenarios bind them, and who said what in a call.

2. What changed

File Change Preserved
futureagi/tfc/settings/settings.py ALK_SUB_ACTORS (default off)
futureagi/simulate/services/hosted_harness_gateway.py Exports HARNESS_SUB_ACTORS=1 to the build, run and builder chat sandboxes when the setting is on; reads cast.json from the live sandbox, the authoring archive and the bundle, and returns it as a cast stage output ({"primary": {...}, "sub_actors": [...]}) with a summary that counts sub-actors correctly Every existing stage output unchanged; no cast output when the environment has none
futureagi/simulate/models/hosted_harness.py, migration 0098_sub_actors HostedHarnessScenario.sub_actors (JSON, nullable): the scenario's bindings as [{"id": ..., "trigger": {...}}] Scenarios indexed before stay null
futureagi/simulate/services/harness_scenarios.py Indexing fills sub_actors; rows return it; a sub_actors filter (category conditions) by id or none (the person alone), offered only when some scenario binds a sub-actor Every existing field, filter and grouping unchanged
futureagi/simulate/models/test_execution.py, migration 0098_sub_actors CallTranscript.SpeakerRole.SUB_ACTOR and CallTranscript.speaker_name (blank by default) Existing roles and rows unchanged
futureagi/simulate/models/chat_message.py, pydantic_schemas/chat.py, migration 0098_sub_actors sub_actor chat role
futureagi/simulate/services/hosted_harness_ingestion.py, alk_simulate_ingestion.py A sub_actor transcript turn is stored with its role and name, and a chat sub-actor message is its own row; both writers keep a speaker name only on sub_actor turns Agent and customer metrics still read only user and assistant, so sub-actor turns stay out of them
futureagi/simulate/serializers/alk_simulate_ingestion.py, serializers/test_execution.py speaker_name accepted on ingestion and returned on transcripts
futureagi/simulate/tasks/alk_sim.py, utils/chat_simulation.py The transcripts built for CSAT and other call-level scoring label a sub-actor turn with its name and "(not the customer)" instead of folding it into the agent's or the customer's lines Transcripts without sub-actors are unchanged
futureagi/simulate/services/harness_environment.py, run_results_v3.py Each sub-goal result column carries actor: the sub-actor the sub-goal is about, or primary for the person served; the results cache key is bumped so cached columns pick it up Every existing column field unchanged
futureagi/simulate/views/harness_environment.py, serializers/harness_environment.py The environments list response carries sub_actors: whether the setting is on, so the frontend gates template casts on the same switch Existing list fields unchanged
frontend/src/api/simulate-environments/environment.js, scenarios.js, runDetail.js Cast mapped from the stage output (with each sub-actor's wants and origin); a scenario's subActors; a turn's speakerName
frontend/src/components/VoiceDetailDrawerV2/TranscriptView.jsx, transcriptUtils.js A sub-actor turn is labelled with its name and "not the customer" Every other turn renders as before
frontend/src/sections/simulate/environments/workspace/... Contract tab: the existing actors panel, until now parked for want of a real cast, shows a built environment's cast read-only with how many scenarios bring each in (the fixture library is never shown as if it were real); scenarios table: the sub-actors' names (from the cast) under a scenario's name; chat transcript: the sub-actor's name Nothing new renders for an environment without a cast
frontend/src/api/simulate-environments/runCalls.js, workspace/runs/detail/trace/RunTraceTable.jsx An "Actor" filter on a run's results narrows the sub-goal result columns, and so their pass rates, to the person served or one sub-actor; shown only when a sub-goal is about a sub-actor Runs without sub-actor sub-goals show no new filter
frontend/src/api/simulate-environments/_fixtures/actors.js, _fixtures/prebuiltEnvironments.js, environments.js, cards/TemplateBuildPanel.jsx Each prebuilt template carries a generic cast for its vertical, in the same shape as a built environment's cast and limited to channels its surface can deliver; adopting a template brings the cast only when the setting is on, and Build waits until the list has reported the setting Templates on surfaces the harness does not reach carry no cast; with the setting off nothing changes
api_contracts/openapi/swagger.json, frontend/src/api/contracts/*.generated.js, frontend/src/generated/api-contracts/* Regenerated for the new role, fields and the list's sub_actors; the MCP tool manifest (futureagi/mcp_server/catalog/tools.generated.json) regenerated to match Additive only

3. Behavior callouts

  • Off by default: with ALK_SUB_ACTORS unset the sandboxes never receive HARNESS_SUB_ACTORS, so builds and runs are unchanged, and adopted templates bring no cast.
  • Migration: 0098_sub_actors (scenario bindings, transcript role and speaker name, chat role). It is additive; no data is rewritten.
  • "Not applicable" is stored as held. A sub-goal about a sub-actor that did not enter arrives from the harness as held=True with a "Not applicable" reason, so it counts as held in pass rates. There is no separate not-applicable verdict.
  • The prebuilt templates are a frontend catalogue today: the templates and adopt endpoints are still mocks in prebuilt.js and environments.js, so the per-vertical casts live there and follow the list response's sub_actors switch. When the templates endpoint lands it should return the same cast shape.
  • Transcript rows may carry speaker_role = "sub_actor" with a speaker_name; rows written before this PR have speaker_name = "". Readers that switch on user / assistant see the new value only for calls that bound a sub-actor.
  • The harness registers channels for computer-use, browser, coding and multi-agent targets but refuses them until runners for those agent kinds exist, so no such data reaches the platform yet.
  • Platform evals other than CSAT do not see sub-actor turns yet: voice transcripts for evals keep only conversational roles.

4. Tests

Test File Covers
test_an_environment_with_a_cast_shows_it_and_one_without_shows_none simulate/tests/test_hosted_harness_gateway.py Cast stage output from an archive and its summary; none without one
test_a_suite_indexes_each_scenarios_sub_actors_and_filters_by_them simulate/tests/test_harness_sub_actors.py Bindings indexed and returned; filter by id, none, negated; field counts
test_a_suite_with_no_sub_actors_offers_no_sub_actor_filter same No filter for a suite with no sub-actors
test_a_sub_actor_turn_is_stored_with_its_name same Voice transcript stores the role and name in time order; the CSAT transcript labels it as not the customer
test_a_second_chat_participant_is_its_own_message_with_its_name same Chat ingestion keeps the participant as its own named row and names no other row; the chat CSAT transcript labels it as not the customer
test_the_sandboxes_are_told_about_sub_actors_only_when_the_setting_is_on same The switch reaches the sandboxes only when on
test_the_environments_list_says_whether_sub_actors_are_on same The list response reports the switch
test_a_sub_goal_column_names_the_actor_it_is_about simulate/tests/test_analytics_functional.py Sub-goal result columns carry actor
stageOutputsToWorld, scenarioFromApi, mapCallDetail, functionCallTranscriptRows, subGoalColumnsFor, template casts frontend/src/api/simulate-environments/__tests__/subActors.test.js Cast mapping; a scenario's sub-actors; a turn's name, for voice and chat; sub-goal columns narrowed by actor; no cast for a surface the harness does not reach
Sub-actor names under a scenario .../scenarios/__tests__/ScenarioTable.test.jsx Names from the cast
Template cast follows the switch .../cards/__tests__/TemplateBuildPanel.test.jsx Adopting brings the cast only when the setting is on; Build is disabled until the setting is known

The touched backend suites (test_harness_sub_actors.py, test_hosted_harness_gateway.py, test_hosted_harness_channels.py, test_alk_simulate_ingestion.py, test_harness_scenario_amend.py, test_harness_scenario_labels.py, test_analytics_functional.py, test_run_results_v3_facets_catalog_parity.py, test_harness_provider.py, test_ended_reason.py): 722 passed; the one failure, test_voice_simulator_uses_single_dataset_language, fails the same way on dev. The frontend simulate-environments API and simulate/environments section suites: 1571 passed, 2 skipped. makemigrations --check reports no changes, and yarn contracts:check and the MCP manifest check pass. CI does not run on this stacked PR until it targets dev.

Tested end to end

On a local platform with this branch, ALK_SUB_ACTORS=true and the companion ALK branch: one voice agent (ride booking, real calls) and one chat agent (order status), each built fresh.

Path (requirement) Result
The switch reaches the build and the run The voice build derived a cast of two on the caller's side; bound scenarios carried their sub-actors into the calls
The cast is stored and shown The environment's cast output lists each sub-actor with its persona and origin, for the built cast and for one added later in the builder chat
Each scenario's sub-actors are indexed sub_actors on the two bound scenarios, null on the other eight
Filter scenarios by sub-actor sub_actors=<id> returned the one scenario binding it; sub_actors=none the eight without
Sub-actor turns stored with role and name Voice calls stored speaker_role = sub_actor with speaker_name ("Alex", "Taylor"), in time order between the caller's and the agent's turns
Sub-goal columns name their actor The friend's sub-goal carries actor: friend_in_thread; the rest primary
The environments list reports the switch sub_actors: true

Found and fixed while testing: a sub-actor added from the builder chat never reached the environment, because checkpoint promotion only copied the contract, scenarios, sub-goals and coverage outputs. It now copies the cast as well (test_a_cast_added_in_the_builder_chat_reaches_the_environment, which fails without the fix).

5. Commands to run the tests

cd futureagi
bin/test simulate/tests/test_harness_sub_actors.py simulate/tests/test_hosted_harness_gateway.py \
  simulate/tests/test_analytics_functional.py simulate/tests/test_run_results_v3_facets_catalog_parity.py -v

cd ../frontend
yarn test:run src/api/simulate-environments src/sections/simulate/environments
yarn contracts:check

Terminal output

<image: backend and frontend test output>

6. Steps to test on local

  1. Set ALK_SUB_ACTORS=true for the backend and run it with the companion ALK branch.
  2. Build an environment for a voice agent and, in the builder chat, add a voice near the person and someone who takes the phone from them.
  3. Open the environment's Contract tab; confirm the actors panel lists both sub-actors with how many scenarios bring each in.
  4. Open the scenarios tab; confirm a scenario that binds them shows them under its name.
  5. Filter scenarios by Sub-actors; confirm none shows only the scenarios without one.
  6. Run a bound scenario and open the call; confirm the second voice's turn is labelled with its name.
  7. Open the run's results and pick a sub-actor under "Actor"; confirm only that sub-actor's sub-goal columns remain, and "Person served" shows the rest.
  8. Open Templates, use the customer support line template, and confirm the overview lists its cast.
  9. Unset ALK_SUB_ACTORS, rebuild, adopt the template again, and confirm none of the above appears.

<video: actors panel, scenario filter, and a call with a sub-actor turn>

7. Scope in / scope out

In Out, and why
Switch, storage, ingestion, contracts A receipt field listing which bound sub-actors entered a call: decided against for now, since it changes the sealed result receipt contract; it is derived from sub-actor turns
Sub-goal results by actor; a generic cast per template vertical Later channels (ui_event, page_content, repo_change, peer_agent): they need harness adapters for those agent kinds first
A cast library shared across environments: parked, it belongs to the environment-versioning design
Actors panel, scenarios table, transcripts A dedicated scenarios-table column: the sub-actors are shown under the scenario name so tables without sub-actors are unchanged

E2E: exempt (harness-gap sub-actors are off by default and only appear in hosted harness runs)

8. Design choices

  • The bindings are stored on the scenario row as written by the harness, so the filter and the UI read the same shape the run used.
  • Sub-actor turns use their own role instead of being folded into user, so attribution survives to evals and metrics stay limited to the person and the agent.
  • One setting gates the feature end to end; the sandbox decides nothing on its own.

Companion PR: future-agi/agent-learning-kit#177

Linear

Closes TH-8651

@KarthikAvinashFI
KarthikAvinashFI changed the base branch from dev to fix/TH-8411-builder-checks-build October 16, 2026 20:08
@KarthikAvinashFI KarthikAvinashFI changed the title feat(simulate): store and show sub-actors, their turns and injected faults feat(simulate): store and show sub-actors and their turns Oct 17, 2026
Base automatically changed from fix/TH-8411-builder-checks-build to dev October 22, 2026 14:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant