The work-requesting schedulers ignore --hpx:numa-sensitive · Issue #7773 · TheHPXProject/hpx · GitHub
Skip to content

The work-requesting schedulers ignore --hpx:numa-sensitive #7773

Open

Description

@hkaiser

The work-requesting schedulers steal across NUMA domains regardless of --hpx:numa-sensitive / hpx.numa_sensitive.

  • local_workrequesting_scheduler::set_scheduler_mode (local_workrequesting_scheduler.hpp:1993) always adds enable_stealing_numa. The thread manager removes it again with update_scheduler_mode for a nonzero level, but the scheduler never reads the flag.
  • Each worker's victim mask only contains the worker itself (:1943), and next_victim (:1635) picks a random worker out of all of them. There is no preference for workers in the same NUMA domain and no way to restrict stealing to it.

For comparison, the local-priority scheduler builds a victim list per worker in on_start_thread (local_priority_queue_scheduler.hpp:1649): workers in the same NUMA domain first, then (if enabled) workers in other domains. #7756 restores the three sensitivity levels (0 - no restriction, 1 - only boundary workers steal from the nearest domains, 2 - no stealing across domains) for the local and local-priority schedulers.

The work-requesting schedulers should at least prefer victims in the same NUMA domain and honor level 2 (no stealing across domains). Until then the documentation of --hpx:numa-sensitive should say that the setting has no effect for them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions