[argobots-discuss] modifying scheduler event frequency?

[Iwasaki, Shintaro]

Hi Phil,

Thanks for using Argobots!  The following is my answers to your questions in addition to some tips.
We would appreciate it if you could share more information about your workload and the purpose so that we can give you more specific suggestions. Also, we welcome any feature requests and bug reports.

1. How to change a scheduler's event frequency?
1.1. Predefined scheduler
First, there is no way to dynamically change the event frequency (even if you hack ABT_sched or a pointer you used in ABT_sched_get_data()... since event_freq is loaded to a local variable).
https://github.com/pmodels/argobots/blob/main/src/sched/basic_wait.c#L102
Currently, using a special ABT_sched_config when you create a scheduler is the cleanest and the only way to change the event frequency.
```
ABT_sched_config config;
int new_freq = 16; // The default value is 50 (https://github.com/pmodels/argobots/blob/main/src/arch/abtd_env.c#L13)
ABT_sched_config_create(&config, ABT_sched_basic_freq, 16, ABT_sched_config_var_end);
```
1.2. Custom scheduler
You can call ABT_xstream_check_events() more frequently after calling ABT_info_trigger_print_all_thread_stacks() (e.g., when a global flag is on, a scheduler calls ABT_xstream_check_events() in every iteration).

2. ABT_info_trigger_print_all_thread_stacks()
ABT_info_trigger_print_all_thread_stacks() is designed for deadlock/livelock detection, so if your program is just (extremely) slow, ABT_info_trigger_print_all_thread_stacks() might not be a right routine to try.

> The first example I tried appeared to essentially defer dump until shutdown.
When one of your ULTs encounters a deadlock, the scheduling loop might not be called. You might want to set timeout for ABT_info_trigger_print_all_thread_stacks(). For example, the following test will forcibly print stacks after 3.0 seconds even if some execution streams have not reached ABT_xstream_check_events().
https://github.com/pmodels/argobots/blob/main/test/basic/info_stackdump2.c#L30
This is dangerous (I mean, it can dump a stack of a running ULT), so Argobots does not guarantee anything but it might be helpful to understand a deadlock issue sometimes.

===

3. Some tips
3.1. gdb
I would use gdb if it would be available to check a deadlock/performance issue. For example, if a program looks hanging, I will attach a debugger to that process and see what's happening.
3.2. libunwind for ABT_info_trigger_print_all_thread_stacks()
Unless you are an extremely skillful low-level programmer, I would recommend you enable libunwind for better understanding of stacks. By default, ABT_info_trigger_print_all_thread_stacks() dumps raw hex stack data.
3.3. "occasionally tied up in system calls"
I'm not sure if it's happening in the Argobots runtime (now Argobots uses futex for synchronization on external threads), but if you are calling ABT_info_trigger_print_all_thread_stacks() in a signal handler, please be aware that system calls terminate (e.g., futex, poll, or pthread_cond_wait) if a signal hits the process.
(Argobots synchronization implementation is aware of it and should not be affected by an external signal. This property is thoroughly tested: https://github.com/pmodels/argobots/blob/main/test/util/abttest.c#L245-L287)
Note that the user can call ABT_info_trigger_print_all_thread_stacks() on a normal thread without any problem. It is implemented just in an async-signal safe manner.
3.4. Stack dump
ABT_info_print_thread_stacks_in_pool() is a less invasive way to print stacks, especially if you know a list of pools. It prints stacks immediately. Basically, ABT_info_trigger_print_all_thread_stacks() sets a flag to call ABT_info_print_thread_stacks_in_pool() for all pools after all the execution streams stop in ABT_xstream_check_events().

Thanks,
Shintaro

________________________________
From: Phil Carns via discuss <discuss at lists.argobots.org>
Sent: Wednesday, April 14, 2021 2:18 PM
To: discuss at lists.argobots.org <discuss at lists.argobots.org>
Cc: Carns, Philip H. <carns at mcs.anl.gov>
Subject: [argobots-discuss] modifying scheduler event frequency?


Hi all,

Is there a clean way to change a scheduler's event frequency on the fly?

Browsing the API, I see two possibilities:

  *   set it when the scheduler is first created (using ABT_sched_basic_freq?)
  *   set it dynamically by manipulating the ABT_sched_get_data() pointer, but this seems especially dangerous since the sched data struct definition isn't public (i.e. it could cause memory corruption if the internal struct def changed)

For some context (in case there is a different way to go about this entirely), I'm trying to figure out how to get ABT_info_trigger_print_all_thread_stacks() to print information more quickly, which IIUC relies on getting the active schedulers to call get_events() sooner.

I'm happy to add some explicit ABT_thread_yield() shortly after the ABT_info_trigger_print_all_thread_stacks() to at least get the calling ES to execute it's scheduler loop immediately, but I think that won't matter much if it doesn't trip the frequency counter when I do it.

Without this (at least with the _wait scheduler and threads that are occasionally tied up in system calls) I think the stack dump is likely to trigger too late to display what I'm hoping to capture when I call it.  The first example I tried appeared to essentially defer dump until shutdown.

thanks!

-Phil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.argobots.org/pipermail/discuss/attachments/20210414/a114c4b9/attachment.html>