From siwasaki at anl.gov Thu Apr 1 07:30:39 2021 From: siwasaki at anl.gov (Iwasaki, Shintaro) Date: Thu, 1 Apr 2021 12:30:39 +0000 Subject: [argobots-discuss] Announcing the release of Argobots 1.1 Message-ID: Dear Argobots Users and Developers, We are excited to announce that a new stable release of Argobots 1.1 is now available for download. Argobots 1.1 added numerous features such as tool interface, extended work-unit data access routines, improved affinity interface, and stack unwinding for debugging. Argobots 1.1 fixed several bugs and clarified the behavior of each API. Production environments are encouraged to upgrade to this release. All the changes from Argobots 1.0 are summarized here: https://www.argobots.org/changes/argobots_1_1/ Please find the tarball on our downloads page: https://www.argobots.org/downloads/ We thank all the Argobots users and developers for supporting us. Best Regards, The Argobots team =============================================================================== Changes in 1.1 =============================================================================== - Added a tool interface to support a profiler (see ABTX_prof) - Supported the stack unwinding feature - Implemented static initializers for ABT_mutex and ABT_cond - Improved work-unit-local storage support - Added several utility functions - Improved support for affinity settings via an environmental variable - Enhanced memory management of ULT stacks and descriptors - Used futex when external threads call Argobots synchronization operations - Revised the API specification - Added support for XLC, PGI, Solaris, and ARM HPC compilers - Supported Autoconf 2.70 - Fixed several bugs -------------- next part -------------- An HTML attachment was scrubbed... URL: From carns at mcs.anl.gov Wed Apr 14 14:18:12 2021 From: carns at mcs.anl.gov (Phil Carns) Date: Wed, 14 Apr 2021 15:18:12 -0400 Subject: [argobots-discuss] modifying scheduler event frequency? Message-ID: Hi all, Is there a clean way to change a scheduler's event frequency on the fly? Browsing the API, I see two possibilities: * set it when the scheduler is first created (using ABT_sched_basic_freq?) * set it dynamically by manipulating the ABT_sched_get_data() pointer, but this seems especially dangerous since the sched data struct definition isn't public (i.e. it could cause memory corruption if the internal struct def changed) For some context (in case there is a different way to go about this entirely), I'm trying to figure out how to get ABT_info_trigger_print_all_thread_stacks() to print information more quickly, which IIUC relies on getting the active schedulers to call get_events() sooner. I'm happy to add some explicit ABT_thread_yield() shortly after the ABT_info_trigger_print_all_thread_stacks() to at least get the calling ES to execute it's scheduler loop immediately, but I think that won't matter much if it doesn't trip the frequency counter when I do it. Without this (at least with the _wait scheduler and threads that are occasionally tied up in system calls) I think the stack dump is likely to trigger too late to display what I'm hoping to capture when I call it.? The first example I tried appeared to essentially defer dump until shutdown. thanks! -Phil -------------- next part -------------- An HTML attachment was scrubbed... URL: From siwasaki at anl.gov Wed Apr 14 14:57:26 2021 From: siwasaki at anl.gov (Iwasaki, Shintaro) Date: Wed, 14 Apr 2021 19:57:26 +0000 Subject: [argobots-discuss] modifying scheduler event frequency? In-Reply-To: References: Message-ID: Hi Phil, Thanks for using Argobots! The following is my answers to your questions in addition to some tips. We would appreciate it if you could share more information about your workload and the purpose so that we can give you more specific suggestions. Also, we welcome any feature requests and bug reports. 1. How to change a scheduler's event frequency? 1.1. Predefined scheduler First, there is no way to dynamically change the event frequency (even if you hack ABT_sched or a pointer you used in ABT_sched_get_data()... since event_freq is loaded to a local variable). https://github.com/pmodels/argobots/blob/main/src/sched/basic_wait.c#L102 Currently, using a special ABT_sched_config when you create a scheduler is the cleanest and the only way to change the event frequency. ``` ABT_sched_config config; int new_freq = 16; // The default value is 50 (https://github.com/pmodels/argobots/blob/main/src/arch/abtd_env.c#L13) ABT_sched_config_create(&config, ABT_sched_basic_freq, 16, ABT_sched_config_var_end); ``` 1.2. Custom scheduler You can call ABT_xstream_check_events() more frequently after calling ABT_info_trigger_print_all_thread_stacks() (e.g., when a global flag is on, a scheduler calls ABT_xstream_check_events() in every iteration). 2. ABT_info_trigger_print_all_thread_stacks() ABT_info_trigger_print_all_thread_stacks() is designed for deadlock/livelock detection, so if your program is just (extremely) slow, ABT_info_trigger_print_all_thread_stacks() might not be a right routine to try. > The first example I tried appeared to essentially defer dump until shutdown. When one of your ULTs encounters a deadlock, the scheduling loop might not be called. You might want to set timeout for ABT_info_trigger_print_all_thread_stacks(). For example, the following test will forcibly print stacks after 3.0 seconds even if some execution streams have not reached ABT_xstream_check_events(). https://github.com/pmodels/argobots/blob/main/test/basic/info_stackdump2.c#L30 This is dangerous (I mean, it can dump a stack of a running ULT), so Argobots does not guarantee anything but it might be helpful to understand a deadlock issue sometimes. === 3. Some tips 3.1. gdb I would use gdb if it would be available to check a deadlock/performance issue. For example, if a program looks hanging, I will attach a debugger to that process and see what's happening. 3.2. libunwind for ABT_info_trigger_print_all_thread_stacks() Unless you are an extremely skillful low-level programmer, I would recommend you enable libunwind for better understanding of stacks. By default, ABT_info_trigger_print_all_thread_stacks() dumps raw hex stack data. 3.3. "occasionally tied up in system calls" I'm not sure if it's happening in the Argobots runtime (now Argobots uses futex for synchronization on external threads), but if you are calling ABT_info_trigger_print_all_thread_stacks() in a signal handler, please be aware that system calls terminate (e.g., futex, poll, or pthread_cond_wait) if a signal hits the process. (Argobots synchronization implementation is aware of it and should not be affected by an external signal. This property is thoroughly tested: https://github.com/pmodels/argobots/blob/main/test/util/abttest.c#L245-L287) Note that the user can call ABT_info_trigger_print_all_thread_stacks() on a normal thread without any problem. It is implemented just in an async-signal safe manner. 3.4. Stack dump ABT_info_print_thread_stacks_in_pool() is a less invasive way to print stacks, especially if you know a list of pools. It prints stacks immediately. Basically, ABT_info_trigger_print_all_thread_stacks() sets a flag to call ABT_info_print_thread_stacks_in_pool() for all pools after all the execution streams stop in ABT_xstream_check_events(). Thanks, Shintaro ________________________________ From: Phil Carns via discuss Sent: Wednesday, April 14, 2021 2:18 PM To: discuss at lists.argobots.org Cc: Carns, Philip H. Subject: [argobots-discuss] modifying scheduler event frequency? Hi all, Is there a clean way to change a scheduler's event frequency on the fly? Browsing the API, I see two possibilities: * set it when the scheduler is first created (using ABT_sched_basic_freq?) * set it dynamically by manipulating the ABT_sched_get_data() pointer, but this seems especially dangerous since the sched data struct definition isn't public (i.e. it could cause memory corruption if the internal struct def changed) For some context (in case there is a different way to go about this entirely), I'm trying to figure out how to get ABT_info_trigger_print_all_thread_stacks() to print information more quickly, which IIUC relies on getting the active schedulers to call get_events() sooner. I'm happy to add some explicit ABT_thread_yield() shortly after the ABT_info_trigger_print_all_thread_stacks() to at least get the calling ES to execute it's scheduler loop immediately, but I think that won't matter much if it doesn't trip the frequency counter when I do it. Without this (at least with the _wait scheduler and threads that are occasionally tied up in system calls) I think the stack dump is likely to trigger too late to display what I'm hoping to capture when I call it. The first example I tried appeared to essentially defer dump until shutdown. thanks! -Phil -------------- next part -------------- An HTML attachment was scrubbed... URL: From carns at mcs.anl.gov Wed Apr 14 15:42:16 2021 From: carns at mcs.anl.gov (Phil Carns) Date: Wed, 14 Apr 2021 16:42:16 -0400 Subject: [argobots-discuss] modifying scheduler event frequency? In-Reply-To: References: Message-ID: Ah, thanks for the thorough information as always Shintaro :) print_all_thread_stacks() was? tempting because it would potentially encompass more (in the Mochi use case, it would pick up hypothetical pools created by higher level components that we don't have a reference to).? Based on the information in this email thread, though, I think I'm better off focusing on pools under our control so that I can use print_thread_stacks_in_pool().? This should work fine; I was just over-thinking the use case.? The pools are under our own control in the vast majority of configurations. In the big picture, I was exploring this because of a bug report we have from one of our collaborators who is getting a nonsensical hang in a complex scenario that we can't easily reproduce or attach a debugger to.? I would like to be able to send an RPC to a process at an arbitrary point in time and dump what it is up to so that we can understand why it didn't complete something it was trying to do. libunwind sounds great :)? I probably would have been asking about that next. I guess I'll use this as an opportunity to request/suggest that the libunwind capability be added as a variant to the argobots spack package (along with a way to enable future mprotect / stack canary checks). We use argobots almost exclusively with spack at this point.? Not that argobots itself is hard to compile manually, but it is often one of a large number of dependencies that we need to build, so it's best to just unify them in one packaging system.? It would be straightforward for us to set up an alternative environment yaml with various argobots debugging capabilities enabled for development/debugging purposes. thanks! -Phil On 4/14/21 3:57 PM, Iwasaki, Shintaro wrote: > Hi Phil, > > Thanks for using Argobots! ?The following is my answers to your > questions in addition to some tips. > We would appreciate it if you could share more information about your > workload and the purpose so that we can give you more specific > suggestions. Also, we welcome any feature requests and bug reports. > > 1. How to change a scheduler's event frequency? > 1.1. Predefined scheduler > First, there is no way to dynamically change the event frequency (even > if you hack ABT_sched or a pointer you used in ABT_sched_get_data()... > since event_freq is loaded to a local variable). > https://github.com/pmodels/argobots/blob/main/src/sched/basic_wait.c#L102 > Currently, using a special ABT_sched_config when you create a > scheduler is the cleanest and the only way to change the event frequency. > ``` > ABT_sched_config config; > int new_freq = 16; // The default value is 50 > (https://github.com/pmodels/argobots/blob/main/src/arch/abtd_env.c#L13) > ABT_sched_config_create(&config, ABT_sched_basic_freq, 16, > ABT_sched_config_var_end); > ``` > 1.2. Custom scheduler > You can call ABT_xstream_check_events() more frequently after calling > ABT_info_trigger_print_all_thread_stacks() (e.g., when a global flag > is on, a scheduler calls ABT_xstream_check_events() in every iteration). > > 2. ABT_info_trigger_print_all_thread_stacks() > ABT_info_trigger_print_all_thread_stacks() is designed for > deadlock/livelock detection, so if your program is just (extremely) > slow, ABT_info_trigger_print_all_thread_stacks() might not be a right > routine to try. > > > The first example I tried appeared to essentially defer dump until > shutdown. > When one of your ULTs encounters a deadlock, the scheduling loop might > not be called. You might want to set timeout for > ABT_info_trigger_print_all_thread_stacks(). For example, the following > test will forcibly print stacks after 3.0 seconds even if some > execution streams have not reached ABT_xstream_check_events(). > https://github.com/pmodels/argobots/blob/main/test/basic/info_stackdump2.c#L30 > This is dangerous (I mean, it can dump a stack of a running ULT), so > Argobots does not guarantee anything but it might be helpful to > understand a deadlock issue sometimes. > > === > > 3. Some tips > 3.1. gdb > I would use gdb if it would be available to check a > deadlock/performance issue. For example, if a program looks hanging, I > will attach a debugger to that process and see what's happening. > 3.2. libunwind for ABT_info_trigger_print_all_thread_stacks() > Unless you are an extremely skillful low-level programmer, I would > recommend you enable libunwind for better understanding of stacks. By > default, ABT_info_trigger_print_all_thread_stacks() dumps raw hex > stack data. > 3.3. "occasionally tied up in system calls" > I'm not sure if it's happening in the Argobots runtime (now Argobots > uses futex for synchronization on external threads), but if you are > calling ABT_info_trigger_print_all_thread_stacks() in a signal > handler, please be aware that system calls terminate (e.g., futex, > poll, or pthread_cond_wait) if a signal hits the process. > (Argobots synchronization implementation is aware of it and should not > be affected by an external signal. This property is thoroughly tested: > https://github.com/pmodels/argobots/blob/main/test/util/abttest.c#L245-L287) > Note that the user can call ABT_info_trigger_print_all_thread_stacks() > on a normal thread without any problem. It is implemented just in an > async-signal safe manner. > 3.4. Stack dump > ABT_info_print_thread_stacks_in_pool() is a less invasive way to print > stacks, especially if you know a list of pools. It prints stacks > immediately. Basically, ABT_info_trigger_print_all_thread_stacks() > sets a flag to call ABT_info_print_thread_stacks_in_pool() for all > pools after all the execution streams stop in ABT_xstream_check_events(). > > Thanks, > Shintaro > > ------------------------------------------------------------------------ > *From:* Phil Carns via discuss > *Sent:* Wednesday, April 14, 2021 2:18 PM > *To:* discuss at lists.argobots.org > *Cc:* Carns, Philip H. > *Subject:* [argobots-discuss] modifying scheduler event frequency? > > Hi all, > > Is there a clean way to change a scheduler's event frequency on the fly? > > Browsing the API, I see two possibilities: > > * set it when the scheduler is first created (using > ABT_sched_basic_freq?) > * set it dynamically by manipulating the ABT_sched_get_data() > pointer, but this seems especially dangerous since the sched data > struct definition isn't public (i.e. it could cause memory > corruption if the internal struct def changed) > > For some context (in case there is a different way to go about this > entirely), I'm trying to figure out how to get > ABT_info_trigger_print_all_thread_stacks() to print information more > quickly, which IIUC relies on getting the active schedulers to call > get_events() sooner. > > I'm happy to add some explicit ABT_thread_yield() shortly after the > ABT_info_trigger_print_all_thread_stacks() to at least get the calling > ES to execute it's scheduler loop immediately, but I think that won't > matter much if it doesn't trip the frequency counter when I do it. > > Without this (at least with the _wait scheduler and threads that are > occasionally tied up in system calls) I think the stack dump is likely > to trigger too late to display what I'm hoping to capture when I call > it.? The first example I tried appeared to essentially defer dump > until shutdown. > > thanks! > > -Phil > -------------- next part -------------- An HTML attachment was scrubbed... URL: From siwasaki at anl.gov Wed Apr 14 15:55:02 2021 From: siwasaki at anl.gov (Iwasaki, Shintaro) Date: Wed, 14 Apr 2021 20:55:02 +0000 Subject: [argobots-discuss] modifying scheduler event frequency? In-Reply-To: References: , Message-ID: Hi Phil, Thanks. I can understand a bigger picture. > ABT_info_print_thread_stacks_in_pool() I hope it works. Note that print_thread_stacks_in_pool() is not async-signal safe (ABT_info_trigger_print_all_thread_stacks() is an exception), so please don't call it in a signal handler. > We use argobots almost exclusively with spack at this point. Many HPC users use Spack to build dependent libraries. I will add some debug options (including libunwind, stack guard, ...) as well as other major options to the Spack Argobots package. We are also implementing an mprotect-based stack guard option (which is not in Argobots 1.1, though). Overall, please give us a week or so in total. There is large room for improvement of the debugging/profiling capability. If you have any questions, requests, and/or suggestions, please feel free to tell us. Thanks, Shintaro ________________________________ From: Carns, Philip H. Sent: Wednesday, April 14, 2021 3:42 PM To: Iwasaki, Shintaro ; discuss at lists.argobots.org Subject: Re: [argobots-discuss] modifying scheduler event frequency? Ah, thanks for the thorough information as always Shintaro :) print_all_thread_stacks() was tempting because it would potentially encompass more (in the Mochi use case, it would pick up hypothetical pools created by higher level components that we don't have a reference to). Based on the information in this email thread, though, I think I'm better off focusing on pools under our control so that I can use print_thread_stacks_in_pool(). This should work fine; I was just over-thinking the use case. The pools are under our own control in the vast majority of configurations. In the big picture, I was exploring this because of a bug report we have from one of our collaborators who is getting a nonsensical hang in a complex scenario that we can't easily reproduce or attach a debugger to. I would like to be able to send an RPC to a process at an arbitrary point in time and dump what it is up to so that we can understand why it didn't complete something it was trying to do. libunwind sounds great :) I probably would have been asking about that next. I guess I'll use this as an opportunity to request/suggest that the libunwind capability be added as a variant to the argobots spack package (along with a way to enable future mprotect / stack canary checks). We use argobots almost exclusively with spack at this point. Not that argobots itself is hard to compile manually, but it is often one of a large number of dependencies that we need to build, so it's best to just unify them in one packaging system. It would be straightforward for us to set up an alternative environment yaml with various argobots debugging capabilities enabled for development/debugging purposes. thanks! -Phil On 4/14/21 3:57 PM, Iwasaki, Shintaro wrote: Hi Phil, Thanks for using Argobots! The following is my answers to your questions in addition to some tips. We would appreciate it if you could share more information about your workload and the purpose so that we can give you more specific suggestions. Also, we welcome any feature requests and bug reports. 1. How to change a scheduler's event frequency? 1.1. Predefined scheduler First, there is no way to dynamically change the event frequency (even if you hack ABT_sched or a pointer you used in ABT_sched_get_data()... since event_freq is loaded to a local variable). https://github.com/pmodels/argobots/blob/main/src/sched/basic_wait.c#L102 Currently, using a special ABT_sched_config when you create a scheduler is the cleanest and the only way to change the event frequency. ``` ABT_sched_config config; int new_freq = 16; // The default value is 50 (https://github.com/pmodels/argobots/blob/main/src/arch/abtd_env.c#L13) ABT_sched_config_create(&config, ABT_sched_basic_freq, 16, ABT_sched_config_var_end); ``` 1.2. Custom scheduler You can call ABT_xstream_check_events() more frequently after calling ABT_info_trigger_print_all_thread_stacks() (e.g., when a global flag is on, a scheduler calls ABT_xstream_check_events() in every iteration). 2. ABT_info_trigger_print_all_thread_stacks() ABT_info_trigger_print_all_thread_stacks() is designed for deadlock/livelock detection, so if your program is just (extremely) slow, ABT_info_trigger_print_all_thread_stacks() might not be a right routine to try. > The first example I tried appeared to essentially defer dump until shutdown. When one of your ULTs encounters a deadlock, the scheduling loop might not be called. You might want to set timeout for ABT_info_trigger_print_all_thread_stacks(). For example, the following test will forcibly print stacks after 3.0 seconds even if some execution streams have not reached ABT_xstream_check_events(). https://github.com/pmodels/argobots/blob/main/test/basic/info_stackdump2.c#L30 This is dangerous (I mean, it can dump a stack of a running ULT), so Argobots does not guarantee anything but it might be helpful to understand a deadlock issue sometimes. === 3. Some tips 3.1. gdb I would use gdb if it would be available to check a deadlock/performance issue. For example, if a program looks hanging, I will attach a debugger to that process and see what's happening. 3.2. libunwind for ABT_info_trigger_print_all_thread_stacks() Unless you are an extremely skillful low-level programmer, I would recommend you enable libunwind for better understanding of stacks. By default, ABT_info_trigger_print_all_thread_stacks() dumps raw hex stack data. 3.3. "occasionally tied up in system calls" I'm not sure if it's happening in the Argobots runtime (now Argobots uses futex for synchronization on external threads), but if you are calling ABT_info_trigger_print_all_thread_stacks() in a signal handler, please be aware that system calls terminate (e.g., futex, poll, or pthread_cond_wait) if a signal hits the process. (Argobots synchronization implementation is aware of it and should not be affected by an external signal. This property is thoroughly tested: https://github.com/pmodels/argobots/blob/main/test/util/abttest.c#L245-L287) Note that the user can call ABT_info_trigger_print_all_thread_stacks() on a normal thread without any problem. It is implemented just in an async-signal safe manner. 3.4. Stack dump ABT_info_print_thread_stacks_in_pool() is a less invasive way to print stacks, especially if you know a list of pools. It prints stacks immediately. Basically, ABT_info_trigger_print_all_thread_stacks() sets a flag to call ABT_info_print_thread_stacks_in_pool() for all pools after all the execution streams stop in ABT_xstream_check_events(). Thanks, Shintaro ________________________________ From: Phil Carns via discuss Sent: Wednesday, April 14, 2021 2:18 PM To: discuss at lists.argobots.org Cc: Carns, Philip H. Subject: [argobots-discuss] modifying scheduler event frequency? Hi all, Is there a clean way to change a scheduler's event frequency on the fly? Browsing the API, I see two possibilities: * set it when the scheduler is first created (using ABT_sched_basic_freq?) * set it dynamically by manipulating the ABT_sched_get_data() pointer, but this seems especially dangerous since the sched data struct definition isn't public (i.e. it could cause memory corruption if the internal struct def changed) For some context (in case there is a different way to go about this entirely), I'm trying to figure out how to get ABT_info_trigger_print_all_thread_stacks() to print information more quickly, which IIUC relies on getting the active schedulers to call get_events() sooner. I'm happy to add some explicit ABT_thread_yield() shortly after the ABT_info_trigger_print_all_thread_stacks() to at least get the calling ES to execute it's scheduler loop immediately, but I think that won't matter much if it doesn't trip the frequency counter when I do it. Without this (at least with the _wait scheduler and threads that are occasionally tied up in system calls) I think the stack dump is likely to trigger too late to display what I'm hoping to capture when I call it. The first example I tried appeared to essentially defer dump until shutdown. thanks! -Phil -------------- next part -------------- An HTML attachment was scrubbed... URL: From carns at mcs.anl.gov Wed Apr 14 15:56:26 2021 From: carns at mcs.anl.gov (Phil Carns) Date: Wed, 14 Apr 2021 16:56:26 -0400 Subject: [argobots-discuss] modifying scheduler event frequency? In-Reply-To: References: Message-ID: Heh, I guess while I'm making a wish list I would like to be able to enable the tool interface as well, for the same reasons (so that we can reproduce a complex spack build stack except for replacing a standard argobots install with one that has extra profiling and debugging capability). Sorry, that's all off topic from my original question, but I wanted to mention this while I'm thinking about it. thanks, -Phil On 4/14/21 4:42 PM, Phil Carns via discuss wrote: > > > I guess I'll use this as an opportunity to request/suggest that the > libunwind capability be added as a variant to the argobots spack > package (along with a way to enable future mprotect / stack canary > checks). > From carns at mcs.anl.gov Wed Apr 14 16:01:40 2021 From: carns at mcs.anl.gov (Phil Carns) Date: Wed, 14 Apr 2021 17:01:40 -0400 Subject: [argobots-discuss] modifying scheduler event frequency? In-Reply-To: References: Message-ID: <45d63d8e-22ad-041c-07a2-653c93c0d208@mcs.anl.gov> On 4/14/21 4:55 PM, Iwasaki, Shintaro wrote: > Hi Phil, > > Thanks. I can understand a bigger picture. > > > ABT_info_print_thread_stacks_in_pool() > I hope it works. Note that print_thread_stacks_in_pool() is not > async-signal safe?(ABT_info_trigger_print_all_thread_stacks() is an > exception), so please don't call it in a signal handler. Ok, no problem.? We don't do much via signals in Mochi (almost all of our control capabilities are triggered via RPCs that launch ULTs to do the work). > > >?We use argobots almost exclusively with spack at this point. > Many HPC users use Spack to build dependent libraries. I will add some > debug options (including libunwind, stack guard, ...) as well as other > major options to the Spack Argobots package. We are also implementing > an mprotect-based stack guard option (which is not in Argobots 1.1, > though). > > Overall, please give us a week or so in total. > > There is large room for improvement of the debugging/profiling capability. > If you have any questions, requests, and/or suggestions, please feel > free to tell us. Sounds great!? Debuggability seems to be the next frontier for our project, so we'll probably be experimenting with more of these capabilities as time goes on.? It's taken a little while for us to recognize which design/debugging patterns would be most useful. thanks, -Phil > > Thanks, > Shintaro > > > ------------------------------------------------------------------------ > *From:* Carns, Philip H. > *Sent:* Wednesday, April 14, 2021 3:42 PM > *To:* Iwasaki, Shintaro ; discuss at lists.argobots.org > > *Subject:* Re: [argobots-discuss] modifying scheduler event frequency? > > Ah, thanks for the thorough information as always Shintaro :) > > > print_all_thread_stacks() was? tempting because it would potentially > encompass more (in the Mochi use case, it would pick up hypothetical > pools created by higher level components that we don't have a > reference to).? Based on the information in this email thread, though, > I think I'm better off focusing on pools under our control so that I > can use print_thread_stacks_in_pool().? This should work fine; I was > just over-thinking the use case.? The pools are under our own control > in the vast majority of configurations. > > > In the big picture, I was exploring this because of a bug report we > have from one of our collaborators who is getting a nonsensical hang > in a complex scenario that we can't easily reproduce or attach a > debugger to.? I would like to be able to send an RPC to a process at > an arbitrary point in time and dump what it is up to so that we can > understand why it didn't complete something it was trying to do. > > > libunwind sounds great :)? I probably would have been asking about > that next. > > > I guess I'll use this as an opportunity to request/suggest that the > libunwind capability be added as a variant to the argobots spack > package (along with a way to enable future mprotect / stack canary > checks). > > > We use argobots almost exclusively with spack at this point. Not that > argobots itself is hard to compile manually, but it is often one of a > large number of dependencies that we need to build, so it's best to > just unify them in one packaging system.? It would be straightforward > for us to set up an alternative environment yaml with various argobots > debugging capabilities enabled for development/debugging purposes. > > > thanks! > > -Phil > > > On 4/14/21 3:57 PM, Iwasaki, Shintaro wrote: >> Hi Phil, >> >> Thanks for using Argobots! ?The following is my answers to your >> questions in addition to some tips. >> We would appreciate it if you could share more information about your >> workload and the purpose so that we can give you more specific >> suggestions. Also, we welcome any feature requests and bug reports. >> >> 1. How to change a scheduler's event frequency? >> 1.1. Predefined scheduler >> First, there is no way to dynamically change the event frequency >> (even if you hack ABT_sched or a pointer you used in >> ABT_sched_get_data()... since event_freq is loaded to a local variable). >> https://github.com/pmodels/argobots/blob/main/src/sched/basic_wait.c#L102 >> >> Currently, using a special ABT_sched_config when you create a >> scheduler is the cleanest and the only way to change the event frequency. >> ``` >> ABT_sched_config config; >> int new_freq = 16; // The default value is 50 >> (https://github.com/pmodels/argobots/blob/main/src/arch/abtd_env.c#L13 >> ) >> ABT_sched_config_create(&config, ABT_sched_basic_freq, 16, >> ABT_sched_config_var_end); >> ``` >> 1.2. Custom scheduler >> You can call ABT_xstream_check_events() more frequently after calling >> ABT_info_trigger_print_all_thread_stacks() (e.g., when a global flag >> is on, a scheduler calls ABT_xstream_check_events() in every iteration). >> >> 2. ABT_info_trigger_print_all_thread_stacks() >> ABT_info_trigger_print_all_thread_stacks() is designed for >> deadlock/livelock detection, so if your program is just (extremely) >> slow, ABT_info_trigger_print_all_thread_stacks() might not be a right >> routine to try. >> >> > The first example I tried appeared to essentially defer dump until >> shutdown. >> When one of your ULTs encounters a deadlock, the scheduling loop >> might not be called. You might want to set timeout for >> ABT_info_trigger_print_all_thread_stacks(). For example, the >> following test will forcibly print stacks after 3.0 seconds even if >> some execution streams have not reached ABT_xstream_check_events(). >> https://github.com/pmodels/argobots/blob/main/test/basic/info_stackdump2.c#L30 >> >> This is dangerous (I mean, it can dump a stack of a running ULT), so >> Argobots does not guarantee anything but it might be helpful to >> understand a deadlock issue sometimes. >> >> === >> >> 3. Some tips >> 3.1. gdb >> I would use gdb if it would be available to check a >> deadlock/performance issue. For example, if a program looks hanging, >> I will attach a debugger to that process and see what's happening. >> 3.2. libunwind for ABT_info_trigger_print_all_thread_stacks() >> Unless you are an extremely skillful low-level programmer, I would >> recommend you enable libunwind for better understanding of stacks. By >> default, ABT_info_trigger_print_all_thread_stacks() dumps raw hex >> stack data. >> 3.3. "occasionally tied up in system calls" >> I'm not sure if it's happening in the Argobots runtime (now Argobots >> uses futex for synchronization on external threads), but if you are >> calling ABT_info_trigger_print_all_thread_stacks() in a signal >> handler, please be aware that system calls terminate (e.g., futex, >> poll, or pthread_cond_wait) if a signal hits the process. >> (Argobots synchronization implementation is aware of it and should >> not be affected by an external signal. This property is thoroughly >> tested: >> https://github.com/pmodels/argobots/blob/main/test/util/abttest.c#L245-L287 >> ) >> Note that the user can call >> ABT_info_trigger_print_all_thread_stacks() on a normal thread without >> any problem. It is implemented just in an async-signal safe manner. >> 3.4. Stack dump >> ABT_info_print_thread_stacks_in_pool() is a less invasive way to >> print stacks, especially if you know a list of pools. It prints >> stacks immediately. Basically, >> ABT_info_trigger_print_all_thread_stacks() sets a flag to call >> ABT_info_print_thread_stacks_in_pool() for all pools after all the >> execution streams stop in ABT_xstream_check_events(). >> >> Thanks, >> Shintaro >> >> ------------------------------------------------------------------------ >> *From:* Phil Carns via discuss >> >> *Sent:* Wednesday, April 14, 2021 2:18 PM >> *To:* discuss at lists.argobots.org >> >> *Cc:* Carns, Philip H. >> *Subject:* [argobots-discuss] modifying scheduler event frequency? >> >> Hi all, >> >> Is there a clean way to change a scheduler's event frequency on the fly? >> >> Browsing the API, I see two possibilities: >> >> * set it when the scheduler is first created (using >> ABT_sched_basic_freq?) >> * set it dynamically by manipulating the ABT_sched_get_data() >> pointer, but this seems especially dangerous since the sched data >> struct definition isn't public (i.e. it could cause memory >> corruption if the internal struct def changed) >> >> For some context (in case there is a different way to go about this >> entirely), I'm trying to figure out how to get >> ABT_info_trigger_print_all_thread_stacks() to print information more >> quickly, which IIUC relies on getting the active schedulers to call >> get_events() sooner. >> >> I'm happy to add some explicit ABT_thread_yield() shortly after the >> ABT_info_trigger_print_all_thread_stacks() to at least get the >> calling ES to execute it's scheduler loop immediately, but I think >> that won't matter much if it doesn't trip the frequency counter when >> I do it. >> >> Without this (at least with the _wait scheduler and threads that are >> occasionally tied up in system calls) I think the stack dump is >> likely to trigger too late to display what I'm hoping to capture when >> I call it.? The first example I tried appeared to essentially defer >> dump until shutdown. >> >> thanks! >> >> -Phil >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From carns at mcs.anl.gov Wed Apr 14 18:44:36 2021 From: carns at mcs.anl.gov (Phil Carns) Date: Wed, 14 Apr 2021 19:44:36 -0400 Subject: [argobots-discuss] dumping stack information for running ULTs Message-ID: <8fa1344f-189f-09c7-f7a1-0f271bc56dc4@mcs.anl.gov> Hi all, I followed Shintaro's suggestions from the previous list email thread and was able to get a stack dump routine that works as expected.? For all pools under my control I can see stack information like this for each ULT, with stack unwinding: == pool (0x557d5b8c4580) == === ULT (0x7f47429590c1) === id??????? : 0 ctx?????? : 0x7f4742959120 p_ctx??? : 0x7f4742958fe0 p_link?? : (nil) stack???? : 0x7f47427590c0 stacksize : 2097152 #0 0x7f4744461760 in ythread_unwind_stack () <+16> (RSP = 0x7f4742959010) #1 0x7f474445e3dd in ABT_thread_yield () <+157> (RSP = 0x7f4742959020) #2 0x7f474447e255 in __margo_hg_progress_fn () <+645> (RSP = 0x7f4742959040) #3 0x7f474446381e in ABTD_ythread_func_wrapper () <+30> (RSP = 0x7f47429590b0) #4 0x7f47444639c1 in make_fcontext () <+33> (RSP = 0x7f47429590c0) 00007f47427590c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 00007f47427590e0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ... That only works for ULTs that are literally in the pool, though, I think.? I don't believe that I am getting information for ULTs that are executing (and thus are not presently in a pool data structure). Is there any way to accomplish that? I tried to at least get the caller's own stack at least by doing something like this: ??? ABT_thread_self(&self); ??? ABT_info_print_thread_stack(outfile, self); That almost works, but I'm just getting the raw address information and not the translated stack unwinding: p_ctx??? : 0x7f4742758fb8 p_link?? : (nil) stack???? : 0x7f4742559000 stacksize : 2097152 00007f4742559000: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 00007f4742559020: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ... For completeness it would be nice if the caller's stack were also human readable, and even better if I could somehow find stack information for ULTs executing on other ESs as well. For our use case we typically spawn many detached threads into a service pool, so we aren't tracking thread references. thanks, -Phil -------------- next part -------------- An HTML attachment was scrubbed... URL: From siwasaki at anl.gov Wed Apr 14 20:02:04 2021 From: siwasaki at anl.gov (Iwasaki, Shintaro) Date: Thu, 15 Apr 2021 01:02:04 +0000 Subject: [argobots-discuss] dumping stack information for running ULTs In-Reply-To: <8fa1344f-189f-09c7-f7a1-0f271bc56dc4@mcs.anl.gov> References: <8fa1344f-189f-09c7-f7a1-0f271bc56dc4@mcs.anl.gov> Message-ID: Hi Phil, > That only works for ULTs that are literally in the pool, though, I think. Yes, that's right. If that ULT yields, the Argobots runtime can print the stack by either ABT_info_print_thread_stacks_in_pool() or ABT_info_print_thread_stack(). > I don't believe that I am getting information for ULTs that are executing Technically, it's very challenging to dump a "running" ULT's stack from another ULT. libunwind needs a stack pointer (in reality more than a stack pointer), but we cannot get it from a running ULT. For example, the following currently does not work as expected. The Argobots runtime cannot unwind the function stack. ABT_info_print_thread_stack(outfile, another_running_thread); // This does not work. We need to stop a ULT to check its stack information. - Complete "ABT_info_print_thread_stack" Printing a function call stack of a running ULT using libunwind is very challenging. The Argobots runtime needs to identify whether a ULT is running or not, and if it is running, it must be stopped by a signal. This is extremely error-prone. I don't know how to guarantee the atomicity (e.g., two threads call this function for the same ULT?). Stopping a specific execution stream (=Pthreads) might not be portable across OSs, so it might need a few fallback implementations (pthread_kill or some signal-related function). No matter Argobots supports it or not, using a signal itself badly affects other runtimes and applications that use Argobots (e.g., system calls fail unexpectedly, pthread_cond_wait() will wake up, ...). Since it's stack unwinding of the running Pthreads (i.e., nothing is ULT-specific), I am not sure if this feature must be supported by the Argobots runtime considering the harmfulness of the potential signal-based implementation. I understand the demand, and most users might feel the current implementation is incomplete. If this feature is really needed, we will more seriously consider the design, but please do not assume that we can provide this implementation very soon, though finally the priority depends on urgency and significance. - "ABT_self_print_thread_stack" The following implementation is easy. We can safely stop the caller ULT, so libunwind can print the function stack. ABT_self_print_thread_stack(FILE *fp); If this is sufficient, I can implement this quickly. To print the running ULT's stack, basically the user can launch a signal handler on its underlying execution stream and call this function in that signal handler. We do not plan to guarantee async-signal safety for ABT_self_print_thread_stack(), so if this is the way to go, please let us know so that we will do our best to make it async-signal safe. Thanks, Shintaro ________________________________ From: Phil Carns via discuss Sent: Wednesday, April 14, 2021 6:44 PM To: discuss at lists.argobots.org Cc: Carns, Philip H. Subject: [argobots-discuss] dumping stack information for running ULTs Hi all, I followed Shintaro's suggestions from the previous list email thread and was able to get a stack dump routine that works as expected. For all pools under my control I can see stack information like this for each ULT, with stack unwinding: == pool (0x557d5b8c4580) == === ULT (0x7f47429590c1) === id : 0 ctx : 0x7f4742959120 p_ctx : 0x7f4742958fe0 p_link : (nil) stack : 0x7f47427590c0 stacksize : 2097152 #0 0x7f4744461760 in ythread_unwind_stack () <+16> (RSP = 0x7f4742959010) #1 0x7f474445e3dd in ABT_thread_yield () <+157> (RSP = 0x7f4742959020) #2 0x7f474447e255 in __margo_hg_progress_fn () <+645> (RSP = 0x7f4742959040) #3 0x7f474446381e in ABTD_ythread_func_wrapper () <+30> (RSP = 0x7f47429590b0) #4 0x7f47444639c1 in make_fcontext () <+33> (RSP = 0x7f47429590c0) 00007f47427590c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 00007f47427590e0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ... That only works for ULTs that are literally in the pool, though, I think. I don't believe that I am getting information for ULTs that are executing (and thus are not presently in a pool data structure). Is there any way to accomplish that? I tried to at least get the caller's own stack at least by doing something like this: ABT_thread_self(&self); ABT_info_print_thread_stack(outfile, self); That almost works, but I'm just getting the raw address information and not the translated stack unwinding: p_ctx : 0x7f4742758fb8 p_link : (nil) stack : 0x7f4742559000 stacksize : 2097152 00007f4742559000: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 00007f4742559020: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ... For completeness it would be nice if the caller's stack were also human readable, and even better if I could somehow find stack information for ULTs executing on other ESs as well. For our use case we typically spawn many detached threads into a service pool, so we aren't tracking thread references. thanks, -Phil -------------- next part -------------- An HTML attachment was scrubbed... URL: From carns at mcs.anl.gov Fri Apr 16 10:35:29 2021 From: carns at mcs.anl.gov (Phil Carns) Date: Fri, 16 Apr 2021 11:35:29 -0400 Subject: [argobots-discuss] dumping stack information for running ULTs In-Reply-To: References: <8fa1344f-189f-09c7-f7a1-0f271bc56dc4@mcs.anl.gov> Message-ID: <3dd7ff5d-c8b5-6e99-480c-e171879a9575@mcs.anl.gov> Hi Shintaro, Thanks for the explanation, that all makes sense. In a perfect world it would be nice to display stack information from running ULTs, but it's not worth incurring the technical debt to pursue any of those methods to collect it right now. I'll stick with the stack traces of ULTs in the pool for our use case (in conjunction with information that I can report from our own explicit tracking).? This discussion was very helpful so that I know how to label the data clearly, though. thanks! -Phil On 4/14/21 9:02 PM, Iwasaki, Shintaro wrote: > Hi Phil, > > > That only works for ULTs that are literally in the pool, though, I think. > Yes, that's right. If that ULT yields, the Argobots runtime can print > the stack by either?ABT_info_print_thread_stacks_in_pool() or > ABT_info_print_thread_stack(). > > >?I don't believe that I am getting information for ULTs that are > executing > Technically, it's very challenging to dump a "running" ULT's stack > from another ULT. libunwind needs a stack pointer (in reality more > than a stack pointer), but we cannot get it from a running ULT. > For example, the following currently does not work as expected. The > Argobots runtime cannot unwind the function stack. > ? ?ABT_info_print_thread_stack(outfile, another_running_thread); // > This does not work. > We need to stop a ULT to check its stack information. > > - Complete "ABT_info_print_thread_stack" > Printing a function call stack of a running ULT using libunwind is > very challenging. The Argobots runtime needs to identify whether a ULT > is running?or not, and if it is running, it must be stopped by a > signal. This is?extremely error-prone. I don't know how to guarantee > the atomicity (e.g., two threads call this function for the same > ULT?). Stopping a specific execution stream (=Pthreads) might not be > portable across OSs, so it might need a few fallback implementations > (pthread_kill or some signal-related function). No matter Argobots > supports it or not, using a signal itself badly affects other runtimes > and applications that use Argobots (e.g., system calls fail > unexpectedly, pthread_cond_wait() will wake up, ...). Since it's stack > unwinding of the running Pthreads (i.e., nothing is ULT-specific), I > am not sure if this feature must be supported by the Argobots runtime > considering the harmfulness of the potential signal-based implementation. > > I understand the demand, and most users might feel the current > implementation is incomplete. If this feature is really needed, we > will more seriously consider the design, but please do not assume that > we can provide this implementation very soon, though finally the > priority depends on urgency and significance. > > - "ABT_self_print_thread_stack" > The following implementation is easy. We can safely stop the caller > ULT, so libunwind can print the function stack. > ?ABT_self_print_thread_stack(FILE *fp); > If this is sufficient, I can implement this quickly. To print the > running ULT's stack, basically the user can launch a signal handler on > its underlying execution stream and call this function in that signal > handler. We do not plan to guarantee async-signal safety for > ABT_self_print_thread_stack(), so if this is the way to go, please let > us know so that we will do our best to make it async-signal safe. > > Thanks, > Shintaro > > ------------------------------------------------------------------------ > *From:* Phil Carns via discuss > *Sent:* Wednesday, April 14, 2021 6:44 PM > *To:* discuss at lists.argobots.org > *Cc:* Carns, Philip H. > *Subject:* [argobots-discuss] dumping stack information for running ULTs > > Hi all, > > I followed Shintaro's suggestions from the previous list email thread > and was able to get a stack dump routine that works as expected.? For > all pools under my control I can see stack information like this for > each ULT, with stack unwinding: > > == pool (0x557d5b8c4580) == > === ULT (0x7f47429590c1) === > id??????? : 0 > ctx?????? : 0x7f4742959120 > p_ctx??? : 0x7f4742958fe0 > p_link?? : (nil) > stack???? : 0x7f47427590c0 > stacksize : 2097152 > #0 0x7f4744461760 in ythread_unwind_stack () <+16> (RSP = > 0x7f4742959010) > #1 0x7f474445e3dd in ABT_thread_yield () <+157> (RSP = 0x7f4742959020) > #2 0x7f474447e255 in __margo_hg_progress_fn () <+645> (RSP = > 0x7f4742959040) > #3 0x7f474446381e in ABTD_ythread_func_wrapper () <+30> (RSP = > 0x7f47429590b0) > #4 0x7f47444639c1 in make_fcontext () <+33> (RSP = 0x7f47429590c0) > 00007f47427590c0: 0000000000000000 0000000000000000 > 0000000000000000 0000000000000000 > 00007f47427590e0: 0000000000000000 0000000000000000 > 0000000000000000 0000000000000000 > ... > > That only works for ULTs that are literally in the pool, though, I > think.? I don't believe that I am getting information for ULTs that > are executing (and thus are not presently in a pool data structure). > > Is there any way to accomplish that? > > I tried to at least get the caller's own stack at least by doing > something like this: > > ??? ABT_thread_self(&self); > ??? ABT_info_print_thread_stack(outfile, self); > > That almost works, but I'm just getting the raw address information > and not the translated stack unwinding: > > p_ctx??? : 0x7f4742758fb8 > p_link?? : (nil) > stack???? : 0x7f4742559000 > stacksize : 2097152 > 00007f4742559000: 0000000000000000 0000000000000000 > 0000000000000000 0000000000000000 > 00007f4742559020: 0000000000000000 0000000000000000 > 0000000000000000 0000000000000000 > ... > > For completeness it would be nice if the caller's stack were also > human readable, and even better if I could somehow find stack > information for ULTs executing on other ESs as well. > > For our use case we typically spawn many detached threads into a > service pool, so we aren't tracking thread references. > > thanks, > > -Phil > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From siwasaki at anl.gov Wed Apr 21 09:39:18 2021 From: siwasaki at anl.gov (Iwasaki, Shintaro) Date: Wed, 21 Apr 2021 14:39:18 +0000 Subject: [argobots-discuss] modifying scheduler event frequency? In-Reply-To: <45d63d8e-22ad-041c-07a2-653c93c0d208@mcs.anl.gov> References: , <45d63d8e-22ad-041c-07a2-653c93c0d208@mcs.anl.gov> Message-ID: Hi Phil, Though you should have already known, I would like to tell you that: - The Argobots Spack package supports several new options including stack guard and libunwind settings (see https://github.com/spack/spack/pull/23133) - Argobots now supports mprotect-based stack guard, which causes SEGV when a ULT smashes a stack (see https://github.com/pmodels/argobots/pull/327) - This mprotect-based mechanism should work on x86/64, ARM, and POWER machines. I tried Linux (Debian/RedHat), FreeBSD, and Intel-based OSX (see https://github.com/pmodels/argobots/pull/328). If you have any requests, suggestions, or bug reports, please let us know. (I am aware of the Spack issue related to Argobots. I plan to write a quick patch tomorrow: https://github.com/spack/spack/issues/23168) Best, Shintaro ________________________________ From: Carns, Philip H. Sent: Wednesday, April 14, 2021 4:01 PM To: Iwasaki, Shintaro ; discuss at lists.argobots.org Subject: Re: [argobots-discuss] modifying scheduler event frequency? On 4/14/21 4:55 PM, Iwasaki, Shintaro wrote: Hi Phil, Thanks. I can understand a bigger picture. > ABT_info_print_thread_stacks_in_pool() I hope it works. Note that print_thread_stacks_in_pool() is not async-signal safe (ABT_info_trigger_print_all_thread_stacks() is an exception), so please don't call it in a signal handler. Ok, no problem. We don't do much via signals in Mochi (almost all of our control capabilities are triggered via RPCs that launch ULTs to do the work). > We use argobots almost exclusively with spack at this point. Many HPC users use Spack to build dependent libraries. I will add some debug options (including libunwind, stack guard, ...) as well as other major options to the Spack Argobots package. We are also implementing an mprotect-based stack guard option (which is not in Argobots 1.1, though). Overall, please give us a week or so in total. There is large room for improvement of the debugging/profiling capability. If you have any questions, requests, and/or suggestions, please feel free to tell us. Sounds great! Debuggability seems to be the next frontier for our project, so we'll probably be experimenting with more of these capabilities as time goes on. It's taken a little while for us to recognize which design/debugging patterns would be most useful. thanks, -Phil Thanks, Shintaro ________________________________ From: Carns, Philip H. Sent: Wednesday, April 14, 2021 3:42 PM To: Iwasaki, Shintaro ; discuss at lists.argobots.org Subject: Re: [argobots-discuss] modifying scheduler event frequency? Ah, thanks for the thorough information as always Shintaro :) print_all_thread_stacks() was tempting because it would potentially encompass more (in the Mochi use case, it would pick up hypothetical pools created by higher level components that we don't have a reference to). Based on the information in this email thread, though, I think I'm better off focusing on pools under our control so that I can use print_thread_stacks_in_pool(). This should work fine; I was just over-thinking the use case. The pools are under our own control in the vast majority of configurations. In the big picture, I was exploring this because of a bug report we have from one of our collaborators who is getting a nonsensical hang in a complex scenario that we can't easily reproduce or attach a debugger to. I would like to be able to send an RPC to a process at an arbitrary point in time and dump what it is up to so that we can understand why it didn't complete something it was trying to do. libunwind sounds great :) I probably would have been asking about that next. I guess I'll use this as an opportunity to request/suggest that the libunwind capability be added as a variant to the argobots spack package (along with a way to enable future mprotect / stack canary checks). We use argobots almost exclusively with spack at this point. Not that argobots itself is hard to compile manually, but it is often one of a large number of dependencies that we need to build, so it's best to just unify them in one packaging system. It would be straightforward for us to set up an alternative environment yaml with various argobots debugging capabilities enabled for development/debugging purposes. thanks! -Phil On 4/14/21 3:57 PM, Iwasaki, Shintaro wrote: Hi Phil, Thanks for using Argobots! The following is my answers to your questions in addition to some tips. We would appreciate it if you could share more information about your workload and the purpose so that we can give you more specific suggestions. Also, we welcome any feature requests and bug reports. 1. How to change a scheduler's event frequency? 1.1. Predefined scheduler First, there is no way to dynamically change the event frequency (even if you hack ABT_sched or a pointer you used in ABT_sched_get_data()... since event_freq is loaded to a local variable). https://github.com/pmodels/argobots/blob/main/src/sched/basic_wait.c#L102 Currently, using a special ABT_sched_config when you create a scheduler is the cleanest and the only way to change the event frequency. ``` ABT_sched_config config; int new_freq = 16; // The default value is 50 (https://github.com/pmodels/argobots/blob/main/src/arch/abtd_env.c#L13) ABT_sched_config_create(&config, ABT_sched_basic_freq, 16, ABT_sched_config_var_end); ``` 1.2. Custom scheduler You can call ABT_xstream_check_events() more frequently after calling ABT_info_trigger_print_all_thread_stacks() (e.g., when a global flag is on, a scheduler calls ABT_xstream_check_events() in every iteration). 2. ABT_info_trigger_print_all_thread_stacks() ABT_info_trigger_print_all_thread_stacks() is designed for deadlock/livelock detection, so if your program is just (extremely) slow, ABT_info_trigger_print_all_thread_stacks() might not be a right routine to try. > The first example I tried appeared to essentially defer dump until shutdown. When one of your ULTs encounters a deadlock, the scheduling loop might not be called. You might want to set timeout for ABT_info_trigger_print_all_thread_stacks(). For example, the following test will forcibly print stacks after 3.0 seconds even if some execution streams have not reached ABT_xstream_check_events(). https://github.com/pmodels/argobots/blob/main/test/basic/info_stackdump2.c#L30 This is dangerous (I mean, it can dump a stack of a running ULT), so Argobots does not guarantee anything but it might be helpful to understand a deadlock issue sometimes. === 3. Some tips 3.1. gdb I would use gdb if it would be available to check a deadlock/performance issue. For example, if a program looks hanging, I will attach a debugger to that process and see what's happening. 3.2. libunwind for ABT_info_trigger_print_all_thread_stacks() Unless you are an extremely skillful low-level programmer, I would recommend you enable libunwind for better understanding of stacks. By default, ABT_info_trigger_print_all_thread_stacks() dumps raw hex stack data. 3.3. "occasionally tied up in system calls" I'm not sure if it's happening in the Argobots runtime (now Argobots uses futex for synchronization on external threads), but if you are calling ABT_info_trigger_print_all_thread_stacks() in a signal handler, please be aware that system calls terminate (e.g., futex, poll, or pthread_cond_wait) if a signal hits the process. (Argobots synchronization implementation is aware of it and should not be affected by an external signal. This property is thoroughly tested: https://github.com/pmodels/argobots/blob/main/test/util/abttest.c#L245-L287) Note that the user can call ABT_info_trigger_print_all_thread_stacks() on a normal thread without any problem. It is implemented just in an async-signal safe manner. 3.4. Stack dump ABT_info_print_thread_stacks_in_pool() is a less invasive way to print stacks, especially if you know a list of pools. It prints stacks immediately. Basically, ABT_info_trigger_print_all_thread_stacks() sets a flag to call ABT_info_print_thread_stacks_in_pool() for all pools after all the execution streams stop in ABT_xstream_check_events(). Thanks, Shintaro ________________________________ From: Phil Carns via discuss Sent: Wednesday, April 14, 2021 2:18 PM To: discuss at lists.argobots.org Cc: Carns, Philip H. Subject: [argobots-discuss] modifying scheduler event frequency? Hi all, Is there a clean way to change a scheduler's event frequency on the fly? Browsing the API, I see two possibilities: * set it when the scheduler is first created (using ABT_sched_basic_freq?) * set it dynamically by manipulating the ABT_sched_get_data() pointer, but this seems especially dangerous since the sched data struct definition isn't public (i.e. it could cause memory corruption if the internal struct def changed) For some context (in case there is a different way to go about this entirely), I'm trying to figure out how to get ABT_info_trigger_print_all_thread_stacks() to print information more quickly, which IIUC relies on getting the active schedulers to call get_events() sooner. I'm happy to add some explicit ABT_thread_yield() shortly after the ABT_info_trigger_print_all_thread_stacks() to at least get the calling ES to execute it's scheduler loop immediately, but I think that won't matter much if it doesn't trip the frequency counter when I do it. Without this (at least with the _wait scheduler and threads that are occasionally tied up in system calls) I think the stack dump is likely to trigger too late to display what I'm hoping to capture when I call it. The first example I tried appeared to essentially defer dump until shutdown. thanks! -Phil -------------- next part -------------- An HTML attachment was scrubbed... URL: From carns at mcs.anl.gov Wed Apr 21 13:18:29 2021 From: carns at mcs.anl.gov (Phil Carns) Date: Wed, 21 Apr 2021 14:18:29 -0400 Subject: [argobots-discuss] modifying scheduler event frequency? In-Reply-To: References: <45d63d8e-22ad-041c-07a2-653c93c0d208@mcs.anl.gov> Message-ID: <86cce623-c009-0fc1-ae07-5482cb0689fb@mcs.anl.gov> That's all fantastic Shintaro, thank you for the updates! -Phil On 4/21/21 10:39 AM, Iwasaki, Shintaro wrote: > Hi Phil, > > Though you should have already known, I would like to tell you that: > - The Argobots Spack package supports several new options including > stack guard and libunwind settings (see > https://github.com/spack/spack/pull/23133 > ) > - Argobots now supports mprotect-based stack guard, which causes SEGV > when a ULT smashes a stack (see > https://github.com/pmodels/argobots/pull/327 > ) > - This mprotect-based mechanism should work on x86/64, ARM, and POWER > machines. I tried Linux (Debian/RedHat), FreeBSD, and Intel-based OSX > (see https://github.com/pmodels/argobots/pull/328 > ). > > If you have any requests, suggestions, or bug reports, please let us know. > (I am aware of the Spack issue related to Argobots. I plan to write a > quick patch tomorrow: https://github.com/spack/spack/issues/23168 > ) > > Best, > Shintaro > ------------------------------------------------------------------------ > *From:* Carns, Philip H. > *Sent:* Wednesday, April 14, 2021 4:01 PM > *To:* Iwasaki, Shintaro ; discuss at lists.argobots.org > > *Subject:* Re: [argobots-discuss] modifying scheduler event frequency? > > > On 4/14/21 4:55 PM, Iwasaki, Shintaro wrote: >> Hi Phil, >> >> Thanks. I can understand a bigger picture. >> >> > ABT_info_print_thread_stacks_in_pool() >> I hope it works. Note that print_thread_stacks_in_pool() is not >> async-signal safe?(ABT_info_trigger_print_all_thread_stacks() is an >> exception), so please don't call it in a signal handler. > > > Ok, no problem.? We don't do much via signals in Mochi (almost all of > our control capabilities are triggered via RPCs that launch ULTs to do > the work). > > >> >> >?We use argobots almost exclusively with spack at this point. >> Many HPC users use Spack to build dependent libraries. I will add >> some debug options (including libunwind, stack guard, ...) as well as >> other major options to the Spack Argobots package. We are also >> implementing an mprotect-based stack guard option (which is not in >> Argobots 1.1, though). >> >> Overall, please give us a week or so in total. >> >> There is large room for improvement of the debugging/profiling >> capability. >> If you have any questions, requests, and/or suggestions, please feel >> free to tell us. > > > Sounds great!? Debuggability seems to be the next frontier for our > project, so we'll probably be experimenting with more of these > capabilities as time goes on.? It's taken a little while for us to > recognize which design/debugging patterns would be most useful. > > > thanks, > > -Phil > >> >> Thanks, >> Shintaro >> >> >> ------------------------------------------------------------------------ >> *From:* Carns, Philip H. >> *Sent:* Wednesday, April 14, 2021 3:42 PM >> *To:* Iwasaki, Shintaro ; >> discuss at lists.argobots.org >> >> *Subject:* Re: [argobots-discuss] modifying scheduler event frequency? >> >> Ah, thanks for the thorough information as always Shintaro :) >> >> >> print_all_thread_stacks() was? tempting because it would potentially >> encompass more (in the Mochi use case, it would pick up hypothetical >> pools created by higher level components that we don't have a >> reference to).? Based on the information in this email thread, >> though, I think I'm better off focusing on pools under our control so >> that I can use print_thread_stacks_in_pool().? This should work fine; >> I was just over-thinking the use case.? The pools are under our own >> control in the vast majority of configurations. >> >> >> In the big picture, I was exploring this because of a bug report we >> have from one of our collaborators who is getting a nonsensical hang >> in a complex scenario that we can't easily reproduce or attach a >> debugger to.? I would like to be able to send an RPC to a process at >> an arbitrary point in time and dump what it is up to so that we can >> understand why it didn't complete something it was trying to do. >> >> >> libunwind sounds great :)? I probably would have been asking about >> that next. >> >> >> I guess I'll use this as an opportunity to request/suggest that the >> libunwind capability be added as a variant to the argobots spack >> package (along with a way to enable future mprotect / stack canary >> checks). >> >> >> We use argobots almost exclusively with spack at this point.? Not >> that argobots itself is hard to compile manually, but it is often one >> of a large number of dependencies that we need to build, so it's best >> to just unify them in one packaging system.? It would be >> straightforward for us to set up an alternative environment yaml with >> various argobots debugging capabilities enabled for >> development/debugging purposes. >> >> >> thanks! >> >> -Phil >> >> >> On 4/14/21 3:57 PM, Iwasaki, Shintaro wrote: >>> Hi Phil, >>> >>> Thanks for using Argobots! ?The following is my answers to your >>> questions in addition to some tips. >>> We would appreciate it if you could share more information about >>> your workload and the purpose so that we can give you more specific >>> suggestions. Also, we welcome any feature requests and bug reports. >>> >>> 1. How to change a scheduler's event frequency? >>> 1.1. Predefined scheduler >>> First, there is no way to dynamically change the event frequency >>> (even if you hack ABT_sched or a pointer you used in >>> ABT_sched_get_data()... since event_freq is loaded to a local variable). >>> https://github.com/pmodels/argobots/blob/main/src/sched/basic_wait.c#L102 >>> >>> Currently, using a special ABT_sched_config when you create a >>> scheduler is the cleanest and the only way to change the event >>> frequency. >>> ``` >>> ABT_sched_config config; >>> int new_freq = 16; // The default value is 50 >>> (https://github.com/pmodels/argobots/blob/main/src/arch/abtd_env.c#L13 >>> ) >>> ABT_sched_config_create(&config, ABT_sched_basic_freq, 16, >>> ABT_sched_config_var_end); >>> ``` >>> 1.2. Custom scheduler >>> You can call ABT_xstream_check_events() more frequently after >>> calling ABT_info_trigger_print_all_thread_stacks() (e.g., when a >>> global flag is on, a scheduler calls ABT_xstream_check_events() in >>> every iteration). >>> >>> 2. ABT_info_trigger_print_all_thread_stacks() >>> ABT_info_trigger_print_all_thread_stacks() is designed for >>> deadlock/livelock detection, so if your program is just (extremely) >>> slow, ABT_info_trigger_print_all_thread_stacks() might not be a >>> right routine to try. >>> >>> > The first example I tried appeared to essentially defer dump until >>> shutdown. >>> When one of your ULTs encounters a deadlock, the scheduling loop >>> might not be called. You might want to set timeout for >>> ABT_info_trigger_print_all_thread_stacks(). For example, the >>> following test will forcibly print stacks after 3.0 seconds even if >>> some execution streams have not reached ABT_xstream_check_events(). >>> https://github.com/pmodels/argobots/blob/main/test/basic/info_stackdump2.c#L30 >>> >>> This is dangerous (I mean, it can dump a stack of a running ULT), so >>> Argobots does not guarantee anything but it might be helpful to >>> understand a deadlock issue sometimes. >>> >>> === >>> >>> 3. Some tips >>> 3.1. gdb >>> I would use gdb if it would be available to check a >>> deadlock/performance issue. For example, if a program looks hanging, >>> I will attach a debugger to that process and see what's happening. >>> 3.2. libunwind for ABT_info_trigger_print_all_thread_stacks() >>> Unless you are an extremely skillful low-level programmer, I would >>> recommend you enable libunwind for better understanding of stacks. >>> By default, ABT_info_trigger_print_all_thread_stacks() dumps raw hex >>> stack data. >>> 3.3. "occasionally tied up in system calls" >>> I'm not sure if it's happening in the Argobots runtime (now Argobots >>> uses futex for synchronization on external threads), but if you are >>> calling ABT_info_trigger_print_all_thread_stacks() in a signal >>> handler, please be aware that system calls terminate (e.g., futex, >>> poll, or pthread_cond_wait) if a signal hits the process. >>> (Argobots synchronization implementation is aware of it and should >>> not be affected by an external signal. This property is thoroughly >>> tested: >>> https://github.com/pmodels/argobots/blob/main/test/util/abttest.c#L245-L287 >>> ) >>> Note that the user can call >>> ABT_info_trigger_print_all_thread_stacks() on a normal thread >>> without any problem. It is implemented just in an async-signal safe >>> manner. >>> 3.4. Stack dump >>> ABT_info_print_thread_stacks_in_pool() is a less invasive way to >>> print stacks, especially if you know a list of pools. It prints >>> stacks immediately. Basically, >>> ABT_info_trigger_print_all_thread_stacks() sets a flag to call >>> ABT_info_print_thread_stacks_in_pool() for all pools after all the >>> execution streams stop in ABT_xstream_check_events(). >>> >>> Thanks, >>> Shintaro >>> >>> ------------------------------------------------------------------------ >>> *From:* Phil Carns via discuss >>> >>> *Sent:* Wednesday, April 14, 2021 2:18 PM >>> *To:* discuss at lists.argobots.org >>> >>> *Cc:* Carns, Philip H. >>> *Subject:* [argobots-discuss] modifying scheduler event frequency? >>> >>> Hi all, >>> >>> Is there a clean way to change a scheduler's event frequency on the fly? >>> >>> Browsing the API, I see two possibilities: >>> >>> * set it when the scheduler is first created (using >>> ABT_sched_basic_freq?) >>> * set it dynamically by manipulating the ABT_sched_get_data() >>> pointer, but this seems especially dangerous since the sched >>> data struct definition isn't public (i.e. it could cause memory >>> corruption if the internal struct def changed) >>> >>> For some context (in case there is a different way to go about this >>> entirely), I'm trying to figure out how to get >>> ABT_info_trigger_print_all_thread_stacks() to print information more >>> quickly, which IIUC relies on getting the active schedulers to call >>> get_events() sooner. >>> >>> I'm happy to add some explicit ABT_thread_yield() shortly after the >>> ABT_info_trigger_print_all_thread_stacks() to at least get the >>> calling ES to execute it's scheduler loop immediately, but I think >>> that won't matter much if it doesn't trip the frequency counter when >>> I do it. >>> >>> Without this (at least with the _wait scheduler and threads that are >>> occasionally tied up in system calls) I think the stack dump is >>> likely to trigger too late to display what I'm hoping to capture >>> when I call it. The first example I tried appeared to essentially >>> defer dump until shutdown. >>> >>> thanks! >>> >>> -Phil >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: