<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Phil,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="margin:0px;font-size:12pt">> That only works for ULTs that are literally in the pool, though, I think.</span><br>
<span style="margin:0px;font-size:12pt"></span>Yes, that's right. If that ULT yields, the Argobots runtime can print the stack by either ABT_info_print_thread_stacks_in_pool() or <span style="background-color:rgb(255, 255, 255);display:inline !important">ABT_info_print_thread_stack().</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> I don't believe that I am getting information for ULTs that are executing</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Technically, it's very challenging to dump a "running" ULT's stack from another ULT. libunwind needs a stack pointer (in reality more than a stack pointer), but we cannot get it from a running ULT.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">For example, the following currently does not work as expected. The Argobots runtime cannot unwind the function stack.</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div> ABT_info_print_thread_stack(outfile, another_running_thread); // This does not work.</div>
<div><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">We need to stop a ULT to check its stack information.</span><br>
</div>
<div><br>
</div>
<div><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="margin:0px;background-color:rgb(255, 255, 255)"><span style="margin:0px;background-color:rgb(255, 255, 255)">- Complete "<span style="margin:0px;background-color:rgb(255, 255, 255);display:inline !important">ABT_info_print_thread_stack</span>"</span></span></span></div>
<div><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="margin:0px;background-color:rgb(255, 255, 255)">Printing a function call stack of a running ULT using libunwind is very challenging. The
Argobots runtime needs to identify whether a ULT is <span style="background-color:rgb(255, 255, 255);display:inline !important">
running<span> or not, and if it is running</span></span>, it must be stopped by a signal. This is extremely error-prone. I don't know how to guarantee the atomicity (e.g., two threads call this function for the same ULT?).
</span></span>Stopping a specific execution stream (=Pthreads) might not be portable across OSs, so it might need a few fallback implementations (pthread_kill or some signal-related function). No matter Argobots supports it or not, using a signal itself badly
affects other runtimes and applications that use Argobots (e.g., system calls fail unexpectedly, pthread_cond_wait() will wake up, ...). Since it's stack unwinding of the running Pthreads (i.e., nothing is ULT-specific), I am not sure if this feature must
be supported by the Argobots runtime considering the harmfulness of the potential signal-based implementation.</div>
<div><br>
</div>
<div>I understand the demand, and most users might feel the current implementation is incomplete. If this feature is really needed, we will more seriously consider the design, but please do not assume that we can provide this implementation very soon, though
finally the priority depends on urgency and significance.</div>
<div><br>
</div>
<div><span style="margin:0px;background-color:rgb(255, 255, 255)">- "<span style="margin:0px;background-color:rgb(255, 255, 255);display:inline !important">ABT_self_print_thread_stack</span>"</span>
<div style="margin:0px;background-color:rgb(255, 255, 255)"><span style="margin:0px;font-size:12pt">The following implementation is easy. We can safely stop the caller ULT, so libunwind can print the function stack.</span></div>
<div style="margin:0px;background-color:rgb(255, 255, 255)"><span style="margin:0px;font-size:12pt"><span style="margin:0px;background-color:rgb(255, 255, 255)"> ABT_self_print_thread_stack(FILE *fp);</span><br>
</span></div>
<div style="margin:0px;background-color:rgb(255, 255, 255)"><span style="margin:0px;font-size:12pt"><span style="margin:0px;background-color:rgb(255, 255, 255)">If this is sufficient, I can implement this quickly. To print the running ULT's stack, basically
the user can launch a signal handler on its underlying execution stream and call this function in that signal handler. We do not plan to guarantee <span style="background-color:rgb(255, 255, 255);display:inline !important">async-signal safety for </span><span style="background-color:rgb(255, 255, 255);display:inline !important">ABT_self_print_thread_stack(),
so if this is the way to go, please let us know so that we will do our best to make it
<span style="background-color:rgb(255, 255, 255);display:inline !important">async-signal safe</span>.</span></span></span></div>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Shintaro</div>
<div><br>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Phil Carns via discuss <discuss@lists.argobots.org><br>
<b>Sent:</b> Wednesday, April 14, 2021 6:44 PM<br>
<b>To:</b> discuss@lists.argobots.org <discuss@lists.argobots.org><br>
<b>Cc:</b> Carns, Philip H. <carns@mcs.anl.gov><br>
<b>Subject:</b> [argobots-discuss] dumping stack information for running ULTs</font>
<div> </div>
</div>
<div>
<p>Hi all,</p>
<p>I followed Shintaro's suggestions from the previous list email thread and was able to get a stack dump routine that works as expected. For all pools under my control I can see stack information like this for each ULT, with stack unwinding:</p>
<blockquote>
<p><font face="monospace">== pool (0x557d5b8c4580) ==<br>
=== ULT (0x7f47429590c1) ===<br>
id : 0<br>
ctx : 0x7f4742959120<br>
p_ctx : 0x7f4742958fe0<br>
p_link : (nil)<br>
stack : 0x7f47427590c0<br>
stacksize : 2097152<br>
#0 0x7f4744461760 in ythread_unwind_stack () <+16> (RSP = 0x7f4742959010)<br>
#1 0x7f474445e3dd in ABT_thread_yield () <+157> (RSP = 0x7f4742959020)<br>
#2 0x7f474447e255 in __margo_hg_progress_fn () <+645> (RSP = 0x7f4742959040)<br>
#3 0x7f474446381e in ABTD_ythread_func_wrapper () <+30> (RSP = 0x7f47429590b0)<br>
#4 0x7f47444639c1 in make_fcontext () <+33> (RSP = 0x7f47429590c0)<br>
00007f47427590c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000<br>
00007f47427590e0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000<br>
...</font></p>
</blockquote>
<p>That only works for ULTs that are literally in the pool, though, I think. I don't believe that I am getting information for ULTs that are executing (and thus are not presently in a pool data structure).</p>
<p>Is there any way to accomplish that?</p>
<p>I tried to at least get the caller's own stack at least by doing something like this:</p>
<p><font face="monospace"> ABT_thread_self(&self);<br>
ABT_info_print_thread_stack(outfile, self);</font><br>
</p>
<p>That almost works, but I'm just getting the raw address information and not the translated stack unwinding:</p>
<blockquote>
<p><font face="monospace">p_ctx : 0x7f4742758fb8<br>
p_link : (nil)<br>
stack : 0x7f4742559000<br>
stacksize : 2097152<br>
00007f4742559000: 0000000000000000 0000000000000000 0000000000000000 0000000000000000<br>
00007f4742559020: 0000000000000000 0000000000000000 0000000000000000 0000000000000000<br>
...</font></p>
</blockquote>
<p>For completeness it would be nice if the caller's stack were also human readable, and even better if I could somehow find stack information for ULTs executing on other ESs as well.</p>
<p>For our use case we typically spawn many detached threads into a service pool, so we aren't tracking thread references.</p>
<p>thanks,</p>
<p>-Phil<br>
</p>
<p><br>
</p>
</div>
</body>
</html>