<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Hi Shintaro,</p>
<p><br>
</p>
<p>Thanks for the explanation, that all makes sense.</p>
<p><br>
</p>
<p>In a perfect world it would be nice to display stack information
from running ULTs, but it's not worth incurring the technical debt
to pursue any of those methods to collect it right now.<br>
</p>
<p><br>
</p>
<p>I'll stick with the stack traces of ULTs in the pool for our use
case (in conjunction with information that I can report from our
own explicit tracking). This discussion was very helpful so that
I know how to label the data clearly, though.<br>
</p>
<p><br>
</p>
<p>thanks!</p>
<p>-Phil<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 4/14/21 9:02 PM, Iwasaki, Shintaro
wrote:<br>
</div>
<blockquote type="cite" cite="mid:DM6PR09MB57505259A03FFB485C9D007AD54D9@DM6PR09MB5750.namprd09.prod.outlook.com">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Hi Phil,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<span style="margin:0px;font-size:12pt">> That only works for
ULTs that are literally in the pool, though, I think.</span><br>
<span style="margin:0px;font-size:12pt"></span>Yes, that's
right. If that ULT yields, the Argobots runtime can print the
stack by either ABT_info_print_thread_stacks_in_pool() or <span style="background-color:rgb(255, 255, 255);display:inline
!important">ABT_info_print_thread_stack().</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
> I don't believe that I am getting information for ULTs that
are executing</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Technically, it's very challenging to dump a "running" ULT's
stack from another ULT. libunwind needs a stack pointer (in
reality more than a stack pointer), but we cannot get it from a
running ULT.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial,
Helvetica, sans-serif; font-size: 12pt;">For example, the
following currently does not work as expected. The Argobots
runtime cannot unwind the function stack.</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<div> ABT_info_print_thread_stack(outfile,
another_running_thread); // This does not work.</div>
<div><span style="color: rgb(0, 0, 0); font-family: Calibri,
Arial, Helvetica, sans-serif; font-size: 12pt;">We need to
stop a ULT to check its stack information.</span><br>
</div>
<div><br>
</div>
<div><span style="color: rgb(0, 0, 0); font-family: Calibri,
Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="margin:0px;background-color:rgb(255, 255, 255)"><span style="margin:0px;background-color:rgb(255, 255, 255)">-
Complete "<span style="margin:0px;background-color:rgb(255, 255,
255);display:inline !important">ABT_info_print_thread_stack</span>"</span></span></span></div>
<div><span style="color: rgb(0, 0, 0); font-family: Calibri,
Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="margin:0px;background-color:rgb(255, 255, 255)">Printing
a function call stack of a running ULT using libunwind is
very challenging. The Argobots runtime needs to identify
whether a ULT is <span style="background-color:rgb(255,
255, 255);display:inline !important">
running<span> or not, and if it is running</span></span>,
it must be stopped by a signal. This is extremely
error-prone. I don't know how to guarantee the atomicity
(e.g., two threads call this function for the same ULT?).
</span></span>Stopping a specific execution stream
(=Pthreads) might not be portable across OSs, so it might need
a few fallback implementations (pthread_kill or some
signal-related function). No matter Argobots supports it or
not, using a signal itself badly affects other runtimes and
applications that use Argobots (e.g., system calls fail
unexpectedly, pthread_cond_wait() will wake up, ...). Since
it's stack unwinding of the running Pthreads (i.e., nothing is
ULT-specific), I am not sure if this feature must be supported
by the Argobots runtime considering the harmfulness of the
potential signal-based implementation.</div>
<div><br>
</div>
<div>I understand the demand, and most users might feel the
current implementation is incomplete. If this feature is
really needed, we will more seriously consider the design, but
please do not assume that we can provide this implementation
very soon, though finally the priority depends on urgency and
significance.</div>
<div><br>
</div>
<div><span style="margin:0px;background-color:rgb(255, 255,
255)">- "<span style="margin:0px;background-color:rgb(255,
255, 255);display:inline !important">ABT_self_print_thread_stack</span>"</span>
<div style="margin:0px;background-color:rgb(255, 255, 255)"><span style="margin:0px;font-size:12pt">The following
implementation is easy. We can safely stop the caller ULT,
so libunwind can print the function stack.</span></div>
<div style="margin:0px;background-color:rgb(255, 255, 255)"><span style="margin:0px;font-size:12pt"><span style="margin:0px;background-color:rgb(255, 255, 255)">
ABT_self_print_thread_stack(FILE *fp);</span><br>
</span></div>
<div style="margin:0px;background-color:rgb(255, 255, 255)"><span style="margin:0px;font-size:12pt"><span style="margin:0px;background-color:rgb(255, 255, 255)">If
this is sufficient, I can implement this quickly. To
print the running ULT's stack, basically the user can
launch a signal handler on its underlying execution
stream and call this function in that signal handler. We
do not plan to guarantee <span style="background-color:rgb(255, 255,
255);display:inline !important">async-signal safety
for </span><span style="background-color:rgb(255, 255,
255);display:inline !important">ABT_self_print_thread_stack(),
so if this is the way to go, please let us know so
that we will do our best to make it
<span style="background-color:rgb(255, 255,
255);display:inline !important">async-signal safe</span>.</span></span></span></div>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Shintaro</div>
<div><br>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Phil
Carns via discuss <a class="moz-txt-link-rfc2396E" href="mailto:discuss@lists.argobots.org"><discuss@lists.argobots.org></a><br>
<b>Sent:</b> Wednesday, April 14, 2021 6:44 PM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:discuss@lists.argobots.org">discuss@lists.argobots.org</a>
<a class="moz-txt-link-rfc2396E" href="mailto:discuss@lists.argobots.org"><discuss@lists.argobots.org></a><br>
<b>Cc:</b> Carns, Philip H. <a class="moz-txt-link-rfc2396E" href="mailto:carns@mcs.anl.gov"><carns@mcs.anl.gov></a><br>
<b>Subject:</b> [argobots-discuss] dumping stack information
for running ULTs</font>
<div> </div>
</div>
<div>
<p>Hi all,</p>
<p>I followed Shintaro's suggestions from the previous list
email thread and was able to get a stack dump routine that
works as expected. For all pools under my control I can see
stack information like this for each ULT, with stack
unwinding:</p>
<blockquote>
<p><font face="monospace">== pool (0x557d5b8c4580) ==<br>
=== ULT (0x7f47429590c1) ===<br>
id : 0<br>
ctx : 0x7f4742959120<br>
p_ctx : 0x7f4742958fe0<br>
p_link : (nil)<br>
stack : 0x7f47427590c0<br>
stacksize : 2097152<br>
#0 0x7f4744461760 in ythread_unwind_stack () <+16>
(RSP = 0x7f4742959010)<br>
#1 0x7f474445e3dd in ABT_thread_yield () <+157> (RSP
= 0x7f4742959020)<br>
#2 0x7f474447e255 in __margo_hg_progress_fn ()
<+645> (RSP = 0x7f4742959040)<br>
#3 0x7f474446381e in ABTD_ythread_func_wrapper ()
<+30> (RSP = 0x7f47429590b0)<br>
#4 0x7f47444639c1 in make_fcontext () <+33> (RSP =
0x7f47429590c0)<br>
00007f47427590c0: 0000000000000000 0000000000000000
0000000000000000 0000000000000000<br>
00007f47427590e0: 0000000000000000 0000000000000000
0000000000000000 0000000000000000<br>
...</font></p>
</blockquote>
<p>That only works for ULTs that are literally in the pool,
though, I think. I don't believe that I am getting
information for ULTs that are executing (and thus are not
presently in a pool data structure).</p>
<p>Is there any way to accomplish that?</p>
<p>I tried to at least get the caller's own stack at least by
doing something like this:</p>
<p><font face="monospace"> ABT_thread_self(&self);<br>
ABT_info_print_thread_stack(outfile, self);</font><br>
</p>
<p>That almost works, but I'm just getting the raw address
information and not the translated stack unwinding:</p>
<blockquote>
<p><font face="monospace">p_ctx : 0x7f4742758fb8<br>
p_link : (nil)<br>
stack : 0x7f4742559000<br>
stacksize : 2097152<br>
00007f4742559000: 0000000000000000 0000000000000000
0000000000000000 0000000000000000<br>
00007f4742559020: 0000000000000000 0000000000000000
0000000000000000 0000000000000000<br>
...</font></p>
</blockquote>
<p>For completeness it would be nice if the caller's stack were
also human readable, and even better if I could somehow find
stack information for ULTs executing on other ESs as well.</p>
<p>For our use case we typically spawn many detached threads
into a service pool, so we aren't tracking thread references.</p>
<p>thanks,</p>
<p>-Phil<br>
</p>
<p><br>
</p>
</div>
</blockquote>
</body>
</html>