<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255); display: inline !important">Hello All,</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
<br>
</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
Thank you for your reports!</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
<br>
</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
As one of the developers, I would like to summarize the current status of Argobots.</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
As pointed out, Argobots, by default, uses 16KB for ULTs.</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
At present, these three ways are relatively reasonable to work around, find, or solve this issue;</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
0. Use knowledge that, if Argobots is used alone, it should "typically" happens in ABT_finalize. The stack size is set by ABT_thread_attr_set_stacksize individually.</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
1. Use larger stacksize (e.g., ABT_THREAD_STACKSIZE=$((4 * 1024 * 1024)) for example) and see what happens.</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
2. [Uncertain] Use Valgrind with --enable-valgrind (although it is extremely slow, so not practical for large applications)</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
For now, I think the workaround 1. (using larger stack size by default) is best among them.</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
<br>
</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
I haven't tried other tools, but I strongly believe that Argobots-unaware tools won't detect this problem; only Valgind can detect it.</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
<br>
</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
---</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
<br>
</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
There are several issues:</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
1. Too small default stack size</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
It might be too small to drive large system libraries (e.g., a ULT as a progress thread)</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
I'm not sure how much it should be increased, or first of all, whether we should increase it.</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
This does not solve the problem of the "silent stack corruption", though.</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
In other words, if Argobots can detect stack overflow, users can change the value</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
by increasing the default stack size <span style="color: rgb(34, 34, 34); font-family: Arial, Helvetica, sans-serif; font-size: small;">or the stack size of a specific thread requiring large amount of stack.</span></div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
<br>
</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
2. Lack of stack overflow detection<br>
</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
<div style="margin: 0px">For example, the following two are often used;</div>
<div style="margin: 0px">- Stack canaries (lazy but cheap)</div>
<div style="margin: 0px">- mprotect (eager but expensive)</div>
<div style="margin: 0px">I will create a GitHub issue for further details if detection is preferable.</div>
<div style="margin: 0px"><br>
</div>
<div style="margin: 0px">3. Check if Valgrind works for this issue</div>
<div style="margin: 0px">If --enable-valgrind is set, Argobots registers <span style="font-family: Arial, Helvetica, sans-serif; font-size: small; background-color: rgb(255, 255, 255); display: inline !important">
ULT's stacks to </span>Valgrind.</div>
<div style="margin: 0px">It should work but I haven't tested it yet.</div>
<div style="margin: 0px"><span style="font-family: Arial, Helvetica, sans-serif; font-size: small; background-color: rgb(255, 255, 255); display: inline !important">Another problem is that --enable-valgrind degrades performance of Argobots even if it is not
run on Valgrind</span></div>
<div style="margin: 0px"><span style="font-family: Arial, Helvetica, sans-serif; font-size: small; background-color: rgb(255, 255, 255); display: inline !important">(see
<a href="https://github.com/pmodels/argobots/issues/78">https://github.com/pmodels/argobots/issues/78</a>).</span></div>
<div style="margin: 0px"><br>
</div>
<div style="margin: 0px">We would appreciate any feedback.<br>
</div>
<div style="margin: 0px"><br>
</div>
</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
Thank You,</div>
<div style="margin: 0px; font-size: small; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255)">
Shintaro Iwasaki</div>
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Lombardi, Johann via discuss <discuss@lists.argobots.org><br>
<b>Sent:</b> Thursday, February 21, 2019 4:30 PM<br>
<b>To:</b> discuss@lists.argobots.org<br>
<b>Cc:</b> Lombardi, Johann; Liu, Xuezhao; Wang, Di<br>
<b>Subject:</b> Re: [argobots-discuss] how to debug a stack overrun in Argobots</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
code
{font-family:"Courier New"}
p.x_msonormal0, li.x_msonormal0, div.x_msonormal0
{margin-right:0cm;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle20
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:70.85pt 70.85pt 70.85pt 70.85pt}
div.x_WordSection1
{}
-->
</style>
<div lang="FR" link="blue" vlink="purple">
<div class="x_WordSection1">
<p class="x_MsoNormal"><span lang="EN-US">Hi Phil,</span></p>
<p class="x_MsoNormal"><span lang="EN-US"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US">I think we hit the same issue recently on the DAOS side and had to bump the stack size as well. Wangdi & Xuezhao should know more.</span></p>
<p class="x_MsoNormal"><span lang="EN-US">Maybe a regression in ABT?<br>
<br>
Cheers,</span></p>
<p class="x_MsoNormal"><span lang="EN-US">Johann</span></p>
<p class="x_MsoNormal"><span lang="EN-US"> </span></p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_MsoNormal" style="margin-left:36.0pt"><b><span style="font-size:12.0pt; color:black">From:
</span></b><span style="font-size:12.0pt; color:black">"Carns, Philip H. via discuss" <discuss@lists.argobots.org><br>
<b>Reply-To: </b>"discuss@lists.argobots.org" <discuss@lists.argobots.org><br>
<b>Date: </b>Thursday, 21 February 2019 at 15:50<br>
<b>To: </b>"discuss@lists.argobots.org" <discuss@lists.argobots.org><br>
<b>Cc: </b>"Carns, Philip H." <carns@mcs.anl.gov><br>
<b>Subject: </b>Re: [argobots-discuss] how to debug a stack overrun in Argobots</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt"> </p>
</div>
<div>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt"><br>
Just to follow up a little bit; I realized from looking at README.envvar just now that the default value of ABT_THREAD_STACKSIZE is 16K. That's almost certainly too low for us because we have ULTs that make calls into a variety of system libraries (including
fairly big things like libfabric) that are beyond our control.<br>
<br>
It seems likely that we will have to run with a larger stack size, but I would still like to have a better understanding of where the problem paths are, and how much head room we really need, if anyone has suggestions.<br>
<br>
thanks!<br>
-Phil</p>
</div>
</div>
<p class="x_MsoNormal" style="margin-left:36.0pt"> </p>
<div>
<p style="margin-left:36.0pt">On 2019-02-21 15:31:53-05:00 Carns, Philip H. via discuss wrote:</p>
<blockquote style="border:none; border-left:solid #CCCCCC 1.0pt; padding:0cm 0cm 0cm 8.0pt; margin-left:0cm; margin-right:0cm">
<div>
<div>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt">Hi all, <br>
<br>
There is a little bit of back story on <a href="https://github.com/pmodels/argobots/issues/93">https://github.com/pmodels/argobots/issues/93</a> , but make a long story short we have realized that we have some code that is overflowing the stack in Argobots.
Many thanks to Shintaro for his help and insight or we may have never figured this out. We can work around the problem with `export <code><span style="font-size:10.0pt">ABT_THREAD_STACKSIZE=$((1024 * 1024))`. This not only fixes a Power8 test case for us,
but also appears to solve a different frustrating, nonsensical segmentation fault that we've been chasing with a different code permutation on x86_64.</span></code><span style="font-size:10.0pt; font-family:"Courier New""><br>
<br>
<code>Any suggestions on how to track down what's triggering this in our code or get a better idea of how much stack we need? </code></span><span style="font-family:"Courier New"">We are using a considerable number of libraries, many of which are not maintained
by us, so I don't even know where to start looking yet. </span>My usual go to tool for this would be asan in gcc or clang, but I don't think that will work correctly with Argobots, and maybe there is a better solution anyway. </p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt"><br>
thanks,<br>
-Phil</p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
<p>---------------------------------------------------------------------<br>
Intel Corporation SAS (French simplified joint stock company)<br>
Registered headquarters: "Les Montalets"- 2, rue de Paris, <br>
92196 Meudon Cedex, France<br>
Registration Number: 302 456 199 R.C.S. NANTERRE<br>
Capital: 4,572,000 Euros</p>
<p>This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.</p>
</div>
</body>
</html>