<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Matthieu,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for your quick update.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin:0px">If a pool created by ABT_pool_create() is associated with the primary execution stream (and thus automatic = false), you do not need to free it explicitly. ABT_finalize() will free everything that is associated with the primary stream.
If you use Argobots 1.0, doing so for a non-automatic pool might cause memory leak, though.<br>
</div>
<div style="margin:0px"><br>
</div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> I think it would be good if Argobots allowed pools created by ABT_pool_create to have an “automatic” flag as well, and be free automatically. Otherwise I’m not even sure how we are supposed to free the pool associated with the primary ES if that pool happens
to be a custom one.
<div><br>
</div>
<div>Yes, I fully agree. If I could redesign all functions from scratch, ABT_pool_create() would have an automatic argument. Unfortunately, Argobots 1.0 and 1.1 do not have such a configuration option, so please maintain flags externally to selectively free
only pools that are created by ABT_pool_create() if necessary. We will extend ABT_pool_config to support automatic configuration in the future.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
---</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
If your program uses a user-defined pool, please check <a href="https://github.com/pmodels/argobots/issues/316">https://github.com/pmodels/argobots/issues/316</a>.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This issue should be fixed somehow before Argobots 1.1, which is planned to be released soon (within one or two weeks).</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Shintaro</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Dorier, Matthieu <mdorier@anl.gov><br>
<b>Sent:</b> Tuesday, March 23, 2021 9:55 AM<br>
<b>To:</b> Iwasaki, Shintaro <siwasaki@anl.gov>; discuss@argobots.org <discuss@argobots.org><br>
<b>Subject:</b> Re: Program aborting in ABT_finalize (ABTI_pool_release failed)</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:blue;
text-decoration:underline}
p.x_xmsonormal, li.x_xmsonormal, div.x_xmsonormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
p.x_xxmsonormal, li.x_xxmsonormal, div.x_xxmsonormal
{margin:0cm;
font-size:12.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle24
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:72.0pt 72.0pt 72.0pt 72.0pt}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal"><span lang="EN-US" style="">Hi Shintaro,</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="">I think I’m narrowing down the problem (I’m testing right now). We initially had all our pools created with automatic=TRUE in Margo, and weren’t calling ABT_pool_free anywhere. But a recent commit from Phil
enabled the use of a custom pool implementation, created with ABT_pool_create. The problem with these pools is that there is no “automatic” flag, so he changed to automatic=FALSE for the basic pools so he could call ABT_pool_free on all the pools, however
not keeping track nor checking that a pool was associated with the primary ES.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="">I think it would be good if Argobots allowed pools created by ABT_pool_create to have an “automatic” flag as well, and be free automatically. Otherwise I’m not even sure how we are supposed to free the pool
associated with the primary ES if that pool happens to be a custom one.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="">Thanks,</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="">Matthieu</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_MsoNormal"><b><span style="font-size:12.0pt; color:black">From: </span>
</b><span style="font-size:12.0pt; color:black">"Iwasaki, Shintaro" <siwasaki@anl.gov><br>
<b>Date: </b>Tuesday, 23 March 2021 at 14:44<br>
<b>To: </b>"Dorier, Matthieu" <mdorier@anl.gov><br>
<b>Subject: </b>Re: Program aborting in ABT_finalize (ABTI_pool_release failed)</span></p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Hi Matthieu,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Even if automatic is false, please do not call ABT_pool_free() for the main ES's pool. It must be automatically freed by ABT_finalize().</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black; background:white">Argobots 1.0 "works" though a few hundred bytes will be leaked because of its bug </span><span style="font-size:12.0pt; color:black">(a few hundred bytes per pool on ABT_finalize()
as far as I checked with Valgrind).</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Argobots 1.1 (or the Argobots main branch) has fixed this memory leak issue, so I would recommend you use the latest one if possible.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">> Argobots isn’t checking whether a pool is in use before freeing it whether it’s used by the first ES.</span></p>
</div>
<div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Currently Argobots checks neither of them, but it should be added to ease debugging.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">I created an issue: <a href="https://github.com/pmodels/argobots/issues/315">https://github.com/pmodels/argobots/issues/315</a></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Thanks,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Shintaro</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Dorier, Matthieu <mdorier@anl.gov><br>
<b>Sent:</b> Tuesday, March 23, 2021 9:20 AM<br>
<b>To:</b> Iwasaki, Shintaro <siwasaki@anl.gov>; discuss@argobots.org <discuss@argobots.org>; discuss@lists.argobots.org <discuss@lists.argobots.org><br>
<b>Subject:</b> Re: Program aborting in ABT_finalize (ABTI_pool_release failed)</span>
</p>
<div>
<p class="x_MsoNormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xmsonormal"><span lang="EN-US">I’m creating the pool using </span>ABT_pool_create_basic and “automatic” set to false, and I do indeed call ABT_pool_free on it.</p>
<p class="x_xmsonormal">I guess Argobots isn’t checking whether a pool is in use before freeing it (or whether it’s used by the first ES)?</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">My code is pretty complex so I tried to write a reproducer but I don’t see the error with it. Maybe I’m corrupting memory, somehow, and it just happens to be fine in small code that don’t reuse the memory after the ABT_pool_free… (though
I ran address sanitizer and didn’t see any invalid access).</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">I’ll see what happens if I remove the ABT_pool_free.</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Thanks,</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Matthieu</p>
<p class="x_xmsonormal"><span lang="EN-US"> </span></p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_xmsonormal"><b><span style="font-size:12.0pt; color:black">From: </span>
</b><span style="font-size:12.0pt; color:black">"Iwasaki, Shintaro" <siwasaki@anl.gov><br>
<b>Date: </b>Tuesday, 23 March 2021 at 14:01<br>
<b>To: </b>"discuss@argobots.org" <discuss@argobots.org>, "discuss@lists.argobots.org" <discuss@lists.argobots.org><br>
<b>Cc: </b>"Dorier, Matthieu" <mdorier@anl.gov><br>
<b>Subject: </b>Re: Program aborting in ABT_finalize (ABTI_pool_release failed)</span></p>
</div>
<div>
<p class="x_xmsonormal"> </p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Hi Matthieu,</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">I don't know when such an error typically happens. I'd be happy if you could share your code with me so that I can closely look at it.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">This is just my guess, but perhaps ABT_pool_free() is used; in Argobots, primary ES's pool and schedulers may not be freed before ABT_finalize().</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">If you free that fifo_wait MPMC pool before ABT_finalize(), an already freed pool is accessed in ABT_finalize(), which may cause the error you wrote.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">(It is related to
<a href="https://github.com/pmodels/argobots/issues/58">https://github.com/pmodels/argobots/issues/58</a>).</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">I attached a sample code to explain this issue. Please read the comments in the code if you are interested in it.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Thanks,</span></p>
</div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Shintaro</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_divRplyFwdMsg">
<p class="x_xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Dorier, Matthieu via discuss <discuss@lists.argobots.org><br>
<b>Sent:</b> Tuesday, March 23, 2021 6:56 AM<br>
<b>To:</b> discuss@argobots.org <discuss@argobots.org><br>
<b>Cc:</b> Dorier, Matthieu <mdorier@anl.gov><br>
<b>Subject:</b> [argobots-discuss] Program aborting in ABT_finalize (ABTI_pool_release failed)</span>
</p>
<div>
<p class="x_xmsonormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">Hi,</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">I’m getting this error in a program, when calling ABT_finalize:</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">../src/include/abti_pool.h:238: ABTI_pool_release: Assertion `ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0' failed.</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">The program only uses the primary ES, it does not create additional ES, however it does replace the primary ES’s pool and scheduler by creating a fifo_wait MPMC pool and passing it to ABT_xstream_set_main_sched_basic,
with </span><span style="font-size:11.0pt">ABT_SCHED_BASIC_WAIT as sched_predef.</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">I’m using Argobots 1.0. I haven’t tested with other versions of Argobots. I’m sure I must be doing something wrong somewhere in my code. Could you tell me what I could look for that would
lead to such an error?</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">For additional information, here it the stack trace when the error happens:</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#1 0x00007f56797fe535 in __GI_abort () at abort.c:79</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#2 0x00007f56797fe40f in __assert_fail_base (fmt=0x7f5679960ee0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> assertion=0x7f5679d0e4c8 "ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0", file=0x7f5679d0e44b "../src/include/abti_pool.h", line=238,</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> function=<optimized out>) at assert.c:92</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#3 0x00007f567980c102 in __GI___assert_fail (assertion=0x7f5679d0e4c8 "ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0",</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> file=0x7f5679d0e44b "../src/include/abti_pool.h", line=238, function=0x7f5679d0e660 <__PRETTY_FUNCTION__.6315> "ABTI_pool_release")</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> at assert.c:101</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#4 0x00007f5679d09e5e in ABTI_sched_free.part.4 ()</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> from /projects/spack/opt/spack/linux-debian10-sandybridge/gcc-8.3.0/argobots-1.0-e4x7h6mgt7igmocc4ajro2g743ej322m/lib/libabt.so.0</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#5 0x00007f5679cfe2c5 in ABTI_xstream_free ()</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> from /projects/spack/opt/spack/linux-debian10-sandybridge/gcc-8.3.0/argobots-1.0-e4x7h6mgt7igmocc4ajro2g743ej322m/lib/libabt.so.0</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#6 0x00007f5679cfa080 in ABT_finalize ()</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">Thanks,</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">Matthieu</span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>