<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Matthieu,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for your quick update.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin:0px">If a pool created by ABT_pool_create() is associated with the primary execution stream (and thus automatic = false), you do not need to free it explicitly. ABT_finalize() will free everything that is associated with the primary stream.
If you use Argobots 1.0, doing so for a non-automatic pool might cause memory leak, though.<br>
</div>
<div style="margin:0px"><br>
</div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> I think it would be good if Argobots allowed pools created by ABT_pool_create to have an �automatic� flag as well, and be free automatically. Otherwise I�m not even sure how we are supposed to free the pool associated with the primary ES if that pool happens
to be a custom one.
<div><br>
</div>
<div>Yes, I fully agree. If I could redesign all functions from scratch, ABT_pool_create() would have an automatic argument. Unfortunately, Argobots 1.0 and 1.1 do not have such a configuration option, so please maintain flags externally to selectively free
only pools that are created by ABT_pool_create() if necessary. We will extend ABT_pool_config to support automatic configuration in the future.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
---</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
If your program uses a user-defined pool, please check <a href="https://github.com/pmodels/argobots/issues/316">https://github.com/pmodels/argobots/issues/316</a>.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This issue should be fixed somehow before Argobots 1.1, which is planned to be released soon (within one or two weeks).</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Shintaro</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Dorier, Matthieu <mdorier@anl.gov><br>
<b>Sent:</b> Tuesday, March 23, 2021 9:55 AM<br>
<b>To:</b> Iwasaki, Shintaro <siwasaki@anl.gov>; discuss@argobots.org <discuss@argobots.org><br>
<b>Subject:</b> Re: Program aborting in ABT_finalize (ABTI_pool_release failed)</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:blue;
text-decoration:underline}
p.x_xmsonormal, li.x_xmsonormal, div.x_xmsonormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
p.x_xxmsonormal, li.x_xxmsonormal, div.x_xxmsonormal
{margin:0cm;
font-size:12.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle24
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:72.0pt 72.0pt 72.0pt 72.0pt}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal"><span lang="EN-US" style="">Hi Shintaro,</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="">I think I�m narrowing down the problem (I�m testing right now). We initially had all our pools created with automatic=TRUE in Margo, and weren�t calling ABT_pool_free anywhere. But a recent commit from Phil
enabled the use of a custom pool implementation, created with ABT_pool_create. The problem with these pools is that there is no �automatic� flag, so he changed to automatic=FALSE for the basic pools so he could call ABT_pool_free on all the pools, however
not keeping track nor checking that a pool was associated with the primary ES.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="">I think it would be good if Argobots allowed pools created by ABT_pool_create to have an �automatic� flag as well, and be free automatically. Otherwise I�m not even sure how we are supposed to free the pool
associated with the primary ES if that pool happens to be a custom one.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="">Thanks,</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="">Matthieu</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style=""> </span></p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_MsoNormal"><b><span style="font-size:12.0pt; color:black">From: </span>
</b><span style="font-size:12.0pt; color:black">"Iwasaki, Shintaro" <siwasaki@anl.gov><br>
<b>Date: </b>Tuesday, 23 March 2021 at 14:44<br>
<b>To: </b>"Dorier, Matthieu" <mdorier@anl.gov><br>
<b>Subject: </b>Re: Program aborting in ABT_finalize (ABTI_pool_release failed)</span></p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Hi Matthieu,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Even if automatic is false, please do not call ABT_pool_free() for the main ES's pool. It must be automatically freed by ABT_finalize().</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black; background:white">Argobots 1.0 "works" though a few hundred bytes will be leaked because of its bug </span><span style="font-size:12.0pt; color:black">(a few hundred bytes per pool on ABT_finalize()
as far as I checked with Valgrind).</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Argobots 1.1 (or the Argobots main branch) has fixed this memory leak issue, so I would recommend you use the latest one if possible.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">> Argobots isn�t checking whether a pool is in use before freeing it whether it�s used by the first ES.</span></p>
</div>
<div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Currently Argobots checks neither of them, but it should be added to ease debugging.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">I created an issue: <a href="https://github.com/pmodels/argobots/issues/315">https://github.com/pmodels/argobots/issues/315</a></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Thanks,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Shintaro</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Dorier, Matthieu <mdorier@anl.gov><br>
<b>Sent:</b> Tuesday, March 23, 2021 9:20 AM<br>
<b>To:</b> Iwasaki, Shintaro <siwasaki@anl.gov>; discuss@argobots.org <discuss@argobots.org>; discuss@lists.argobots.org <discuss@lists.argobots.org><br>
<b>Subject:</b> Re: Program aborting in ABT_finalize (ABTI_pool_release failed)</span>
</p>
<div>
<p class="x_MsoNormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xmsonormal"><span lang="EN-US">I�m creating the pool using </span>ABT_pool_create_basic and �automatic� set to false, and I do indeed call ABT_pool_free on it.</p>
<p class="x_xmsonormal">I guess Argobots isn�t checking whether a pool is in use before freeing it (or whether it�s used by the first ES)?</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">My code is pretty complex so I tried to write a reproducer but I don�t see the error with it. Maybe I�m corrupting memory, somehow, and it just happens to be fine in small code that don�t reuse the memory after the ABT_pool_free� (though
I ran address sanitizer and didn�t see any invalid access).</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">I�ll see what happens if I remove the ABT_pool_free.</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Thanks,</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Matthieu</p>
<p class="x_xmsonormal"><span lang="EN-US"> </span></p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p class="x_xmsonormal"><b><span style="font-size:12.0pt; color:black">From: </span>
</b><span style="font-size:12.0pt; color:black">"Iwasaki, Shintaro" <siwasaki@anl.gov><br>
<b>Date: </b>Tuesday, 23 March 2021 at 14:01<br>
<b>To: </b>"discuss@argobots.org" <discuss@argobots.org>, "discuss@lists.argobots.org" <discuss@lists.argobots.org><br>
<b>Cc: </b>"Dorier, Matthieu" <mdorier@anl.gov><br>
<b>Subject: </b>Re: Program aborting in ABT_finalize (ABTI_pool_release failed)</span></p>
</div>
<div>
<p class="x_xmsonormal"> </p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Hi Matthieu,</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">I don't know when such an error typically happens. I'd be happy if you could share your code with me so that I can closely look at it.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">This is just my guess, but perhaps ABT_pool_free() is used; in Argobots, primary ES's pool and schedulers may not be freed before ABT_finalize().</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">If you free that fifo_wait MPMC pool before ABT_finalize(), an already freed pool is accessed in ABT_finalize(), which may cause the error you wrote.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">(It is related to
<a href="https://github.com/pmodels/argobots/issues/58">https://github.com/pmodels/argobots/issues/58</a>).</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">I attached a sample code to explain this issue. Please read the comments in the code if you are interested in it.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Thanks,</span></p>
</div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Shintaro</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_divRplyFwdMsg">
<p class="x_xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Dorier, Matthieu via discuss <discuss@lists.argobots.org><br>
<b>Sent:</b> Tuesday, March 23, 2021 6:56 AM<br>
<b>To:</b> discuss@argobots.org <discuss@argobots.org><br>
<b>Cc:</b> Dorier, Matthieu <mdorier@anl.gov><br>
<b>Subject:</b> [argobots-discuss] Program aborting in ABT_finalize (ABTI_pool_release failed)</span>
</p>
<div>
<p class="x_xmsonormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">Hi,</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">I�m getting this error in a program, when calling ABT_finalize:</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">../src/include/abti_pool.h:238: ABTI_pool_release: Assertion `ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0' failed.</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">The program only uses the primary ES, it does not create additional ES, however it does replace the primary ES�s pool and scheduler by creating a fifo_wait MPMC pool and passing it to ABT_xstream_set_main_sched_basic,
with </span><span style="font-size:11.0pt">ABT_SCHED_BASIC_WAIT as sched_predef.</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">I�m using Argobots 1.0. I haven�t tested with other versions of Argobots. I�m sure I must be doing something wrong somewhere in my code. Could you tell me what I could look for that would
lead to such an error?</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt">For additional information, here it the stack trace when the error happens:</span></p>
<p class="x_xxmsonormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#1 0x00007f56797fe535 in __GI_abort () at abort.c:79</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#2 0x00007f56797fe40f in __assert_fail_base (fmt=0x7f5679960ee0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> assertion=0x7f5679d0e4c8 "ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0", file=0x7f5679d0e44b "../src/include/abti_pool.h", line=238,</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> function=<optimized out>) at assert.c:92</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#3 0x00007f567980c102 in __GI___assert_fail (assertion=0x7f5679d0e4c8 "ABTD_atomic_acquire_load_int32(&p_pool->num_scheds) > 0",</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> file=0x7f5679d0e44b "../src/include/abti_pool.h", line=238, function=0x7f5679d0e660 <__PRETTY_FUNCTION__.6315> "ABTI_pool_release")</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> at assert.c:101</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#4 0x00007f5679d09e5e in ABTI_sched_free.part.4 ()</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> from /projects/spack/opt/spack/linux-debian10-sandybridge/gcc-8.3.0/argobots-1.0-e4x7h6mgt7igmocc4ajro2g743ej322m/lib/libabt.so.0</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#5 0x00007f5679cfe2c5 in ABTI_xstream_free ()</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> from /projects/spack/opt/spack/linux-debian10-sandybridge/gcc-8.3.0/argobots-1.0-e4x7h6mgt7igmocc4ajro2g743ej322m/lib/libabt.so.0</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">#6 0x00007f5679cfa080 in ABT_finalize ()</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">Thanks,</span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_xxmsonormal"><span style="font-size:11.0pt">Matthieu</span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>