[argobots-discuss] question about stack reuse on dedicated pools

[Halim Amer]

Phil,

Yes, your understanding is correct. Single-producer-multiple-consumer 
cases trigger this issue. I had another user complain about this and I 
advised him to tune the ABT_MEM_MAX_NUM_STACKS for his needs. I was 
considering lowering the default value for a better performance vs. 
memory footprint tradeo-ff. Since this is the second time the issue is 
reported, I think it's time to raise it's priority. We will do some 
performance testing on our end and update the default value.

Thanks for reporting.

Halim
http://www.mcs.anl.gov/~aamer

On 2/23/18 1:13 PM, Phil Carns wrote:
> Ok, I think I'm on the trail now.
> 
> The default num stacks (how many it will queue for reuse on an ES) in 
> Argobots is 65536.  You can override this with "export 
> ABT_MEM_MAX_NUM_STACKS=64".  If I do that, then my memory consumption 
> stays down and I see that the stacks are overflowing to a global pool 
> where they (presumably) are available for reuse by the ES that is 
> launching ULTs in my case.
> 
> Anecdotally, the default limit of 65536 looks like it is wasting around 
> 500 MiB of memory *per ES* that we create in our usage scenario, since 
> the stacks can't be reused from the local pool of the ES's that are 
> executing our ULTs.
> 
> Is the moral of the story that we should set ABT_MEM_MAX_NUM_STACKS much 
> lower in our use case?  I'm going see if I can do that automatically 
> from the library that we are using to set up this scenario (Margo).
> 
> thanks,
> -Phil
> 
> On 02/23/2018 02:01 PM, Phil Carns wrote:
>> Is the issue that when a ULT is created it draws from the stack queue 
>> on the calling ES, but when the ULT is free'd it is put on the stack 
>> queue of the ES that executed it?
>>
>> In the example I have that is consuming more memory than expected, the 
>> ULTs are always dispatched by one specific ES (the one that is driving 
>> progress on the network transport) and executed on a different ES (the 
>> dedicated pool set aside for servicing I/O operations).
>>
>> thanks,
>> -Phil
>>
>> On 02/23/2018 01:09 PM, Phil Carns wrote:
>>> Hi all,
>>>
>>> I'm trying to debug problem with a service that spawns a new ULT for 
>>> each incoming client request.  The service works correctly, but 
>>> memory consumption is considerably higher over time if I create a 
>>> dedicated pool with it's own execution streams to run the ULTs 
>>> instead of just running them on the main ES.
>>>
>>> There are some details here: 
>>> https://xgitlab.cels.anl.gov/sds/margo/issues/40
>>>
>>> That specific test example is creating a total of 40,000 ULTs over 
>>> the course of execution.  It doesn't have that many active 
>>> concurrently, though.  The client program is  issuing no more than 4 
>>> concurrent requests.  A ULT will slightly outlive the request 
>>> handler, but the number of active ULTS isn't growing indefinitely.  I 
>>> can confirm that argobots is freeing them throughout execution if I 
>>> turn on Argobots logging.
>>>
>>> I can see in the Argobots source code that when a ULT is freed it 
>>> doesn't literally free() the stack memory for that ULT; it is kept on 
>>> a queue:
>>>
>>> https://github.com/pmodels/argobots/blob/master/src/include/abti_mem.h#L287 
>>>
>>>
>>> If I print the value of num_stacks in Argobots at the above line, I 
>>> get the following:
>>>
>>> - when using the default/main ES: no higher than 512
>>>
>>> - when explicitly creating a new pools and ES: 40,000
>>>
>>> I haven't tracked down the logic to understand why there is a 
>>> difference here, but is this expected?  It looks like Argobots is 
>>> keeping a much larger number of stacks around (without reusing them?) 
>>> if I create a new pool and ES to run my ULTs.
>>>
>>> thanks,
>>>
>>> -Phil
>>>
>>> _______________________________________________
>>> discuss mailing list
>>> discuss at lists.argobots.org
>>> https://lists.argobots.org/mailman/listinfo/discuss
>>
>>
>> _______________________________________________
>> discuss mailing list
>> discuss at lists.argobots.org
>> https://lists.argobots.org/mailman/listinfo/discuss
> 
> 
> _______________________________________________
> discuss mailing list
> discuss at lists.argobots.org
> https://lists.argobots.org/mailman/listinfo/discuss