[argobots-discuss] question about stack reuse on dedicated pools

[Phil Carns]

Hi all,

I'm trying to debug problem with a service that spawns a new ULT for 
each incoming client request.  The service works correctly, but memory 
consumption is considerably higher over time if I create a dedicated 
pool with it's own execution streams to run the ULTs instead of just 
running them on the main ES.

There are some details here: 
https://xgitlab.cels.anl.gov/sds/margo/issues/40

That specific test example is creating a total of 40,000 ULTs over the 
course of execution.  It doesn't have that many active concurrently, 
though.  The client program is  issuing no more than 4 concurrent 
requests.  A ULT will slightly outlive the request handler, but the 
number of active ULTS isn't growing indefinitely.  I can confirm that 
argobots is freeing them throughout execution if I turn on Argobots logging.

I can see in the Argobots source code that when a ULT is freed it 
doesn't literally free() the stack memory for that ULT; it is kept on a 
queue:

https://github.com/pmodels/argobots/blob/master/src/include/abti_mem.h#L287

If I print the value of num_stacks in Argobots at the above line, I get 
the following:

- when using the default/main ES: no higher than 512

- when explicitly creating a new pools and ES: 40,000

I haven't tracked down the logic to understand why there is a difference 
here, but is this expected?  It looks like Argobots is keeping a much 
larger number of stacks around (without reusing them?) if I create a new 
pool and ES to run my ULTs.

thanks,

-Phil