[argobots-discuss] condition variable and rwlock

Phil Carns carns at mcs.anl.gov
Tue Aug 10 13:26:06 CDT 2021


Hi Shintaro,


I just confirmed your description with a Margo test program. I've filled 
in some description of the test case (and why this is an issue for our 
use case) in a new github issue so we can track it and discuss possible 
solutions there.


https://github.com/pmodels/argobots/issues/361.


thanks!

-Phil


On 7/23/21 1:36 PM, Iwasaki, Shintaro wrote:
> Hello Phil,
>
> > We use ABT_SCHED_BASIC_WAIT and ABT_POOL_FIFO_WAIT by default in 
> Mochi (Margo).  Even so, if the default (simple) mutex is 
> busy-yielding, that will effectively become a busy spin if the ES has 
> no other units to execute, right?
>
> If a ULT is waiting in a lock (e.g., ABT_mutex_lock() or 
> ABT_rwlock_rwlock()), yes, you're right.  Sorry for the confusing 
> behavior.  It should have been clarified somewhere.
>
> This "default" behavior is only for a historical reason.  If 
> `--disable-simple-mutex` solves some performance issues (this 
> configuration option works for Argobots 1.1), please let us know.  I 
> just need to change the default option for this.
>
> Thanks,
> Shintaro
>
> ------------------------------------------------------------------------
> *From:* Phil Carns via discuss <discuss at lists.argobots.org>
> *Sent:* Friday, July 23, 2021 12:25 PM
> *To:* Iwasaki, Shintaro via discuss <discuss at lists.argobots.org>
> *Cc:* Carns, Philip H. <carns at mcs.anl.gov>
> *Subject:* Re: [argobots-discuss] condition variable and rwlock
>
>
> On 7/23/21 11:53 AM, Iwasaki, Shintaro via discuss wrote:
>> Hi Matthieu,
>>
>> By default, ABT_rwlock() and ABT_mutex() waits in a busy-yield loop 
>> (keeps yielding until a flag is set). If that configuration is set, 
>> waiters will suspend, not yield.
>
>
> I'm confused :)
>
>
> We use ABT_SCHED_BASIC_WAIT and ABT_POOL_FIFO_WAIT by default in Mochi 
> (Margo).  Even so, if the default (simple) mutex is busy-yielding, 
> that will effectively become a busy spin if the ES has no other units 
> to execute, right?
>
>
> If so, that wasn't our intention.  I thought that the ES would idle 
> while trying to acquire an abt mutex if no other ULTs were runnable.  
> We might need to do some testing to better understand what's going on 
> there.
>
>
> thanks,
>
> -Phil
>
>
>
>>
>> This behavior becomes default because of a historical reason 
>> (https://github.com/pmodels/argobots/pull/102 
>> <https://github.com/pmodels/argobots/pull/102>).  There's a 
>> performance trade-off between a simple mutex (current default) and a 
>> non-simple mutex, so we hesitate to change it silently.
>>
>> If this default behavior is not what the user expects, I will create 
>> a PR to change it (if so I'd be happy if you create a GitHub issue 
>> that briefly describes this issue).
>>
>> Thanks,
>> Shintaro
>> ------------------------------------------------------------------------
>> *From:* Dorier, Matthieu <mdorier at anl.gov> <mailto:mdorier at anl.gov>
>> *Sent:* Friday, July 23, 2021 10:34 AM
>> *To:* discuss at lists.argobots.org <mailto:discuss at lists.argobots.org> 
>> <discuss at lists.argobots.org> <mailto:discuss at lists.argobots.org>; 
>> Iwasaki, Shintaro <siwasaki at anl.gov> <mailto:siwasaki at anl.gov>
>> *Subject:* Re: condition variable and rwlock
>> Thanks, I'm curious about your comment about  --disable-simple-mutex. 
>> What you describe is what I would expect rwlock to do by default... 
>> How do rwlock by default, then?
>>
>> Thanks
>> Matthieu
>>
>> Get Outlook for Android <https://aka.ms/ghei36>
>>
>> ------------------------------------------------------------------------
>> *From:* Iwasaki, Shintaro <siwasaki at anl.gov> <mailto:siwasaki at anl.gov>
>> *Sent:* Friday, July 23, 2021 3:51:21 PM
>> *To:* discuss at lists.argobots.org <mailto:discuss at lists.argobots.org> 
>> <discuss at lists.argobots.org> <mailto:discuss at lists.argobots.org>; 
>> Dorier, Matthieu <mdorier at anl.gov> <mailto:mdorier at anl.gov>
>> *Subject:* Re: condition variable and rwlock
>> Hello Matthieu,
>>
>> Thanks for your question!
>>
>> > Is there a way of using an Argobots condition variable with an 
>> rwlock instead of a mutex?
>> No. The user cannot combine ABT_rwlock with ABT_cond.
>>
>> > my use-case is a structure that receives many read requests and a 
>> few writes, clearly justifying using a rwlock instead of a mutex, but 
>> I may want some of the reads to block until a write has happened, 
>> which means I need a condition variable
>>
>> First, if readers encounter an rwlock locked by a writer, they will 
>> suspend until the writer releases the rwlock (if 
>> --disable-simple-mutex is set, which is not set by default).
>>
>> For this specific purpose(especially if "some", not "all" of the 
>> readers should block), what I came up with in my mind first is the 
>> following.  It looks fine except for a seemingly complex structure 
>> that uses multiple synchronization objects.
>>
>> void reader() {
>>   while (1) {
>>     if (work_queue.is_empty() and I_AM_SOME_OF_READERS()) {
>>       // 1. Internally ABT_self_suspend()-like mechanism needs
>>       //    to take a lock (even if it is in the readers' lock)
>>       //    since multiple readers might access the same data
>>       //    structure (for example, a user-maintained
>>       //    suspended ULT list).
>>       // 2. Anyway this path is not performance sensitive
>>       ABT_mutex_lock(mutex);
>>       ABT_cond_wait(cond, mutex);
>>       ABT_mutex_unlock(mutex);
>>       // Now someone woke me up after pushing work.
>>     }
>>     ABT_rwlock_rdlock(rwlock);
>>     if (!work_queue.is_empty())
>>       ; // Do real work.
>>     ABT_rwlock_unlock(rwlock);
>>   }
>> }
>>
>> void writer() {
>>   ABT_rwlock_wrlock(rwlock);
>>   work_queue.push_work(work);
>>   // You do not need to take a mutex to call ABT_cond_broadcast.
>>   // It is fine even if there's no waiter.
>>   ABT_cond_broadcast(cond);
>>   ABT_rwlock_unlock(rwlock);
>> }
>>
>> I would welcome any suggestions! (For example, does 
>> ABT_rwlock_trywrlock() help?)
>>
>> Best,
>> Shintaro
>> ------------------------------------------------------------------------
>> *From:* Dorier, Matthieu via discuss <discuss at lists.argobots.org> 
>> <mailto:discuss at lists.argobots.org>
>> *Sent:* Friday, July 23, 2021 5:03 AM
>> *To:* discuss at argobots.org <mailto:discuss at argobots.org> 
>> <discuss at argobots.org> <mailto:discuss at argobots.org>
>> *Cc:* Dorier, Matthieu <mdorier at anl.gov> <mailto:mdorier at anl.gov>
>> *Subject:* [argobots-discuss] condition variable and rwlock
>>
>> Hi,
>>
>> I suspect the answer is no, but is there a way of using an Argobots 
>> condition variable with an rwlock instead of a mutex?
>>
>> (my use-case is a structure that receives many read requests and a 
>> few writes, clearly justifying using a rwlock instead of a mutex, but 
>> I may want some of the reads to block until a write has happened, 
>> which means I need a condition variable).
>>
>> Thanks,
>>
>> Matthieu
>>
>>
>> _______________________________________________
>> discuss mailing list
>> discuss at lists.argobots.org  <mailto:discuss at lists.argobots.org>
>> https://lists.argobots.org/mailman/listinfo/discuss  <https://lists.argobots.org/mailman/listinfo/discuss>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.argobots.org/pipermail/discuss/attachments/20210810/cf1e20e7/attachment.html>


More information about the discuss mailing list