How to use python multiprocessing proxy objects in multithreading

Question

I am using pyhton's multiprocessing package in a standard client-server model.
I have a few types of objects in the server that I register through the BaseManager.register method, and use from the client through proxies (based on the AutoProxy class).

I've had random errors pop up when I was using those proxies from multiple client threads, and following some reading I discovered that the Proxy instances themselves are not thread safe. See from the python multiprocessing documentation:

Thread safety of proxies
Do not use a proxy object from more than one thread unless you protect it with a lock. (There is never a problem with different processes using the same proxy.)

My scenario fits this perfectly then. OK, I know why it fails. But I want it to work :) so I seek an advice - what is the best method to make this thread-safe?
My particular case being that (a) I work with a single client thread 90% of the time, (b) the actual objects behind the proxy are thread safe, and (c) that I would like to call multiple methods of the same proxied-object concurrently.

As always, Internet, those who help me shall live on and never die! Those who do their best might get a consolation prize too.

Thanks,
Yonatan

Why are you using process proxies for threads? Why not just use processes and have everything work? — S.Lott
– S.Lott, Commented Feb 27, 2012 at 15:08
I am using proxies in the "client process" to refer to real objects in the "server process". I just happen to use those same proxy instances (which are in themselves, real instances in the "client process") from two client threads. Imagine the client process is a web server, acting as an interface to the server state. It is multithreaded and yet proxies the "server process" objects. — Yonatan
– Yonatan, Commented Feb 27, 2012 at 15:54

Yonatan · Accepted Answer · 2012-03-05 07:43:03Z

2

Unfortunately, this issue probably doesn't relate to too many people :(

Here's what we're doing, for future readers:

We'll be using regular thread locks to guard usage of the 'main' proxy
The main proxy provides proxies to other instances in the server process, and these are thread-specific in our contexts, so we'll be using those without a lock

Not a very interesting solution, yeah, I know. We also considered making a tailored version of the AutoProxy (the package supports that) with built-in locking. We might do that if this scenario repeats in our system. Auto-locking should be done carefully, since this is done 'behind the scenes' and can lead to race-condition deadlocks.

If anyone has similar issues in the future - please comment or contact me directly.

answered Mar 5, 2012 at 7:43

Yonatan

1,2151 gold badge18 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to use python multiprocessing proxy objects in multithreading

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related