multiprocessing import modules on mac

modules exists inside new processes before import, using python-multiprocessing on osx.

Here's an example:

import multiprocessing
import sys
import os

print 'importing module'
def main():
    import uuid
    print 'setting var'
    uuid.some_variable = True

def process(name):
    print 'uuid module loaded:', 'uuid' in sys.modules
    print 'process pid', os.getpid()
    import uuid
    var = uuid.some_variable
    print 'var exists on mac', var

if __name__ == '__main__':
    print 'main pid', os.getpid()
    main()
    p = multiprocessing.Process(target=process, args=('test',))
    p.start()
    p.join()

output on wi开发者_JAVA技巧ndows/linux is what I would expect. The script import twice, uuid.some_variable does not exists, and uuid is not loaded before import:

importing module
main pid 4352
setting var
importing module
uuid module loaded: False
process pid 4988
AttributeError: 'module' object has no attribute 'some_variable'

However on osx, the script will only import once, uuid is loaded before import, and uuid.some_variable exists in the new process:

importing module
main pid 4399
setting var
uuid module loaded: True
process pid 4400
var exists on mac True

Somehow, the uuid module from the main process gets into the sub-process, without importing it.

Am I missing something, is this a bug, or does mac have a good reason to do this?

How did the module even get into the subprocess?

The problem is that multiprocessing uses os.fork on unix. So the sub-process becomes a copy of the main process. Dont know what's going on with Fedora, but it shouldn't work on linux either.
There doesn't seem to be any easy way to get around it.

It seems that mac implementation treats imports differently. Nevertheless, you can hack it by implementing your own repository of modules, using:

my_modules['context'] = __import__('uuid')
uuid = my_modules['context']
uuid.UUID(...)

where context is each one of your processes. I don't know why you want to do this, but this should work.