collective.xsendfile, ZODB blobs and UNIX file permissions

I am currently trying to configure collective.xsendfile, Apache mod_xsendfile and Plone 4.

Apparently the Apache process does not see blobstrage files on the file system because they contain permissions:

ls -lh var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x18/0开发者_JAVA技巧xd5/0x19/0x038ea09d0eddc611.blob -r-------- 1 plone plone 1006K May 28 15:30 var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x18/0xd5/0x19/0x038ea09d0eddc611.blob

How do I configure blobstorage to give additional permissions, so that Apache could access these files?

The modes with which the blobstorage writes it's directories and files is hardcoded in ZODB.blob. Specifically, the standard ZODB.blob.FileSystemHelper class creates secure directories (only readable and writable for the current user) by default.

You could provide your own implementation of FileSystemHelper that would either make this configurable, or just sets the directory modes to 0750, and then patch ZODB.blob.BlobStorageMixin to use your class instead of the default:

import os
from ZODB import utils
from ZODB.blob import FilesystemHelper, BlobStorageMixin
from ZODB.blob import log, LAYOUT_MARKER

class GroupReadableFilesystemHelper(FilesystemHelper):
    def create(self):
        if not os.path.exists(self.base_dir):
            os.makedirs(self.base_dir, 0750)
            log("Blob directory '%s' does not exist. "
                "Created new directory." % self.base_dir)
        if not os.path.exists(self.temp_dir):
            os.makedirs(self.temp_dir, 0750)
            log("Blob temporary directory '%s' does not exist. "
                "Created new directory." % self.temp_dir)

        if not os.path.exists(os.path.join(self.base_dir, LAYOUT_MARKER)):
            layout_marker = open(
                os.path.join(self.base_dir, LAYOUT_MARKER), 'wb')
            layout_marker.write(self.layout_name)
        else:
            layout = open(os.path.join(self.base_dir, LAYOUT_MARKER), 'rb'
                          ).read().strip()
            if layout != self.layout_name:
                raise ValueError(
                    "Directory layout `%s` selected for blob directory %s, but "
                    "marker found for layout `%s`" %
                    (self.layout_name, self.base_dir, layout))

    def isSecure(self, path):
        """Ensure that (POSIX) path mode bits are 0750."""
        return (os.stat(path).st_mode & 027) == 0

    def getPathForOID(self, oid, create=False):
        """Given an OID, return the path on the filesystem where
        the blob data relating to that OID is stored.

        If the create flag is given, the path is also created if it didn't
        exist already.

        """
        # OIDs are numbers and sometimes passed around as integers. For our
        # computations we rely on the 64-bit packed string representation.
        if isinstance(oid, int):
            oid = utils.p64(oid)

        path = self.layout.oid_to_path(oid)
        path = os.path.join(self.base_dir, path)

        if create and not os.path.exists(path):
            try:
                os.makedirs(path, 0750)
            except OSError:
                # We might have lost a race.  If so, the directory
                # must exist now
                assert os.path.exists(path)
        return path


def _blob_init_groupread(self, blob_dir, layout='automatic'):
    self.fshelper = GroupReadableFilesystemHelper(blob_dir, layout)
    self.fshelper.create()
    self.fshelper.checkSecure()
    self.dirty_oids = []

BlobStorageMixin._blob_init = _blob_init_groupread

Quite a hand-full, you may want to make this a feature request for ZODB3 :-)

While setting up a backup routine for a ZOPE/ZEO setup, I ran into the same problem with blob permissions.

After trying to apply the monkey patch that Mikko wrote (which is not that easy) i came up with a "real" patch to solve the problem.

The patch suggested by Martijn is not complete, it still does not set the right mode on blob files.

So here's my solution:

1.) Create a patch containing:

Index: ZODB/blob.py
===================================================================
--- ZODB/blob.py    (Revision 121959)
+++ ZODB/blob.py    (Arbeitskopie)
@@ -337,11 +337,11 @@

     def create(self):
         if not os.path.exists(self.base_dir):
-            os.makedirs(self.base_dir, 0700)
+            os.makedirs(self.base_dir, 0750)
             log("Blob directory '%s' does not exist. "
                 "Created new directory." % self.base_dir)
         if not os.path.exists(self.temp_dir):
-            os.makedirs(self.temp_dir, 0700)
+            os.makedirs(self.temp_dir, 0750)
             log("Blob temporary directory '%s' does not exist. "
                 "Created new directory." % self.temp_dir)

@@ -359,8 +359,8 @@
                     (self.layout_name, self.base_dir, layout))

     def isSecure(self, path):
-        """Ensure that (POSIX) path mode bits are 0700."""
-        return (os.stat(path).st_mode & 077) == 0
+        """Ensure that (POSIX) path mode bits are 0750."""
+        return (os.stat(path).st_mode & 027) == 0

     def checkSecure(self):
         if not self.isSecure(self.base_dir):
@@ -385,7 +385,7 @@

         if create and not os.path.exists(path):
             try:
-                os.makedirs(path, 0700)
+                os.makedirs(path, 0750)
             except OSError:
                 # We might have lost a race.  If so, the directory
                 # must exist now
@@ -891,7 +891,7 @@
             file2.close()
         remove_committed(f1)
     if chmod:
-        os.chmod(f2, stat.S_IREAD)
+        os.chmod(f2, stat.S_IRUSR | stat.S_IRGRP)

 if sys.platform == 'win32':
     # On Windows, you can't remove read-only files, so make the

You can also take a look at the patch here -> http://pastebin.com/wNLYyXvw

2.) Store the patch under name 'blob.patch' in your buildout root directory

3.) Extend your buildout configuration:

parts += 
    patchblob
    postinstall

[patchblob]
recipe = collective.recipe.patch
egg = ZODB3
patches = blob.patch

[postinstall]
recipe = plone.recipe.command
command = 
    chmod -R g+r ${buildout:directory}/var
    find ${buildout:directory}/var -type d | xargs chmod g+x
update-command = ${:command}

The postinstall sections sets desired group read permissions on already existing blobs. Note, also execute permission must be given to the blob folders, that group can enter the directories.

I've tested this patch with ZODB 3.10.2 and 3.10.3.

As Martijn suggested, this should be configurable and part of the ZODB directly.