============================== Using a shared build directory ============================== For builds that take a long time, it can be convenient to reuse them across several buildouts. To do this, use the `shared` option: >>> cache = tmpdir('cache') >>> write('buildout.cfg', ... """ ... [buildout] ... parts = foo ... download-cache = %s ... ... [foo] ... recipe = zc.recipe.cmmi ... url = file://%s/foo.tgz ... shared = True ... """ % (cache, distros)) When run the first time, the build is executed as usual: >>> print system('bin/buildout') Installing foo. foo: Unpacking and configuring configuring foo /cache/cmmi/build/ echo building foo building foo echo installing foo installing foo But after that, the existing shared build directory is used instead of running the build again: >>> remove('.installed.cfg') >>> print system('bin/buildout') Installing foo. foo: using existing shared build The shared directory ==================== By default, the shared build directory is named with a hash of the recipe's configuration options (but it can also be configured manually, see below): >>> ls(cache, 'cmmi', 'build') d For example, if the download url changes, the build is executed again: >>> import os >>> import shutil >>> shutil.copy(os.path.join(distros, 'foo.tgz'), ... os.path.join(distros, 'qux.tgz')) >>> remove('.installed.cfg') >>> write('buildout.cfg', ... """ ... [buildout] ... parts = qux ... download-cache = %s ... ... [qux] ... recipe = zc.recipe.cmmi ... url = file://%s/qux.tgz ... shared = True ... """ % (cache, distros)) >>> print system('bin/buildout') Installing qux. qux: Unpacking and configuring configuring foo /cache/cmmi/build/ echo building foo building foo echo installing foo installing foo and another shared directory is created: >>> ls(cache, 'cmmi', 'build') d d (Other recipes can retrieve the shared build directory from our part's `location` as usual, so the SHA-names shouldn't be a problem.) Configuring the shared directory ================================ If you set `shared` to an existing directory, that will be used as the build directory directly (instead of a name computed from to the recipe options): >>> shared = os.path.join(cache, 'existing') >>> os.mkdir(shared) >>> write('buildout.cfg', ... """ ... [buildout] ... parts = foo ... ... [foo] ... recipe = zc.recipe.cmmi ... url = file://%s/foo.tgz ... shared = %s ... """ % (distros, shared)) >>> remove('.installed.cfg') >>> print system('bin/buildout') Installing foo. foo: Unpacking and configuring configuring foo /cache/existing/cmmi echo building foo building foo echo installing foo installing foo If no download-cache is set, and `shared` is not a directory, an error is raised: >>> write('buildout.cfg', ... """ ... [buildout] ... parts = foo ... ... [foo] ... recipe = zc.recipe.cmmi ... url = file://%s/foo.tgz ... shared = True ... """ % distros) >>> print system('bin/buildout') While: Installing. Getting section foo. Initializing part foo. ... ValueError: Set the 'shared' option of zc.recipe.cmmi to an existing directory, or set ${buildout:download-cache} Build errors ============ If an error occurs during the build (or it is aborted by the user), the build directory is removed, so there is no risk of accidentally mistaking some half-baked build directory as a good cached shared build. Let's simulate a build error. First, we backup a working build. >>> shutil.copy(os.path.join(distros, 'foo.tgz'), ... os.path.join(distros, 'foo.tgz.bak')) Then we create a broken tarball: >>> import tarfile >>> import StringIO >>> import sys >>> tarpath = os.path.join(distros, 'foo.tgz') >>> tar = tarfile.open(tarpath, 'w:gz') >>> configure = 'invalid' >>> info = tarfile.TarInfo('configure') >>> info.size = len(configure) >>> info.mode = 0755 >>> tar.addfile(info, StringIO.StringIO(configure)) Now we reset the cache to force our broken tarball to be used: >>> shutil.rmtree(cache) >>> cache = tmpdir('cache') >>> write('buildout.cfg', ... """ ... [buildout] ... parts = foo ... download-cache = %s ... ... [foo] ... recipe = zc.recipe.cmmi ... url = file://%s/foo.tgz ... shared = True ... """ % (cache, distros)) >>> remove('.installed.cfg') >>> res = system('bin/buildout') >>> print res Installing foo. ... ValueError: Couldn't find configure The temporary directory where tarball was unpacked was left behind for debugging purposes. >>> import re >>> shutil.rmtree(re.search('foo: cmmi failed: (.*)', res).group(1)) When we now fix the error (by copying back the working version and resetting the cache), the build will be run again, and we don't use a half-baked shared directory: >>> shutil.copy(os.path.join(distros, 'foo.tgz.bak'), ... os.path.join(distros, 'foo.tgz')) >>> shutil.rmtree(cache) >>> cache = tmpdir('cache') >>> write('buildout.cfg', ... """ ... [buildout] ... parts = foo ... download-cache = %s ... ... [foo] ... recipe = zc.recipe.cmmi ... url = file://%s/foo.tgz ... shared = True ... """ % (cache, distros)) >>> print system('bin/buildout') Installing foo. foo: Unpacking and configuring configuring foo /cache/cmmi/build/ echo building foo building foo echo installing foo installing foo Interaction with other users of shared builds ============================================= While shared builds are a way to cache a build between installation runs of a given buildout part, they are, more importantly, shared between multiple parts and most probably, multiple buildouts. This implies two general rules of behaviour: We should never delete shared builds, and we need to be prepared for shared builds to be deleted by other system at any time. In other words: Every install or update run of the recipe that uses a shared build needs to check whether the build still exists on disk and rebuild it if it does not. On the other hand, a part using the shared build must not declare the shared build its own property lest buildout remove it when the shared build is no longer needed, either because the part no longer uses it or because the part itself is no longer used. The last thing we did above was to install a shared build: >>> ls(cache, 'cmmi', 'build') d If someone deletes this shared build, updating the buildout part that needs it will cause it to be rebuilt: >>> rmdir(cache, 'cmmi', 'build') >>> print system('bin/buildout') Updating foo. foo: Unpacking and configuring configuring foo /cache/cmmi/build/ echo building foo building foo echo installing foo installing foo >>> ls(cache, 'cmmi', 'build') d If we stop using the shared build, it stays in the build cache: >>> write('buildout.cfg', ... """ ... [buildout] ... parts = foo ... download-cache = %s ... ... [foo] ... recipe = zc.recipe.cmmi ... url = file://%s/foo.tgz ... """ % (cache, distros)) >>> print system('bin/buildout') Uninstalling foo. Installing foo. foo: Unpacking and configuring configuring foo /sample-buildout/parts/foo echo building foo building foo echo installing foo installing foo >>> ls(cache, 'cmmi', 'build') d Regression: Keeping track of a reused shared build ================================================== Let's first remove and rebuild everything to get some measure of isolation from the story so far: >>> remove('.installed.cfg') >>> rmdir(cache, 'cmmi', 'build') >>> write('buildout.cfg', ... """ ... [buildout] ... parts = foo ... download-cache = %s ... ... [foo] ... recipe = zc.recipe.cmmi ... url = file://%s/foo.tgz ... shared = True ... """ % (cache, distros)) >>> print system('bin/buildout') Installing foo. foo: Unpacking and configuring configuring foo /cache/cmmi/build/ echo building foo building foo echo installing foo installing foo zc.recipe.cmmi 1.2 had a bug that manifested after reusing a shared build: The part wouldn't keep track of the shared build and thus wasn't able to restore it if it got deleted from the cache. This is how it should work: >>> remove('.installed.cfg') >>> print system('bin/buildout') Installing foo. foo: using existing shared build >>> rmdir(cache, 'cmmi', 'build') >>> print system('bin/buildout') Updating foo. foo: Unpacking and configuring configuring foo /cache/cmmi/build/ echo building foo building foo echo installing foo installing foo Regression: Don't leave behind a build directory if the download failed ======================================================================= zc.recipe.cmmi up to version 1.3.1 had a bug that caused an empty build directory to be left behind if a download failed, causing it to be mistaken for a good shared build. We cause the download to fail by specifying a nonsensical MD5 sum: >>> shutil.rmtree(cache) >>> cache = tmpdir('cache') >>> write('buildout.cfg', ... """ ... [buildout] ... parts = foo ... download-cache = %s ... ... [foo] ... recipe = zc.recipe.cmmi ... url = file://%s/foo.tgz ... md5sum = 1234 ... shared = True ... """ % (cache, distros)) >>> remove('.installed.cfg') >>> print system('bin/buildout') Installing foo. ... Error: MD5 checksum mismatch for local resource at '/distros/foo.tgz'. The build directory must not exist anymore: >>> ls(cache, 'cmmi') Another buildout run must fail the same way as the first attempt: >>> print system('bin/buildout') Installing foo. ... Error: MD5 checksum mismatch for local resource at '/distros/foo.tgz'.