update: install python libraries atomically.

We want to avoid "upgrading" python libraries using pip, because this
may not give predictable results, and if installation fails it can
leave the system in a broken half-installed state.

Consequently, instead of installing packages into
/physionet/python-env/physionet, install them into a directory named
for the MD5 hash of requirements.txt.

Installation may produce a lot of output (e.g. when compiling
extension modules from source), so store the log in the installation
directory rather than further cluttering update.log.

After the pip command is successful, copy requirements.txt to a new
file installed.txt.  This serves as a flag to indicate that the
packages were installed successfully, and also provides documentation
of what was supposedly installed.

If requirements.txt matches installed.txt, then we know that we don't
need to reinstall anything.

After the packages have been successfully installed and we have tested
that they're working (at least well enough to invoke
getmigrationtargets), then update the symlink
/physionet/python-env/physionet to point to the new prefix.

Author: Benjamin Moody

deploy/update

@@ -4,7 +4,8 @@ set -e
 
 install_dir=/physionet/physionet-build
 working_dir=/physionet/physionet-build.new
-venv_dir=/physionet/python-env/physionet
+venv_prefix=/physionet/python-env
+venv_dir=$venv_prefix/physionet
 log_file=/data/log/pn/update.log
 branch=$(cat /physionet/deploy-branch) # production or staging
 
@@ -76,18 +77,28 @@ mkdir -p $working_dir
 git archive "$newrev" | tar -x -C $working_dir
 ln -s $install_dir/.env $working_dir/.env
 
-. $venv_dir/bin/activate
-
-# Install new dependencies from 'requirements.txt' into $venv_dir
+# Install new dependencies from 'requirements.txt' into $new_venv_dir
 if [ -n "$no_pip" ]; then
     echo "- SKIPPING requirements due to --push-option=no-pip"
+    new_venv_dir=$venv_dir
 else
-    echo "* Installing new requirements..."
-    pip3 install -r $working_dir/requirements.txt \
-         --quiet --require-hashes --log $log_file
-    echo >> $log_file
+    hash=$(md5sum "$working_dir/requirements.txt" | cut -d' ' -f1)
+    new_venv_dir=$venv_prefix/$hash
+    if [ -d "$new_venv_dir" ] && cmp -s "$working_dir/requirements.txt" \
+                                      "$new_venv_dir/installed.txt"; then
+        echo "- Requirements previously installed ($hash)."
+    else
+        echo "* Installing new requirements ($hash)..."
+        virtualenv -ppython3 --quiet --clear --no-download "$new_venv_dir"
+        pip_log=$new_venv_dir/pip-install.log
+        "$new_venv_dir/bin/pip3" install -r "$working_dir/requirements.txt" \
+                                 --quiet --require-hashes --log $pip_log
+        cp "$working_dir/requirements.txt" "$new_venv_dir/installed.txt"
+    fi
 fi
 
+. $new_venv_dir/bin/activate
+
 # Run 'getmigrationtargets' to check whether migrations make sense
 # (e.g. broken dependencies, ghost migrations)
 echo "* Checking migrations..."
@@ -98,4 +109,9 @@ echo "* Checking migrations..."
     ./manage.py getmigrationtargets > /dev/null
 )
 
+# Update $venv_dir to point to $new_venv_dir
+if [ "$venv_dir" != "$new_venv_dir" ]; then
+    ln -s -f -T "$new_venv_dir" "$venv_dir"
+fi
+
 echo "$(date '+%F %T %z'): $branch: update finished" >> $log_file