Automating development environment set-up with Direnv

For each development project I need a distinct environment so that any dependencies and alterations are isolated from the rest of the system. There are a variety of ways of doing this, from using environmental variables, through to chroots and virtual environments. Direnv automates activating a virtual environment - when I go to a project directory it activates the environment I need for that project. It's a two-stage process, I use virtualenvwrapper to create a virtual environment for each Python project, and then Direnv is used to automate executing the virtual environment.

Direnv is a general automation utility, its role is to alter the shell environment or run a script when the user enters a specific directory. It fulfils a similar role to other tools such as Autoenv, Smartcd, Zsh-autoenv etc. I started with Autoenv but it can't automatically deactivate an environment when the user leaves the project directory [1]. Direnv is able to do this, which makes it a more general automation utility. It plugs into the shell rather than over-riding cd which makes it more flexible. It's also compatible with Bash, Zsh, Tcsh or Fish.

Commands to be run when entering a target directory are stored in a .envrc file that's stored in the directory. To provide security direnv makes you authorise automatically executing this file the first time it sees it, and if there are any subsequent changes to the file it won't execute it until you re-authorise it. You can see the checksums in ~/.config/direnv/allow

Installing

Direnv is available on Github so can be installed directly from source. For Ubuntu and Debian users it's available in the repositories.

  • Download the binary package for your release:
https://launchpad.net/ubuntu/+source/direnv
  • Install it:
$ dpkg --install direnv_2.7.0-1_amd64.deb

Configuration

To configure direnv we have to set it to automatically execute when our shell changes directory. It does this by hooking into the shell directly. For me that's Bash so the command to set this up is:

$ echo 'eval "$(direnv hook bash)"' >> ~/.bashrc

Execute a new shell and it will pull this hook in.

The second stage is to create a specific .envrc for the commands that you want it to automate. Change to the directory where you want the automation and do direnv edit.

Automating Python virtualenv's

For Python development I use Virtualenvwrapper to set-up project specific configurations. See my post Virtualenv and virtualenvwrapper for Python for more information. With direnv I can automate it so that when I change into the directory it executes the virtualenv I've set-up for that project, and when I leave the directory it resets my bash shell to normal.

Lets say I have a project in ~/projects/project1 which already has a virtualenv defined for it, I create an .envrc in that directory and put in:

source /usr/share/virtualenvwrapper/virtualenvwrapper.sh
workon project1

I then run direnv allow which tells it that this .envrc is correct. Direnv will now automatically run that command when I enter the project directory.

When I use virtualenvwrapper it changes the shell prompt when I'm in a virtualenv so I have a visual indicator that I'm not in my standard environment. We can do the same by editing [2] .bashrc and adding:

show_virtual_env() {
    if [ -n "$VIRTUAL_ENV" ]; then
        virtualenv=`basename "$VIRTUAL_ENV"`
        echo "($virtualenv)"
    fi
}
PS1='$(show_virtual_env)'$PS1

That's it - every time I enter ~/projects/project1 it will execute the virtualenvwrapper project project1, and when I move out of the directory it will reset my environment:

~/workspace$ cd project1/
direnv: loading ~/.direnvrc
direnv: loading .envrc
direnv: using vwrapper
direnv: export +VIRTUAL_ENV ~PATH
~/workspace/project1$ cd ..
direnv: unloading

The only problems I've found so far is that the deactivate command doesn't work any more, you have to change to another directory rather than explicitly deactivating the environment - not sure why. It will also deactivate the environment if you use cdsitepackages as you're moving outside the directory hierarchy the .envrc is controlling, but that's fine for me.

Other automation

Direnv comes with a standard library of automation functions that you can use. You can list them and read the documentation with:

$ direnv stdlib | less

The ones that appear the most useful are:

PATH_add <path>
Adds the specified path first in the PATH variable. For example, if you have a local bin/ directory you could do PATH_add ~/Projects/Project1/bin
load_prefix <path>
Adds the specified path prefix to CPATH, LD_LIBRARY_PATH, LIBRARY_PATH, MANPATH, PATH, PKG_CONFIG_PATH. Useful for anything where you need to compile and then run with a specific project prefix.
source_env / source_up
Both of these lets you pull in another .envrc or shell script for using in setting up this specific environment.
layout
These are defined set-ups for Python, Ruby, Go, Node and Perl. For Python you can specify the python version to use.
has <binary>
Used to test if a binary exists or shell function exists. For example, we could test for whether a development environment has a specific binary before executing it.
use
Lets you call a specific binary, used to call a specific version of the Python or Ruby interpreter.
dotenv <dotenv file>
Stores things in a .env file into the environment. There are a lot of tools that are using .env for storing specific environmental variables used during development.

Extending direnv

Aside from the direnv standard library it's possible to add your own shell functions in a ~/.direnvrc. This article by Puzan [3] covers adding a shell function for using virtualenv, so a simple extension to use virtualenvwrapper is below:

#export DIRENV_LOG_FORMAT=

use_venv () {
     export VIRTUAL_ENV = "$ {HOME} /.virtualenvs/ $ { 1}"
     PATH_add "$ VIRTUAL_ENV / bin"
 }

use_vwrapper () {
    source /usr/share/virtualenvwrapper/virtualenvwrapper.sh
}

Then in an .envrc we can now do:

use vwrapper
workon <somevirtualenv>

Tell Git to ignore it

If you're using direnv to store secret environmental variables (ie login secrets) then the .envrc should not be stored in your version control [4].

$ git config --global core.excludesfile "~/.gitignore_global"

Then add to ~/.gitignore_global:

# Direnv stuff
.direnv
.envrc
EXCL

Direnv resources

The main resources I found for Direnv are:

Final words

With Direnv and Virtualenvwrapper my development set-up is completely automated. Just need to think what else I can automate now!


[1]The lead developer closed the long-running Issue asking for this feature after I reawakened it which is a pretty clear answer he considers it out of scope!
[2]From Python direnv tips
[3]From Direnv by Puzan
[4]From Git - Direnv wiki

Posted in Tech Wednesday 03 February 2016
Tagged with Linux Python Direnv Virtualenv Virtualenvwrapper