2020 Leiningen developer experience and REPL tools

In this post we're going to set-up Leiningen so that we have some standard development tools every time we start the REPL. We'll also cover the basics of Leiningen and the REPL - just enough to be dangerous!

In the previous post we set-up Clojure with Lein and Rebel Readline to create a solid REPL user experience. If you haven't read that check it out now ... I'll wait.

Back? ... lets get started then.

The most important concept to understand with Leiningen is a 'project'. Every project in Lein consists of a set of directories that contain all the assets for the application. When Leiningen is invoked it looks for a file called project.clj where all the settings for the project are stored: it looks for that file in the current working directory. When you create a Clojure project with Leiningen it will create a standard project structure for you.

Lein Quick Overview

This is just enough to get going - for all the details go through the tutorial. You can see the tasks that Lein knows about by calling it:

$ lein
Leiningen is a tool for working with Clojure projects.

Several tasks are available:
ancient             Check your projects and profiles for outdated dependencies/plugins.
bat-test            Run clojure.test tests.
change              Rewrite project.clj with f applied to the value at key-or-path.
... <lots more> ...

Each task has help:

$ lein help upgrade

The most important ones are:

  • Create a new project

    Use the new task to create a new project with a standard project layout: it will create the project directory and a set of standard locations for source and tests. It also creates a project.clj for the project that we can then customise for the specific project.

    lein new app <your-project-name>
    
  • Run tests

    Run tests, the default is that these are in test/<namespace>/core_test.clj. Clojure comes with a test framework.

    lein test [tests]
    
  • Start the REPL

    Using the alias we set-up last time to launch the rebel-readline REPL.

    lein rebl
    lein repl ;; the standard repl
    

Lein Plugins

Leiningen is extensible through plugins. There are plugins for all sorts of developer experience tools such as testing, formatting, git integration and build automation.

Since everything in Lein is normal Clojure code plugins are distributed through Clojars . A plugin's page gives you some sense of it's popularity, as well as instructions on how to install it.

As an example we're going to install Lein-Ancient. It's the most important plugin, practically universal, as it checks and updates dependencies. Clojure projects often have large sets of dependencies, so automation is important.

To install a plugin we add it to one of the configuration files and Leiningen will download and configure it. Plugins can be loaded either globally (~/lein/profiles.clj), or for a particular project (<project root>/project.clj).

Lein Ancient is a plugin we want available all the time so we'll put it into ~/.lein/profiles.clj. The specific Clojars page tells us what to add:

[lein-ancient "0.6.15"]

It's added in the :plugins section, like this:

{:user
  {:plugins [; ... other plugins ...
               [lein-ancient "0.6.15"]
            ]

If we run lein deps or start the REPL it will download and install the plugin.

Lein Ancient can check projects (by looking in the project.clj), and the global profiles (looking at ~/.lein/profiles.clj) for out of date dependencies. To see the full help do lein help ancient, but the main commands are:

lein ancient check                      ;;default check for outdated project dependencies
lein ancient check-profiles             ;;check profiles for outdated dependencies
lein ancient upgrade                    ;;upgrade project dependencies
lein ancient upgrade-profiles           ;;upgrade profile dependencies

As we've seen Plugins change Lein's behaviour, adding commands and altering the REPL start-up: we'll look at this 'middleware' concept later. It follows that you'll want different plugins depending on the task, that's the purpose of Lein Profiles.

Lein Profiles

Profiles load librarys when we're performing a Lein task (such as starting the REPL). Last time we put some settings into the :user section of our global ~/.lein/profiles.clj so that we could start rebel-readline. This was an example of using a Lein profile - specifically the user profile. We're going to continue developing our user profile with general developer experience capabilities that are loaded every time we start the REPL.

Profiles let you change the settings that Lein uses when a command like lein rebl is run. They are maps of options that are applied to tasks. They can be configured in the following locations:

1. ~/.lein/profiles.clj
2. <project root>/project.clj
3. <project root>/profiles.clj

The ordering is that the project settings over-ride the user-wide settings specified in ~/.lein/profiles.clj. The project specific profiles.clj will over-ride the project project.clj.

We can list all profiles with lein show-profiles:

$ lein show-profiles

base
debug
default
leiningen/default
leiningen/test
offline
repl
uberjar
update
user

The list above shows the default profiles. We can create our own profile and use them with a task. The more common step is to configure the profiles that are used for certain tasks.

User Profile

The user profile is the place to store any settings or tools that we want every time we run a Lein command such as launching a REPL. This is a global capability so it's stored in ~/.lein/profiles.clj.

When starting out I found the naming around this area confusing. You'll sometimes find instructions to put plugins or settings into the user profile and sometimes into the dev profile. This is two separate things:

  • user profile is global (in ~/lein/profiles.clj): anything that you put into it will be launched when Lein runs.
  • dev profile is local (in <project dir>/project.clj): setting here only impact this specific project.

To avoid collision you should not define a :user profile in a project - if you do specify them in two locations then you'll get a conflict because profiles are not merged. Side-note: there is a way to merge profiles but it's quite advanced and I'm ignoring it for this post.

Project Dev Profile

We now have some global tools and settings (in the user profile), but what about project specific settings?

That's the job of the dev profile. The dev profile in the project project.clj keeps any settings or plugins just for that project. When Lein runs a task the dev profile takes precedence over the user profile. That's because project local settings take precedence.

We can load plugins or librarys in the same way as shown earlier. The new element is we can load libraries so that they're immediately available into the REPL environment (Rebel Readline or Vim for me).

This is done by telling Lein about a Clojure file that will be loaded automatically as the REPL is started. Technically, the file can be anywhere you'd like in the project hierarchy. Luminus uses the concept of an env directory within each project. Environment specific files are stored here, for example env/dev/, env/test/ and env/prod/.

Lets say that we've created a project called test-rebel-readline, in the projects project.clj we add the following:

:profiles {:dev
            {
             :dependencies [; ... dev dependancies ...
                           ]
             :source-paths ["env/dev"]    ;additional source path
             :main user/-main             ;which namespace and function to start in
            }
          }

In the env/dev/ directory we add a user.clj which has the following:

(ns user
  (:require
      [clojure.repl :refer :all]
      [clojure.pprint :refer (pprint)]
      [puget.printer :refer [cprint]]
      [clojure.tools.namespace.repl :refer [refresh refresh-all]]
      [clojure.test :refer :all :exclude [run-tests]]
  )
)

When the project dev profile runs it looks in the env/dev directory and loads the user.clj. If you don't specify a namespace with :main then you'll be put into the user namespace.

As you can see we're loading a bunch of tools into the REPL. The easiest one to test is to see if the clojure.repl namespace is loading - if it is then (doc print) will print the documentation for the print function.

Standard REPL Experience

With the tooling all loaded up we're ready to look at the REPL experience. We've automatically loaded the clojure.repl namespace so the main REPL functions are available:

;; shows information about the function
user=> (doc print)

;; shows the source code for the function
user=> (source println)

;; find any and all docs with print in them - don't use it much
user=> (find-doc "print")

;; find any Var with 'per' in the name
user=> (apropos 'pea)

The apropos function is for finding any var that matches the search in any loaded namespace. Here's an example:

user=> (def pear 4) ;;=> #'user/pears
user=> (apropos 'pea)
(clojure.core/repeat clojure.core/repeatedly rewrite-clj.reader/read-repeatedly user/pears)

As you can see it does find our pears Var, as well as some other functions that match.

To see all the functions and vars available within a namespace use the clojure.repl.dir function:

user=> (dir user)

user=> (dir clojure.string)

Errors and Stacktraces

Errors and Stacktraces in Clojure just aren't very nice - there's no getting around it. The default is that they spew lots of information which is very opaque.

It's definitely a big wart on the language and makes it difficult to understand what's happening as you're learning. It's a well-known critique of Clojure, and it's right on the money - but there's not much chance of it changing so lets move on!

Pretty is the biggest improvement I know of, it does a great job improving stacktraces. It shortens the stacktrace and tries to show you the root cause clearly. Here's an example:

;; create an error - pretty trys to shorten it for you
user=> (/ 1 0)
java.lang.AirthmeticException: Divide by zero

;; *e stores the last exception
user=> *e
#error {
 :cause "Divide by zero"
 <... full stacktrace ...>

;; prints a stacktrace of the exception with the pst function
user=> (pst)
clojure.core/eval   core.clj: 3214
              ...
user/eval7349  REPL Input
          ...
java.lang.ArithmeticException: Divide by zero

The most useful command to know is pst which prints the last stacktrace. The formatting is changed by Pretty to make it easier to understand.

The full stacktrace in all it's glory is stored in *e, so it's accessible if we want to know more detail.

Reloading

Clojure takes a while to start and the more libraries and tools your project adds the longer it takes. The most common workflow is to avoid having to restart the REPL by reloading functions and namespaces. Imagine that you've written a new version of a function and you want to load it into the REPL.

A helper library tools.namespace has an improved refresh function:

; it was defined in the user.clj higher, but can also be loaded in the REPL like this
user=> (require '[clojure.tools.namespace.repl :refer [refresh]])
user=> (refresh)
:reloading (test-rebel-readline.core test-rebel-readline.core-test user)
:ok

According to the documentation the refresh function scans source code directories. When you change some files and then refresh it will only reload the namespaces you've changed.

The more sophisticated version is a workflow called 'Reloaded' by Stuart Sierra that is popular when you have complex applications - see My Clojure Workflow, Reloaded. There's also a set of tooling around this concept.

Further Resources

There's lots of good content on Leiningen and the REPL.

Final Thoughts

At this point we have the best core Leiningen and REPL experience I know of. It's a 'pure REPL' experience where we work directly in the REPL. It's perfect for learning Clojure and playing with small functions.

For a full development experience we still need to connect an editor: we'll cover that in the future.

In this post we've learnt Lein's basics and how to configure it through plugins. We've also gone through how to configure a project dev profile and how to use a user.clj for a project specific developer experience. Finally, we looked at the REPL experience.

There's lots of other plugins and libraries to try, hopefully the ones I've listed are a good start! Which important ones did I miss?


Posted in Tech Saturday 07 March 2020
Tagged with tech clojure