Using Python from Clojure (libpython-clj)

If you've used Clojure you know that it's a hosted language where you can easily interoperate with Java. Recently Chris Nuernberger announced Libpython-clj which makes it easy to interoperate with Python.

This opens up Python's range of libraries - many of these are widely used and are very accessible with great examples. I'm far more familiar with Python than Java so the chance to use Python libraries is really interesting.

Today, it's not as easy to use Python libraries from Clojure as Java Libraries, but it wasn't too difficult - I could figure out most of the things I wanted to get done. I'm doing this on February 2nd 2020 - 02022020 Palindrome Day, whoop!

My interest was sparked by Gigasquids Parens for Pyplot post that showed using Python's pyplot from Clojure. I've written about pyplot before (specifically about styles) and while it is complex it's also a very good library.

For Libpython-clj there's a Usage document in the repository that is worth reading through. And, Chris' video Extending Python with Clojure is also worth going through.

Install

To install it you need to set-up your python3 environment and install any Python libraries that you want to use. I'm on Ubuntu 18.04 (Bionic) so for me that was apt for the python3 parts and then I use pip3 for local libraries (numpy, matplotlib, etc).

$ sudo apt install libpython3.6-dev python3-pip
$ pip3 install --user numpy pandas matplotlib

The examples I found use Clojure Deps, but I use Lein - so to create a small project location and set-up a Lein project:

$ lein new app libpython-clj-play
$ cd libpython-clj-play
$ <edit the project.clj>

Add the following to the project.clj and run lein deps to install everything:

:dependencies [[org.clojure/clojure "1.10.1"]
                [cnuernber/libpython-clj "1.33"]]

REPL Clojure and Python

We can now start the Clojure REPL and start playing around: lein repl (lein rebl for me). I'm using the REPL for all my Clojure code, and ipython for Python.

First we need to import libpython-clj:

(require '[libpython-clj.require :refer [require-python]])
(require '[libpython-clj.python :as py :refer [py. py.. py.-]])

The simplest thing we can do is run a simple Python statement from within Clojure. There's a special function run-simple-string to do this. In Python we would do:

print ("Hello World from Python!")
# Hello World from Python!

Now we're going to run the same statement but from within Clojure's REPL:

;; Run a string expression and return a global map of Pythons
;; globals, locals and result
(doc py/run-simple-string)

(doc py/run-simple-string "print ('Hello World from Python!') ")

When Chris introduces the library he spends time explaining that you can create objects in python that copy data from Clojure into Python: this is the py-{something} functions. And, that you can also bridge data between the two of them - this creates an object in Python and a Var in Clojure so that they are sharing data.

I'm going to ignore most of that part for this post: first because I don't understand it properly and I couldn't get all the functions to work. Second, because it appears you can mostly ignore this as an implementation detail.

Note

If anyone can get as-list, as-map to work with a simple example that would be great.

Python Attributes

In Chris's introduction to libpython-clj he talks about the fact that Python objects are essentially two dictionaries:

  • Python attributes: in Python we use the . operator to access them
  • Python items: in Python we use [] to access them, under the hood it's using __getitem__ and __setitem__

We need to know this later when we want to access these attributes from within Clojure. I would have liked to have done this example using plain Python but couldn't figure it out, so this is basically Chris' examples using numpy.

Lets first use numpy in Python and access various attributes:

import numpy as np

# create a ones array
onesary = np.ones((2, 1))
onesary

# see all the attributes of the object
dir(onesary)
help(onesary.shape)

# array([[1.],[1.]])
# using the len attribute
len(onesary)

# call the len attribute
getattr(onesary,'__len__')()

In Clojure's libpython-clj we do the same thing, but use its call-attr function:

;; import numpy and call the ones attribute to create a var called
;; onesary with the output of the numpy ones function
(def np (py/import-module "numpy"))
(def onesary (py/call-attr np "ones" [2 3]))
;;=>#'user/onesary
onesary
;;=> [[1. 1. 1.] [1. 1. 1.]]

;; see all the attributes of the Python object
(py/att-type-map onesary)

;; call an attribute of the object
(py/call-attr onesary "__len__")
;;=> shows the attributes

;; call a function as an attribute
(py/get-attr onesary "sum")
(py/call-attr onesary "sum")

The important libpython-clj function here are:

  • import-module - importing a Python module, more on this later
  • att-type-map - listing all the attributes on the Python object
  • get-attr - accessing an attribute, basically calling a Python method
  • call-attr - accessing an attribute, basically calling a Python method

We can also bring numpy functions directly into Clojure and use them as normal. Lets use the 'average' function from numpy - first in Python and then inside Clojure:

help (np.average)
np.average([1, 8, 4, 10])
# 5.75
;; we can use other numpy functions
;; here we import the function directly into our namespace
(py/from-import numpy average)

;; see information about it use libpython-clj attr-type-map
;; also normal doc shows the docstring from the object
(py/attr-type-map average)

(average [1, 8, 4, 10])
;;=> 5.75

We use libpython-clj's attr-type-map to see information about the function and then we can call it 'as normal' - this is really neat!

Import modules

We can import modules using using libpython-clj's import-module capabilities. There's import-module, import-as and require-python.

The simplest form is to import using the import-module option:

; import and see what's in the module
(def pymath (py/import-module "math"))
(py/att-type-map pymath)
(py/dir pymath) ;;=> [list of attributes sorted]

; call a specific attribute of the pymath object
(py/call-attr pymath "factorial" 5) ;;=> 120

The easiest way to call a function is to use the call-attr function as you see above: notice that we tell it the Python module and the function within the module (as a string). The other option is to use some syntax sugar that libpython-clj gives us is - which is to call py. <something>:

(py. pymath factorial 5)

The nice thing with the py. option is that you don't have to specify the function as a string.

We can also import the specific capability into the current namespace so it's transparent using from-import:

(py/from-import math sin)
(sin 1.0)

If you want to call the Python objects methods in a more complex way there is call-attr-kw which lets you send a vector of positional args and a map of keyword args. This is because Python has specific rules about how positional arguments and keyword arguments are used. I noticed that you have to use double quotes for these arguments. Here's something a bit more complex:

;; import the module and see what's available
(def io (py/import-module "io"))
(py/att-type-map io)
(py/dir io)

;; chain together and print the doc for the function
(-> (py/get-attr io "open")(py/get-attr "__doc__") print)

;; call a object attribute with a vector of positional args
;; and a map of keyword args
(def fh (py/call-attr-kw io "open" ["README.md", "r"] {:encoding "utf-8"}))
(py/dir fh)

(py/call-attr fh "readlines")
(py. fh seek 0)
(py. fh readline 2)

(py/call-attr fh "close")

The last function available is require-python. I initially thought I could use this and not the other import functions. But, had a lot of trouble with it.

There is fundamental difference that I don't understand, which could be my lack of knowledge in Clojure or in Python! Anyway, using require-python didn't seem to leave the functions accessible in a way I could use when I played with matplotlib. Bottom line I landed up using import-module instead.

These examples from the docs seem to work:

;; this works fine
(doc require-python)
(require-python '[math :as pmath])
(pmath/sin 1.0) ;;=> 0.84147

;; this one complains you're replacing Clojure's get
(require-python '[requests :refer [get post]])
(requests/get "https://www.google.com")

;; bind to the specific namespaces
(require-python '[requests :bind-ns true])

Matplotlib

Restart your REPL and then we can play around with matplotlib. If you want a quick introductory Matplotlib tutorial then Real Python's is good.

What I'm going to do is build a bar chart plot and see if I can get them to look the same in Python and in Clojure. The two final plots are just below. As you can see they come out exactly the same - the one on the left is from Clojure, and the one on the right is from Python

Pyplot bar chart created in Clojure Pyplot bar chart created in Python

The basic reality of matplotlib is - there's more than one way to do everything - because it's been around a long time and has a variety of interfaces and capabilities. In Carin's Parens for Pyplot post she created a macro to handle plotting. Her system saves the image and opens it using Java. If you try to open matplotlib's viewer (as you would in Python) then it will crash the Clojure REPL. I just save the file using matplotlibs save function and open it in another window.

First lets play around with Matplotlib styles. Styles let us make our plots look good easily. Here's how we set them in Python and then how we set them in Clojure:

dir(mplt.style)
help(mplt.style)

mplt.style.available
mplt.style.use('fivethirtyeight')

The difficulty in Clojure is that we need to go down Matplotlib's object hierarchy, essentially we have pyplot.style. There are two options, we could just import that specific object (py/from-import matplotlib style). Alternatively, Libpython-clj provides a way to access down the hierarchy with py.. <object> <object>. There are two ways to then call functions lower down the hierarchy, without arguments and with arguments. It's easiest to see it in action:

(require '[libpython-clj.require :refer [require-python]])
(require '[libpython-clj.python :as py :refer [py. py.. py.-]])

(def plt (py/import-module "matplotlib.pyplot"))

;; nested call through modules - in Python plt.style.available
;; Var is plt and then use minus in front of each level down
(py.. plt -style -available)
;; prints out a vector of styles

;; uses an extended format to access nested functions
;; here the last one is the function with arguments
(py.. plt -style (use "ggplot"))

In the first call using py.. we specify the module (plt), and then for each object we use a minus in front of it (-style) - we're then calling the 'available' function without any parameters. For the second call we use brackets and we can provide argments.

Now lets create a plot, first in Python and then in Clojure. In this case we're going to specify a style, change some simple defaults and then create a bar chart.

import matplotlib.pyplot as plt
plt.style.use('ggplot')

plt.rcParams['font.serif'] = 'Ubuntu'
plt.rcParams['font.size'] = 10

width, height = plt.figaspect(1.68)
fig = plt.figure(figsize=(width,height), dpi=400)

plt.bar(['Mon', 'Tues', 'Wed', 'Thur', 'Fri'], [3, 5, 4, 4, 7], color='#5c00e6')

plt.xlabel('Week days')
plt.ylabel('Fruit eaten')

plt.title("Fruit eaten by days")
plt.savefig('python-example.png', bbox_inches='tight')

plt.show()
plt.close(fig)
;;(require 'clojure.repl)
(require '[libpython-clj.require :refer [require-python]])
(require '[libpython-clj.python :as py :refer [py. py.. py.-]])

(def plt (py/import-module "matplotlib.pyplot"))

;; nested call through modules - in Python plt.style.available
(py.. plt -style -available)

;; uses an extended format to access nested functions
;; here the last one is the function with arguments
(py.. plt -style (use "ggplot"))

;; list out all of the rcParams
(py.- plt rcParams)
(py/$. plt rcParams)

;; this isn't working!!
;;(py.. plt (rc [:font.serif "Ubuntu" :font.size: 15]))
;;(py.. plt (rc ["font.serif" "Ubuntu" "font.size" 15]))

;; can't get this to work
;; (py. plt rcParams :font.serif "Ubuntu")

(def size (py.. plt (figaspect 1.68)))
(def fig (py.. plt (figure :figsize size :dpi 400)))

(py.. plt (bar ["Mon", "Tues", "Wed", "Thur", "Fri"] [3, 5, 4, 4, 7] :color "#5c00e6"))

(py. plt xlabel "Week days")
(py. plt ylabel "Fruit eaten")
(py. plt title "Fruit eaten by days")

(py. plt savefig "clojure-example.png" :bbox_inches "tight")

Note

The part I couldn't get working is accessing the rcParams Python object - if anyone knows how to do that let me know.

The call to set-up the figure is interesting - I think it's working - here we're passing keyword arguments as clojure keywords and they are being mapped for us - so in python the call to the method figure is being called with dpi=400. That's really neat!

The final result is two nice looking plots!

Final Thoughts

It looks like libpython-clj is changing pretty rapidly, some of syntactic sugar was only added in the last few days. Personally, it's exciting to be able to call Python libraries that I understand easily as I learn and play with Clojure.

I couldn't find that many resources on how to use libpython-clj, the most useful ones are at the start of the post. Alot of the interest is being driven by Python's data science and ML capabilities. If that's your area then Panthera looks really intersting.


Posted in Tech Thursday 20 February 2020
Tagged with clojure tech