Emacs: Managing Python virtual environments

Wednesday 17 July 2024 09:07 PM (Dhaka)

elpy is a great minor-mode for python-mode (major-mode) in Emacs, but managing multiple venvs (virtual environments) has always been tricky as pyvenv (that elpy uses) doesn't have a way to have buffer-local venv (virtual environment) -- any venv activated using e.g. pyvenv-activate/pyvenv-workon is (process) global.

Below is a multi-step approach I use to allow for auto-enabling venvs based on the visiting Python file (and it's dir-local/file-local/buffer-local variables).

Step 1:

A tiny elisp recipe to make switching between venvs easier when working on multiple projects. The recipe assumes:

the concept of "generic" venvs i.e. all the virtual environments that are not related to any particular project
all the generic venvs are stored in a python_<version> directory under a common base directory e.g. Python 3.11 venv directory could be /home/user/generic_venvs/python_3.11 and for Python 3.12 that could be /home/user/generic_venvs/python_3.12. As seen, only thing different in the two venv paths is the Python version number (3.11 and 3.12, respectively); in other words, the prefix is exactly the same in both cases -- /home/user/generic_venvs/python_.

The recipe itself:

(defun local-python-venv-activate (&optional emacs-python-venv generic-venv-dir-prefix generic-venv-python-version only-return)
  "Activate and return the path to venv root to be used in `python-mode'.

EMACS-PYTHON-VENV is either a full path to the venv root or a partial string like
<major>.<minor> e.g. 3.12. In the latter case generic venv for that Python version
would be used. The generic venv base directory should contain the different Python
versions e.g. `/path/to/base_dir/python_3.12'; the GENERIC-VENV-DIR-PREFIX in this
case would be `/path/to/base_dir/python_', and the GENERIC-VENV-PYTHON-VERSION
would be `3.12'. If EMACS-PYTHON-VENV is not passed, the env var `EMACS_PYTHON_VENV'
 is tried.  Otherwise, Python 3.11 is used as the default.

If the value of ONLY-RETURN is non-nil, only return the path to venv root without
activating it.

This doesn't activate the target venv if it's already the activated one."

  (interactive)
  (let ((emacs-python-venv (or emacs-python-venv (getenv "EMACS_PYTHON_VENV")))
        (generic-venv-dir-prefix (or generic-venv-dir-prefix "/home/user/generic_venvs/python_"))
        (generic-venv-python-version (or generic-venv-python-version "3.11")))

    (cond ((not emacs-python-venv)
           (setq emacs-python-venv (concat generic-venv-dir-prefix generic-venv-python-version)))
          ((string-match-p "^[[:digit:]]+\\.[[:digit:]]+$" emacs-python-venv)
           (setq emacs-python-venv (concat generic-venv-dir-prefix emacs-python-venv))))

    (cond
     (pyvenv-virtual-env
       (string-match "^\\(.+?\\)/?\\'" pyvenv-virtual-env)
       (setq pyvenv-virtual-env-without-trailing-/ (match-string 1 pyvenv-virtual-env)))
     (t
      (setq pyvenv-virtual-env-without-trailing-/ "")))

    (string-match "^\\(.+?\\)/?\\'" emacs-python-venv)
    (setq emacs-python-venv-without-trailing-/ (match-string 1 emacs-python-venv))

    (if (string= pyvenv-virtual-env-without-trailing-/ emacs-python-venv-without-trailing-/)
        (message "Already activated local Python venv: %s" emacs-python-venv)
      (unless only-return
        (message "Activating local Python venv: %s" emacs-python-venv)
        (pyvenv-activate emacs-python-venv)))

    emacs-python-venv))

Step 2:

I've added the above as an :after python-mode advice as I have many other stuff to do after a python-mode buffer is created. But adding as a hook in the python-mode-hook list would be fine as well (actually preferable in most cases).

There's a caveat though -- as the elpy-shell-send-region-or-buffer (C-c C-c) command opens up a new inferior python process (inferior-python-mode) so the above function would run before the inferior process is created, if added as a python-mode-hook or in a python-mode :after advice function. To tackle this, we can run the step 1's local-python-venv-activate function when the current major mode is not inferior-python-mode (or any mode derived from it):

(unless (derived-mode-p 'inferior-python-mode)
        (local-python-venv-activate))

To use in a hook function, we need to wrap the above in an anonymous (lambda) or named function (recommended) e.g.:

(defun local-python-venv-activate-hook ()
  (unless (derived-mode-p 'inferior-python-mode)
          (local-python-venv-activate)))

(add-hook 'python-mode-hook #'local-python-venv-activate-hook)

As I already have the :after python-mode advice function, I've put the check there, instead.

Step 3:

Now, time to deal with the common Python packages that are needed in all of the Python projects.

I tend to have the default generic venv activated always when I am playing around. For specific projects, I create a virtual environment in a directory named venv in the project root. Afterwards, I create a .dir-locals.el file in the project root to evaluate the above local-python-venv-activate function with the venv dir as argument to activate it.

For example, if the project's root dir is /home/user/project/, the following can be added to the .dir-locals.el i.e. in the /home/user/project/.dir-locals.el to have /home/user/project/venv as the venv dir for the project:

((python-mode (eval . (let ((this-file (buffer-file-name)))
                        (string-match "^\\(/.*\\)/[^/]+$" this-file)
                        (local-python-venv-activate (concat (match-string-no-properties 1 (buffer-file-name)) "/venv"))))))

As variables/forms defined in .dir-locals.el applies to all files in the directory and all sub-directories recursively (unless there's any overriding file-local/dir-local declarations in the files/subdirs), the above will set the virtual environment to /home/user/project/venv for any file visiting buffer opened in python-mode i.e. any .py file in the /home/user/project/ dir and any subdirs.

(The .dir-locals.el can be added to the global .gitignore to avoid adding personal preferences to the project repository.)

Step 4:

I've created a separate venv for packages not related to any particular project i.e. applicable for all projects; these packages are mostly related to debugging, formatting stuff. This venv contains the following packages:

black
elpy
flake8
ipdb
ipython
isort
jedi
mypy

As we can't have two venvs activated at the same time, I've used the PYTHONPATH env var to point to this common venv. In my ~/.emacs/init.el, I have the following:

(setenv "PYTHONPATH" "/path/to/common/venv/lib/python3.11/site-packages")

This would add /path/to/common/venv/lib/python3.11/site-packages to Python's module search path -- sys.path.

Step 5:

So far so good! But there is still another important issue that needs attention -- when switching between existing buffers with different virtual environments, Emacs keeps the last activated virtual environment regardless of the desired venv for the current buffer.

To fix this, we can leverage the window hooks, more specifically the window-selection-change-functions list (an abnormal hook) -- any function added to this list would be run whenever buffer is changed in any window inside any frame.

Here's something I came up with:

(defun local-python-set-correct-venv-on-buffer-buffer-change (_frame)
  "Activate the venv applicable for the current (existing) `python-mode' buffer."

  (let* ((current-buffer (get-buffer (buffer-name)))
         (current-buffer-major-mode (buffer-local-value 'major-mode current-buffer)))
    (when (equal current-buffer-major-mode 'python-mode)
      (cond
       ;; Dir-local venv
       ((string-match-p "[[:blank:](]local-python-venv-activate[[:blank:])]" (format "%s" dir-local-variables-alist))
        (hack-local-variables-apply))
       ;; Generic venv
       (t
        (local-python-venv-activate))))))

Adding the above to the window-selection-change-functions list:

(add-to-list 'window-selection-change-functions #'local-python-set-correct-venv-on-window-buffer-change)

Phew! That's a lot of stuff but my Python coding experience is way better now as If don't need to worry about which venv I'm working on and which one needs to be activated.

Bonus:

Adding the currently active venv info on the mode line would obviously be a great addition to this workflow -- pyvenv already does this by leveraging the mode-line-misc-info alist. It shows the parent directory name of the current venv directory in the mode-line. For example, if the currently active venv dir is /home/user/venv/, it would show [venv] in the mode line; similarly if the venv dir is /home/user/venv/python_3.12/, it would show [python_3.12]. It was a bit verbose for me as all of my venvs are structured similarly and I want to see only the version number there i.e. [3.12] instead of [python_3.12] in the last case.

Here's a elisp function to show the shorter version (if venv dir name follows the pattern):

(defun local-python-venv-short-version-mode-line-update ()
  "Show the current Python venv as a `[version]' string in mode-line.

This uses the `mode-line-misc-info' alist to show the current venv. The venv
is shown as only the Python version number enclosed by square brackets; for
example, [3.12].

This assumes that the venv dir name ends in a `_<version>` substring e.g.
`/home/user/venv/python_3.12/`. If not, it would show the last dir name
of the venv path. For example, if the venv dir is `/home/user/venv/python_3.12/`,
it would show `[3.12]` in the mode-line, whereas if the venv dir is
`/home/user/venv/`, it would show `[venv]` instead.

The pyvenv-mode's mode-line display of the venv is disabled first by removing
from the `mode-line-misc-info' alist."
  (let ((mode-line-venv-alist-key ""))
    (cond
     ((derived-mode-p 'python-mode)
      (setq mode-line-misc-info (assoc-delete-all 'pyvenv-mode mode-line-misc-info))
      (when pyvenv-virtual-env
        (setq mode-line-misc-info (assoc-delete-all mode-line-venv-alist-key mode-line-misc-info))
        (add-to-list 'mode-line-misc-info (list mode-line-venv-alist-key (concat "[" (car (last (string-split pyvenv-virtual-env "[/_]+" t "/+"))) "] ")))))
     (t
      (setq mode-line-misc-info (assoc-delete-all mode-line-venv-alist-key mode-line-misc-info))))))

Adding this to the pyvenv-post-activate-hooks should do:

(add-hook 'pyvenv-post-activate-hooks #'local-python-venv-short-version-mode-line-update)

Note:

pyvenv does have a tracking-mode (pyvenv-tracking-mode) but that uses post-command-hook for tracking. As the post-command-hooks are run for any input event, I try to avoid it whenever possible.

References:

Readul Hasan Chayan [Heemayl]

Step 1:

Step 2:

Step 3:

Step 4:

Step 5:

Comments