Emacs: Manage Python virtual environments using `pyvenv`

Sunday 14 July 2024 04:32 PM (Dhaka)

elpy is a great minor-mode for python-mode in Emacs, but managing multiple virtual environments has always been tricky as pyvenv (that elpy uses) doesn't have a way to have buffer-local venv -- any venv activated using e.g. pyvenv-activate/pyvenv-workon is (process) global.

Below is a tinyelisp recipe to make switching between venvs easier when working on multiple projects. The recipe assumes:

the concept of "generic" venvs i.e. all the virtual environments that are not related to any particular project
all the generic venvs are stored in a python_<version> directory under a common base directory -- e.g. Python 3.11 venv directory could be /home/user/generic_venvs/python_3.11 and for Python 3.12 that should be /home/user/generic_venvs/python_3.12.

The recipe iteself:

(defun local-python-venv-activate (&optional emacs-python-venv generic-venv-dir-prefix generic-venv-python-version only-return)
  "Activate and return the path to venv root to be used in `python-mode'.

EMACS-PYTHON-VENV is either a full path to the venv root or a partial string like
<major>.<minor> e.g. 3.12. In the latter case generic venv for that Python version
would be used. The generic venv base directory should contain the different Python
versions e.g. `/path/to/base_dir/python_3.12'; the GENERIC-VENV-DIR-PREFIX in this
case would be `/path/to/base_dir/python_', and the GENERIC-VENV-PYTHON-VERSION
would be `3.12'. If EMACS-PYTHON-VENV is not passed, the env var `EMACS_PYTHON_VENV'
 is tried.  Otherwise, Python 3.11 is used as the default.

If the value of ONLY-RETURN is non-nil, only return the path to venv root without
activating it."

  (interactive)
  (let ((emacs-python-venv (or emacs-python-venv (getenv "EMACS_PYTHON_VENV")))
        (generic-venv-dir-prefix (or generic-venv-dir-prefix "/home/user/generic_venvs/python_"))
        (generic-venv-python-version (or generic-venv-python-version "3.11")))

    (cond ((not emacs-python-venv)
           (setq emacs-python-venv (concat generic-venv-dir-prefix generic-venv-python-version)))
          ((string-match-p "^[[:digit:]]+\\.[[:digit:]]+$" emacs-python-venv)
           (setq emacs-python-venv (concat generic-venv-dir-prefix emacs-python-venv))))

    (unless only-return
      (message "Activating local Python venv: %s" emacs-python-venv)
      (pyvenv-activate emacs-python-venv))
    emacs-python-venv))

I've added the above as :after python-mode advice as I have many other stuff after a python-mode is created. But adding as a hook in the python-mode-hook list would be fine as well (actually preferable in most cases).

There's a caveat though -- as the elpy-shell-send-region-or-buffer (C-c C-c) command opens up a new inferior python process (inferior-python-mode) so the above function would run if added as a python-mode-hook or in a python-mode :after advice function. To tackle this, we can run the above local-python-venv-activate function when the current major mode is not inferior-python-mode (or any mode derived from inferior-python-mode):

(unless (derived-mode-p 'inferior-python-mode)
        (local-python-venv-activate))

To use in a hook function, we need to wrap the above in a lambda or a named function (recommended) e.g.:

(defun local-python-venv-activate-hook ()
  (unless (derived-mode-p 'inferior-python-mode)
          (local-python-venv-activate)))

(add-hook 'python-mode-hook #'local-python-venv-activate-hook)

Now, I tend to have the default generic venv activated always when I am playing around. For specific projects, I have the venv path as the value of env var EMACS_PYTHON_VENV in the project root's .dir-locals.el. By default the env vars are process-global but as Emacs saves the env vars in the list variable process-environment, we can make the variable buffer-local and then set the env var. Eventually, as all the variables declared in .dir-locals.el are dir-local, which in-turn makes them file-local and buffer-local -- the env var EMACS_PYTHON_VENV would only be used for all the python-mode buffers in the current project.

For example, if the project's venv dir is /home/user/project/venv, the following can be added to the project root's .dir-locals.el i.e. in the /home/user/project/.dir-locals.el:

((python-mode . ((eval . (progn
                           (make-local-variable 'process-environment)
                           (setq process-environment (copy-sequence process-environment))
                           (setenv "EMACS_PYTHON_VENV" "3.11"))))))

As variables/forms defined in .dir-locals.el applies to all files in the directory and all sub-directories, the above set the env var EMACS_PYTHON_VENV TO 3.11 for any file visiting buffer opened in python-mode i.e. any .py file in /home/user/project/ dir and any subdir.

(The .dir-locals.el should be added to the global .gitignore to avoid adding personal preferences to the project repository.)

I've created a separate venv for packages not related to any particular project i.e. applicable for all projects; these packages are mostly related to debugging, formatting stuff. This venv contains the following packages:

black
elpy
flake8
ipdb
ipython
isort
jedi
mypy

As we can't have two venvs activated at the same time, I've used the PYTHONPATH env var to point to this common venv. In my ~/.emacs/init.el, I have the following:

(setenv "PYTHONPATH" "/path/to/common/venv/lib/python3.11/site-packages")

This would add /path/to/common/venv/lib/python3.11/site-packages to the module search path -- sys.path.

So far so good! But there is still another issue that needs attention -- when switching between existing buffers with different virtual environments, Emacs keeps the last activated virtual environment as current regardless of the desired venv for the current buffer.

To fix this, we can leverage the window-selection-change-functions list -- any function added to this list would be run whenever buffer is changed in any window inside a frame. Here's something I came up with:

(defun local-python-set-correct-venv-on-window-buffer-change (_frame)
  "Activate the venv applicable for the current (existing) `python-mode' buffer."
  (let* ((current-buffer (get-buffer (buffer-name)))
         (current-buffer-major-mode (buffer-local-value 'major-mode current-buffer)))
    (when (equal current-buffer-major-mode 'python-mode)
      ;; Don't re-activate the same venv
      (unless (string= (string-trim (or pyvenv-virtual-env "") nil "/+") (string-trim (local-python-venv-activate nil nil nil t) nil "/+"))
        ;; Kill all inferior Python shell buffer processes (no need to kill the whole buffer)
        (let ((inferior-python-buffer-regex-pattern "^\\*Python\\(?:\\[[^]]*]\\)?\\*$")
              (inferior-python-buffer-process nil))
          (dolist (buffer (buffer-list))
            (when (string-match-p inferior-python-buffer-regex-pattern (buffer-name buffer))
              (setq inferior-python-buffer-process (get-buffer-process buffer))
              (when inferior-python-buffer-process
                (kill-process inferior-python-buffer-process)))))
        (local-python-venv-activate)))))

Adding the above to the window-selection-change-functions list:

(add-to-list 'window-selection-change-functions #'local-python-set-correct-venv-on-window-buffer-change)

Phew!

If I can summarize the workflow:

Readul Hasan Chayan [Heemayl]

Comments