Emacs: Managing Python virtual environments
elpy
is a great minor-mode for python-mode
(major-mode) in Emacs, but managing multiple venvs (virtual environments) has always been tricky as pyvenv
(that elpy
uses) doesn't have a way to have buffer-local venv (virtual environment) -- any venv activated using e.g. pyvenv-activate
/pyvenv-workon
is (process) global.
Below is a multi-step approach I use to allow for auto-enabling venvs based on the visiting Python file (and it's dir-local/file-local/buffer-local variables).
Step 1:
A tiny elisp
recipe to make switching between venvs easier when working on multiple projects. The recipe assumes:
- the concept of "generic" venvs i.e. all the virtual environments that are not related to any particular project
- all the generic venvs are stored in a
python_<version>
directory under a common base directory e.g. Python 3.11 venv directory could be/home/user/generic_venvs/python_3.11
and for Python 3.12 that could be/home/user/generic_venvs/python_3.12
. As seen, only thing different in the two venv paths is the Python version number (3.11
and3.12
, respectively); in other words, the prefix is exactly the same in both cases --/home/user/generic_venvs/python_
.
The recipe itself:
(defun local-python-venv-activate (&optional emacs-python-venv generic-venv-dir-prefix generic-venv-python-version only-return) "Activate and return the path to venv root to be used in `python-mode'. EMACS-PYTHON-VENV is either a full path to the venv root or a partial string like <major>.<minor> e.g. 3.12. In the latter case generic venv for that Python version would be used. The generic venv base directory should contain the different Python versions e.g. `/path/to/base_dir/python_3.12'; the GENERIC-VENV-DIR-PREFIX in this case would be `/path/to/base_dir/python_', and the GENERIC-VENV-PYTHON-VERSION would be `3.12'. If EMACS-PYTHON-VENV is not passed, the env var `EMACS_PYTHON_VENV' is tried. Otherwise, Python 3.11 is used as the default. If the value of ONLY-RETURN is non-nil, only return the path to venv root without activating it." (interactive) (let ((emacs-python-venv (or emacs-python-venv (getenv "EMACS_PYTHON_VENV"))) (generic-venv-dir-prefix (or generic-venv-dir-prefix "/home/user/generic_venvs/python_")) (generic-venv-python-version (or generic-venv-python-version "3.11"))) (cond ((not emacs-python-venv) (setq emacs-python-venv (concat generic-venv-dir-prefix generic-venv-python-version))) ((string-match-p "^[[:digit:]]+\\.[[:digit:]]+$" emacs-python-venv) (setq emacs-python-venv (concat generic-venv-dir-prefix emacs-python-venv)))) (unless only-return (message "Activating local Python venv: %s" emacs-python-venv) (pyvenv-activate emacs-python-venv)) emacs-python-venv))
Step 2:
I've added the above as an :after
python-mode
advice as I have many other stuff to do after a python-mode
buffer is created. But adding as a hook in the python-mode-hook
list would be fine as well (actually preferable in most cases).
There's a caveat though -- as the elpy-shell-send-region-or-buffer
(C-c C-c
) command opens up a new inferior python process (inferior-python-mode
) so the above function would run before the inferior process is created, if added as a python-mode-hook
or in a python-mode
:after
advice function. To tackle this, we can run the step 1's local-python-venv-activate
function when the current major mode is not inferior-python-mode
(or any mode derived from it):
(unless (derived-mode-p 'inferior-python-mode) (local-python-venv-activate))
To use in a hook function, we need to wrap the above in an anonymous (lambda
) or named function (recommended) e.g.:
(defun local-python-venv-activate-hook () (unless (derived-mode-p 'inferior-python-mode) (local-python-venv-activate))) (add-hook 'python-mode-hook #'local-python-venv-activate-hook)
As I already have the :after
python-mode
advice function, I've put the check there, instead.
Step 3:
Now, time to deal with the common Python packages that are needed in all of the Python projects.
I tend to have the default generic venv activated always when I am playing around. For specific projects, I have the venv path as the value of env var EMACS_PYTHON_VENV
in the project root's .dir-locals.el
. By default, the env vars are process-global but as Emacs saves the env vars in the list variable process-environment
, we can make this variable buffer-local and then set the env var for the project. Eventually, as all the variables declared in .dir-locals.el
are dir-local, which in-turn makes them file-local and buffer-local -- the env var EMACS_PYTHON_VENV
would only be used for all the python-mode
buffers in the current project (when declared at the project-root dir).
For example, if the project's root dir is /home/user/project/
, the following can be added to the .dir-locals.el
i.e. in the /home/user/project/.dir-locals.el
to have /home/user/project/venv
as the venv dir for the project:
((python-mode . ((eval . (progn (make-local-variable 'process-environment) (setq process-environment (copy-sequence process-environment)) (setenv "EMACS_PYTHON_VENV" "/home/user/project/venv"))))))
As variables/forms defined in .dir-locals.el
applies to all files in the directory and all sub-directories recursively (unless there's any overriding file-local/dir-local declarations in the files/subdirs), the above will set the env var EMACS_PYTHON_VENV
TO /home/user/project/venv
for any file visiting buffer opened in python-mode
i.e. any .py
file in the /home/user/project/
dir and any subdirs.
(The .dir-locals.el
should be added to the global .gitignore
to avoid adding personal preferences to the project repository.)
Step 4:
I've created a separate venv for packages not related to any particular project i.e. applicable for all projects; these packages are mostly related to debugging, formatting stuff. This venv contains the following packages:
black elpy flake8 ipdb ipython isort jedi mypy
As we can't have two venvs activated at the same time, I've used the PYTHONPATH
env var to point to this common venv. In my ~/.emacs/init.el
, I have the following:
(setenv "PYTHONPATH" "/path/to/common/venv/lib/python3.11/site-packages")
This would add /path/to/common/venv/lib/python3.11/site-packages
to Python's module search path -- sys.path
.
Step 5:
So far so good! But there is still another important issue that needs attention -- when switching between existing buffers with different virtual environments, Emacs keeps the last activated virtual environment regardless of the desired venv for the current buffer.
To fix this, we can leverage the window hooks, more specifically the window-selection-change-functions
list (an abnormal hook) -- any function added to this list would be run whenever buffer is changed in any window inside any frame.
Here's something I came up with:
(defun local-python-set-correct-venv-on-window-buffer-change (_frame) "Activate the venv applicable for the current (existing) `python-mode' buffer." (let* ((current-buffer (get-buffer (buffer-name))) (current-buffer-major-mode (buffer-local-value 'major-mode current-buffer))) (when (equal current-buffer-major-mode 'python-mode) ;; Don't re-activate the same venv (unless (string= (string-trim (or pyvenv-virtual-env "") nil "/+") (string-trim (local-python-venv-activate nil nil nil t) nil "/+")) ;; Kill all inferior Python shell buffer processes (no need to kill the whole buffer) (let ((inferior-python-buffer-regex-pattern "^\\*Python\\(?:\\[[^]]*]\\)?\\*$") (inferior-python-buffer-process nil)) (dolist (buffer (buffer-list)) (when (string-match-p inferior-python-buffer-regex-pattern (buffer-name buffer)) (setq inferior-python-buffer-process (get-buffer-process buffer)) (when inferior-python-buffer-process (kill-process inferior-python-buffer-process))))) (local-python-venv-activate)))))
Adding the above to the window-selection-change-functions
list:
(add-to-list 'window-selection-change-functions #'local-python-set-correct-venv-on-window-buffer-change)
Phew! That's a lot of stuff but my Python coding experience is way better now as If don't need to worry about which venv I'm working on and which one needs to be activated.
Bonus:
Adding the currently active venv info on the mode line would obviously be a great addition to this workflow -- pyvenv
already does this by leveraging the mode-line-misc-info
alist. It shows the parent directory name of the current venv directory in the mode-line. For example, if the currently active venv dir is /home/user/venv/
, it would show [venv]
in the mode line; similarly if the venv dir is /home/user/venv/python_3.12/
, it would show [python_3.12]
. It was a bit verbose for me as all of my venvs are structured similarly and I want to see only the version number there i.e. [3.12]
instead of [python_3.12]
in the last case.
Here's a elisp
function to show the shorter version (if venv dir name follows the pattern):
(defun local-python-venv-short-version-mode-line-update () "Show the current Python venv as a `[version]' string in mode-line. This uses the `mode-line-misc-info' alist to show the current venv. The venv is shown as only the Python version number enclosed by square brackets; for example, [3.12]. This assumes that the venv dir name ends in a `_<version>` substring e.g. `/home/user/venv/python_3.12/`. If not, it would show the last dir name of the venv path. For example, if the venv dir is `/home/user/venv/python_3.12/`, it would show `[3.12]` in the mode-line, whereas if the venv dir is `/home/user/venv/`, it would show `[venv]` instead. The pyvenv-mode's mode-line display of the venv is disabled first by removing from the `mode-line-misc-info' alist." (let ((mode-line-venv-alist-key "")) (cond ((derived-mode-p 'python-mode) (setq mode-line-misc-info (assoc-delete-all 'pyvenv-mode mode-line-misc-info)) (when pyvenv-virtual-env (setq mode-line-misc-info (assoc-delete-all mode-line-venv-alist-key mode-line-misc-info)) (add-to-list 'mode-line-misc-info (list mode-line-venv-alist-key (concat "[" (car (last (string-split pyvenv-virtual-env "[/_]+" t "/+"))) "] "))))) (t (setq mode-line-misc-info (assoc-delete-all mode-line-venv-alist-key mode-line-misc-info))))))
Adding this to the pyvenv-post-activate-hooks
should do:
(add-hook 'pyvenv-post-activate-hooks #'local-python-venv-short-version-mode-line-update)
Note:
pyvenv
does have a tracking-mode (pyvenv-tracking-mode
) but that uses post-command-hook
for tracking. As the post-command-hook
s are run for any input event, I try to avoid it whenever possible.
References:
elpy
python-mode
pyvenv
- pyvenv doesn't support buffer-local venv
.dir-locals.el
process-environment
.gitignore
PYTHONPATH
- Emacs init file
sys.path
window-selection-change-functions
- Emacs advice
inferior-python-mode
elpy-shell-send-region-or-buffer
- Emacs mode line
pyvenv-tracking-mode
post-command-hook
mode-line-misc-info
pyvenv
adds the current venv dir tomode-line-misc-info
alistpyvenv-post-activate-hooks
- Emacs abnormal hooks
Comments
Comments powered by Disqus