Recently, I have been slowly moving several of my side projects and client projects from various Python versions to Python 3.12.
I never see people write about this, so it might be nice to write and share some notes.
Where to start
The first thing we do with a relatively simple upgrade is figure out what Python version we use. Thankfully, the project we picked mentioned in the README.md that it was using Python 3.11.
Once we know which version of Python we are using, we can open up iTerm and get a git checkout of the project.
Next, we will run git grep 11, where “11” is the shortened form of the Python version that we are running. There are so many variations of 3.11 and 311 that using the minor version tends to be about right.
$ git grep 11
... really long list...
.github/workflows/actions.yml: - name: Set up Python 3.11
.github/workflows/actions.yml: python-version: '3.11'
.github/workflows/actions.yml: - name: Set up Python 3.11
.github/workflows/actions.yml: python-version: '3.11'
.pre-commit-config.yaml: python: python311
.pre-commit-config.yaml: args: [--py311-plus]
README.md:This project will use Python 3.11, Docker, and Docker Compose.
README.md:Make a Python 3.11.x virtualenv.
docker/Dockerfile:FROM python:3.11-slim as builder-py
docker/Dockerfile:FROM python:3.11-slim AS release
pyproject.toml:requires-python = ">= 3.11"
pyproject.toml:# Assume Python >=3.11.
pyproject.toml:target-version = "py311"
requirements.txt:# This file is autogenerated by pip-compile with Python 3.11
... lots and lots of files...
This output will give us a long list of files. Usually, this is 100s or 1000s of files we will pipe or copy into our code editor. We will make a few passes to remove all of the CSS, SVG, and HTML files in the list, and that pairs down the results to half a dozen or a dozen files.
Create a new git branch
Next, we will create a new git branch called upgrade-to-python-3.12, and we will open each file one by one, and replace every “3.11” and “311” reference with “3.12” and “312” respectively.
Lint/format our code base
Once we have all of our files updated, we will commit everything. Then we will note special files like .pre-commit-config.yaml and pyproject.toml, impacting how my Python files are linted and formatted. Then, we will run pre-commit immediately after and commit any formatting changes.
Rebuild our Docker image
Since this project contains docker/Dockerfile that tells us the project uses Docker, we will need to rebuild our container image and note anything that breaks.
Re-pin/freeze our Python dependencies
Next, we will run pip-tools compile from within our newly rebuilt Docker container to build a new requirements.txt using Python 3.12.
Re-rebuild our Docker image (again)
“Insert Xzibit Yo Dawg meme." Next, we rebuild our Docker image using the newly pinned requirements.txt file, and this should be our final image.
Did our tests pass?
Assuming Docker builds cleanly, we will run my test suite using pytest. Once our tests pass, we’ll commit any uncommitted changes, git push our branch to GitHub, and open a Pull Request for review.
Did our tests pass in CI?
If our tests pass on GitHub Actions in CI, then we know our upgrade was successful, and we are reasonably confident that
When things don’t “just work.”
If you keep up with your upgrades, most of the time, everything works. Half a dozen projects did work for me, but I had one that did not work on Monday. There was a sub-dependency issue, so I closed my branch and opened a new issue to revisit this upgrade once the next version of Python 3.12.3 is released.
Even though this wasn’t a Python 3.12.2 bug, it takes the Python ecosystem time to catch up with newer versions. Since Python 3.11 is still supported for another 3 years and 7 months (as of this writing), it won’t hurt to wait a few weeks or months and revisit these changes.
If you are curious about how I decide when to adopt a new version, I wrote about that last month: Choosing the Right Python and Django Versions for Your Projects
