I know people outside the US 🙄 at this, but please stop releasing major security updates and backward incompatible changes over major US, international, and religious holidays.

Given that major security updates are embargoed and scheduled weeks and months in advance, it’s essential to coordinate and avoid conflicts. A simple check of the calendar before scheduling announcements can prevent such issues.

Even if you give everyone two weeks' notice, aka what GitHub just did, wait to schedule them for release over a holiday weekend.

Historically, the Python and Django communities have also been guilty of this, so I’m not just finger-pointing at GitHub. We can all do better here.

Update: 100% unrelated to this: Django security releases issued: 5.1.1, 5.0.9, and 4.2.16. Thank you, Natalia (and Sarah) for scheduling this after the US is back from a major holiday.

Django 5.1 was released, and I was reminded of the article I wrote earlier this year about Choosing the Right Python and Django Versions for Your Projects.

While I encouraged you to wait until the second, third, or even fourth patch release of Django and Python before upgrading, I received a bit of pushback. One interesting perspective claimed that if everyone waits to upgrade, we don’t find critical bugs until the later versions. While that may be plausible, I don’t believe that the dozens of people who read my blog will be swayed by my recommendation to wait for a few patch releases.

I could have emphasized the potential risks of not testing early. Please start testing during the alpha and release candidate phase so that when Django 5.1 is released, your third-party applications will be ready and working on launch day, minimizing the risk of last-minute issues.

Today, I tried to upgrade Django Packages to run on Django 5.1 to see if our test suite would run on Django 5.1, and it very quickly failed in CI due to at least one package not supporting 5.1 yet. Even if it had passed, I’m 90% sure another package would have failed because that’s the nature of running a new major Django or Python release on day one. Even if the third-party package is ready, the packaging ecosystem needs time to catch up.

Which version of Django should I use today?

I’m sticking with Django 5.0 until Django 5.1’s ecosystem has caught up. I plan to update the third-party packages I help maintain to have Django 5.1 support. After a few patch releases of Django 5.1 have come out and the ecosystem has time to catch up, I will try to migrate again.

Which version of Python should I use today?

I’m starting new projects on Python 3.12, with a few legacy projects still being done on Python 3.11. While I am adding Django 5.1 support, I plan to add Python 3.13 support in my testing matrixes to prepare everything for Python 3.13’s release this fall.

Office hours

I plan to spend some of my Office Hours this week working on Django 5.1 and Python 3.13 readiness for projects I maintain. Please join me if you have a project to update or would like some light-hearted banter to end your week.

Renaming a table in Django seems more complex than it is. Last week, a client asked me how much pain it might be to rename a Django model from Party to Customer. We already used the model’s verbose_name, so it has been referencing the new name for months.

Renaming the model should be as easy as renaming the model while updating any foreign key and many-to-many field references in other models and then running Django’s make migrations sub-command to see where we are at.

The main issue with this approach is that Django will attempt to create a new table first, update model references, and then drop the old table.

Unfortunately, Django will either fail mid-way through this migration and roll the changes back or even worse, it may complete the migration only for you to discover that your new table is empty.

Deleting data is not what we want to happen.

As it turns out, Django supports a RenameModel migration option, but it did not prompt me to ask if we wanted to rename Party to Customer.

I am also more example-driven, and the Django docs don’t have an example of how to use RenameModel. Thankfully, this migration operation is about as straightforward as one can imagine: class RenameModel(old_model_name, new_model_name)

I re-used the existing migration file that Django created for me. I dropped the CreateModel and DeleteModel operations, added a RenameField operation, and kept the RenameField operations which resulted in the following migration:

from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ('resources', '0002_alter_party_in_the_usa'),
    ]

    operations = [
        migrations.RenameModel('Party', 'Customer'),
        migrations.RenameField('Customer', 'party_number', 'customer_number'),
        migrations.RenameField('AnotherModel', 'party', 'customer'),
    ]

The story’s moral is that you should always check and verify that your Django migrations will perform as you expect before running them in production. Thankfully, we did, even though glossing over them is easy.

I also encourage you to dive deep into the areas of the Django docs where there aren’t examples. Many areas of the docs may need examples or even more expanded docs, and they are easy to gloss over or get intimidated by.

You don’t have to be afraid to create and update your migrations by hand. After all, Django migrations are Python code designed to give you a jumpstart. You can and should modify the code to meet your needs. Migration Operations have a clean API once you dig below the surface and understand what options you have to work with.

Yes, Django Extensions package is worth installing, especially for its show_urls command, which can be very useful for debugging and understanding your project’s URL configurations.

Here’s a short example of how to use it because I sometimes want to include a link to the Django Admin in a menu for staff users, and I am trying to remember what name I need to reference to link to it.

First, you will need to install it via:

pip install django-extensions

# or if you prefer using uv like me:
uv pip install django-extensions

Next, you’ll want to add django_extensions to your INSTALLED_APPS in your settings.py file:

INSTALLED_APPS = [
    ...
    "django_extensions",
]

Finally, to urn the show_urls management command you may do some by running your manage.py script and passing it the following option:

$ python -m manage show_urls

Which will give this output:

$ python -m manage show_urls | grep admin
...
/admin/	django.contrib.admin.sites.index	admin:index
/admin/<app_label>/	django.contrib.admin.sites.app_index	admin:app_list
/admin/<url>	django.contrib.admin.sites.catch_all_view
# and a whole lot more...

In this case, I was looking for admin:index which I can now add to my HTML document this menu link/snippet:

... 
<a href="{% url 'admin:index' %}">Django Admin</a>
... 

What I like about this approach is that I can now hide or rotate the url pattern I’m using to get to my admin website, and yet Django will always link to the correct one.

In March, I wrote about my robots.txt research and how I started proactively and defensively blocking AI Agents in my 🤖 On Robots.txt. Since March, I have updated my Django projects to add more robots.txt rules.

Earlier this week, I ran across this Blockin’ bots. blog post and this example, the mod_rewrite rule blocks AI Agents via their User-Agent strings.

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /
# block “AI” bots
RewriteCond %{HTTP_USER_AGENT} (AdsBot-Google|Amazonbot|anthropic-ai|Applebot|AwarioRssBot|AwarioSmartBot|Bytespider|CCBot|ChatGPT|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|DataForSeoBot|Diffbot|FacebookBot|FacebookBot|Google-Extended|GPTBot|ImagesiftBot|magpie-crawler|omgili|Omgilibot|peer39_crawler|PerplexityBot|YouBot) [NC]
RewriteRule ^ – [F]
</IfModule>

Since none of my projects use Apache, and I was short on time, I decided to leave this war to the bots.

Django Middleware

I asked ChatGPT to convert this snippet to a piece of Django Middleware called Super Bot Fight. After all, if we don’t have time to keep up with bots, then we could leverage this technology to help fight against them.

In theory, this snippet passed my eyeball test and was good enough:

# middleware.py

from django.http import HttpResponseForbidden

# List of user agents to block

BLOCKED_USER_AGENTS = [
    "AdsBot-Google",
    "Amazonbot",
    "anthropic-ai",
    "Applebot",
    "AwarioRssBot",
    "AwarioSmartBot",
    "Bytespider",
    "CCBot",
    "ChatGPT",
    "ChatGPT-User",
    "Claude-Web",
    "ClaudeBot",
    "cohere-ai",
    "DataForSeoBot",
    "Diffbot",
    "FacebookBot",
    "Google-Extended",
    "GPTBot",
    "ImagesiftBot",
    "magpie-crawler",
    "omgili",
    "Omgilibot",
    "peer39_crawler",
    "PerplexityBot",
    "YouBot",
]

class BlockBotsMiddleware:

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        # Check the User-Agent against the blocked list
        user_agent = request.META.get("HTTP_USER_AGENT", "")
        if any(bot in user_agent for bot in BLOCKED_USER_AGENTS):
            return HttpResponseForbidden("Access denied")
        response = self.get_response(request)
        return response

To use this middleware, you would update your Django settings.py to add it to your MIDDLEWARE setting.

# settings.py

MIDDLEWARE = [
    ...
    "middleware.BlockBotsMiddleware",
    ...
]

Tests?

If this middleware works for you and you care about testing, then these tests should also work:

import pytest

from django.http import HttpRequest
from django.test import RequestFactory

from middleware import BlockBotsMiddleware

@pytest.mark.parametrize("user_agent, should_block", [
    ("AdsBot-Google", True),
    ("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", False),
    ("ChatGPT-User", True),
    ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3", False),
])
def test_user_agent_blocking(user_agent, should_block):
    # Create a request factory to generate request instances
    factory = RequestFactory()
    request = factory.get('/', HTTP_USER_AGENT=user_agent)

    # Middleware setup
    middleware = BlockBotsMiddleware(get_response=lambda request: HttpResponse())
    response = middleware(request)

    # Check if the response should be blocked or allowed
    if should_block:
        assert response.status_code == 403, f"Request with user agent '{user_agent}' should be blocked."
    else:
        assert response.status_code != 403, f"Request with user agent '{user_agent}' should not be blocked."

Enhancements

To use this code in production, I would normalize the user_agent and BLOCKED_USER_AGENTS variables to be case-insensitive.

I would also consider storing my list of user agents in a Django model or using a project like django-robots instead of a hard-coded Python list.

One of Django’s most powerful features is the ORM, which includes a robust migration framework. One of Django’s most misunderstood features is Django migrations because it just works 99% of the time.

Even when working solo, Django migrations are highly reliable, working 99.9% of the time and offering better uptime than most web services you may have used last week.

The most common stumbling block for developers of all skill levels is rolling back a Django migration and prepping a pull request for review.

I’m not picky about pull requests or git commit history because I default to using the “Squash and merge” feature to turn all pull request commits into one merge commit. The merge commit tells me when, what, and why something changed if I need extra context.

I am pickier about seeing >2 database migrations for any app unless a data migration is involved. It’s common to see 4 to 20 migrations when someone works on a database feature for a week. Most of the changes tend to be fiddly, where someone adds a field, renames the field, renames it again, and then starts using it, which prompts another null=True change followed by a blank=True migration.

For small databases, none of this matters.

For a database with 10s or 100s of millions of records, these small changes can cause minutes of downtime per migration, which amounts to a throwaway change. While there are ways to mitigate most migration downtime situations, that’s different from my point today.

I’m also guilty of being fiddly with my Django model changes because I know I can delete and refactor them before requesting approval. The process I use is probably worth sharing because once every new client comes up.

Let’s assume I am working on Django News Jobs, and I am looking over my pull request one last time before I ask someone to review it. That’s when I noticed four migrations that could quickly be rebuilt into one, starting with my 0020* migration in my jobs app.

The rough steps that I would do are:

# step 1: see the state of our migrations
$ python -m manage showmigrations jobs
jobs
 [X] 0001_initial
 ...
 [X] 0019_alter_iowa_versus_unconn
 [X] 0020_alter_something_i_should_delete
 [X] 0021_alter_uconn_didnt_foul
 [X] 0022_alter_nevermind_uconn_cant_rebound
 [X] 0023_alter_iowa_beats_uconn
 [X] 0024_alter_south_carolina_sunday_by_four

# step 2: rollback migrations to our last "good" state
$ python -m manage migrate jobs 0019

# step 3: delete our new migrations
$ rm jobs/migrations/002*

# step 4: rebuild migrations 
python -m manage makemigrations jobs 

# step 5: profit 
python -m manage migrate jobs

95% of the time, this is all I ever need to do.

Occasionally, I check out another branch with conflicting migrations, and I’ll get my local database in a weird state.

In those cases, check out the --fake (“Mark migrations as run without actually running them.") and --prune (“Delete nonexistent migrations from the django_migrations table.") options. The fake and prune operations saved me several times when my django_migrations table was out of sync, and I knew that SQL tables were already altered.

What not squashmigrations?

Excellent question. Squashing migrations is wonderful if you care about keeping every or most of the operations each migration is doing. Most of the time, I do not, so I overlook it.

The django-waffle feature flag library is helpful for projects where we want to release and test new features in production and have a controlled rollout. I also like using feature flags for resource-intensive features on a website that we want to toggle off during high-traffic periods. It’s a nice escape hatch to fall back on if we need to turn off a feature and roll out a fix without taking down your website.

While Waffle is a powerful tool, I understand the challenge of keeping track of feature flags in both code and the database. It’s a pain point that many of us have experienced.

Waffle has a WAFFLE_CREATE_MISSING_FLAGS=True setting that we can use to tell Waffle to create any missing flags in the database should it find one. While this helps discover which flags our application is using, we need to figure out how to clean up old flags in the long term.

The pattern I landed on combines storing all our known feature flags and a note about what they do in our main settings file.

# settings.py
... 

WAFFLE_CREATE_MISSING_FLAGS=True

WAFFLE_FEATURE_FLAGS = {
   "flag_one": "This is a note about flag_one",
   "flag_two": "This is a note about flag_two",
}

We will use a management command to sync every feature flag we have listed in our settings file, and then we will clean up any missing feature flags.

# management/commands/sync_feature_flags.py
import djclick as click

from django.conf import settings
from waffle.models import Flag


@click()
def command():
    # Create flags that don't exist
    for name, note in settings.WAFFLE_FEATURE_FLAGS.items():
        flag, created = Flag.objects.update_or_create(
            name=name, defaults={"note": note}
        )
        if created:
            print(f"Created flag {name} ({flag.pk})")

    # Delete flags that are no longer registered in settings
    for flag in Flag.objects.exclude(name__in=settings.FEATURE_FLAGS.keys()):
        flag.delete()
        print(f"Deleted flag {flag.name} ({flag.pk})")

We can use the WAFFLE_CREATE_MISSING_FLAGS settings as a failsafe to create any flags we might have accidently missed. They will stick out because they will not have a note associated with them.

This pattern is also helpful in solving similar problems for scheduled tasks, which might also store their schedules in the database.

Check out this example in the Django Styleguide for how to sync Celery’s scheduled tasks.

Upgrade Django is a REVSYS project we created six years ago and launched three years ago.

The goal of Upgrade Django was to create a resource that made it easy to see at a glance which versions of the Django web framework are maintained and supported. We also wanted to catalog every release and common gotchas and link to helpful information like release notes, blog posts, and the tagged git branch on GitHub.

We also wanted to make it easier to tell how long a given version of Django would be supported and what phase of its release cycle it is in.

Future features

We have over a dozen features planned, but it’s a project that primarily serves its original purpose.

One feature on my list is that I’d love to see every backward incompatible change between two Django versions. This way, if someone knows their website is running on Django 3.2, they could pick Django 4.2 or Django 5.0 version and get a comprehensive list with links to everything they need to upgrade between versions.

Projects like Upgrade Django are fun to work on because once you collect a bunch of data and start working with it, new ways of comparing and presenting the information become more apparent.

If you have ideas for improving Upgrade Django that would be useful to your needs, we’d love to hear about them.

I am several weeks into working on a project with my colleague, Lacey Henschel. Today, while reviewing one of her pull requests, I was reminded how to test a Django Signal via mocking.

Testing Django signals is valuable to me because I need help remembering how to test a signal, and even with lots of effort, it never works. So bookmark this one, friends. It works.

Thankfully, she wrote it up in one of her TIL: How I set up django-activity-stream, including a simple test

https://mastodon.social/@lacey@hachyderm.io

Python is such a fantastic glue language. Last night, while watching March Madness basketball games, I had a programming itch I wanted to scratch.

I dusted off a demo I wrote several years ago. It used Python’s subprocess module, which strings together a bunch of shell commands to perform a git checkout, run a few commands, and then commit the results. The script worked, but I struggled to get it fully working in a production environment.

To clean things up and as an excuse to try out a new third-party package, I converted the script to use:

• GitPython - GitPython is a Python library used to interact with Git repositories.

• Shelmet - A shell power-up for working with the file system and running subprocess commands.

• Django Q2 - A multiprocessing distributed task queue for Django based on Django-Q.

Using Django might have been overkill, but having a Repository model to work with felt nice. Django Q2 was also overkill, but if I put this app into production, I’ll want a task queue, and Django Q2 has a manageable amount of overhead.

GitPython was a nice improvement over calling git commands directly because their API makes it easier to see which files were modified and to check against existing branch names. I was happy with the results after porting my subprocess commands to the GitPython API.

The final package I used is a new package called Shelmet, which was both a nice wrapper around subprocess plus they have a nice API for file system operations in the same vein as Python’s Pathlib module.

Future goals

I was tempted to cobble together a GitHub bot, but I didn’t need one. I might dabble with the GitHub API more to fork a repo, but for now, this landed in a better place, so when I pick it back up again in a year, I’m starting in a good place.

If you want to write a GitHub bot, check out Mariatta’s black_out project.

Recently, I have been maintaining forks for several projects that are no longer maintained. Usually, these are a pain to update, but I have found a workflow that takes the edge off by leveraging pre-commit.

My process:

• Fork the project on GitHub to whichever organization I work with or my personal account.
• Check out a local copy of my forked copy with git.
• Install pre-commit
• Create a .pre-commit-config.yaml with ZERO formatting or lint changes. This file will only include django-upgrade and pyupgrade hooks.

We skip the formatters and linters to avoid unnecessary changes if we want to open a pull request in the upstream project. If the project isn’t abandoned, we will want to do that.

• For django-upgrade, change the—-target-version option to target the latest version of Django I’m upgrading to, which is currently 5.0.
• For pyupgrade, update the python settings under default_language_version to the latest version of Python that I’m targetting. Currently, that’s 3.12.

The django-upgrade and pyupgrade projects attempt to run several code formatters and can handle most of the more tedious upgrade steps.

• Run pre-commit autoupdate to ensure we have the latest version of our hooks.
• Run pre-commit run --all-files to run pyupgrade and django-upgrade on our project.
• Run any tests contained in the project and review all changes.
• Once I’m comfortable with the changes, I commit them all via git and push them upstream to my branch.

Example .pre-commit-config.yaml config

From my experience, less is more with this bane bones .pre-commit-config.yaml config file.

# .pre-commit-config.yaml

default_language_version:
  python: python3.12

repos:
  - repo: https://github.com/asottile/pyupgrade
    rev: v3.15.1
    hooks:
      - id: pyupgrade

  - repo: https://github.com/adamchainz/django-upgrade
    rev: 1.16.0
    hooks:
      - id: django-upgrade
        args: [--target-version, "5.0"]

If I’m comfortable that the project is abandoned, I’ll add ruff support with a more opinionated config to ease my maintenance burden going forward.