• Django

    🧳 DjangoCon US, Black Python Devs Leadership Summit, and Django Girls Durham

    I’m heading to Durham, NC, for seven days of DjangoCon US this Friday. This is my 10th year volunteering and the 9th year that DEFNA, the non-profit I co-founded, has run a DjangoCon US event. Here is an overview of the week.

    Black Python Devs Leadership Summit (Saturday)

    I’m attending and speaking on a discussion panel on Saturday at the Black Python Devs Leadership Summit. Tickets are free, and they will be streaming online in the afternoon. Donations are accepted and appreciated.

    Django Girls Durham (Saturday)

    Django Girls are hosting a Django workshop and teaching beginners a crash course on building their first website using Django.

    DjangoCon US Tutorials (Sunday)

    On Sunday morning, I’ll be volunteering and helping out at the tutorials. In the afternoon, we have a tradition of stuffing swag bags, which takes a big group and is a fun way to kick off the conference. You do not need a tutorial ticket or an organizer to help out. Ask at the registration desk, and they can direct you to when and where we are doing this.

    Django Social meetup (Sunday)

    My company REVSYS is sponsoring a DjangoSocial Raleigh/Durham Pre-DjangoCon Special meetup on Sunday evening before the conference kicks off. The meetup will be great for meeting other attendees the night before the conference.

    DjangoCon US Talks (Monday through Wednesday)

    The talks are great, but the busiest three days of the conference are also the busiest. There is always a lot going on, from sun up to sun down.

    DjangoCon US Sprints (Thursday and Friday)

    The sprints are one of my favorite parts of the conference. In past years, I have been so exhausted by the sprints that it’s hard to sit down and focus. It’s one of the best times to discuss Django and the Django ecosystem. If you have a project or want to find a project to help with, the sprints are great for getting your feet wet.

    Outro

    Tickets are still available if you live near Durham and want to attend. Both events have online and in-person options, so there is no pressure to make last-minute travel plans.

    If you live around Durham and want to meet up, please reach out. Let’s see if we can meet for coffee.

    Friday September 20, 2024
  • Django

    ,

    Python

    ,

    UV

    🤠 UV Roundup: Five good articles and a pre-commit tip

    I have written quite a bit about UV on my micro blog, and I am happy to see more and more people adopt it. I have stumbled on so many good articles recently that I wanted to share them because every article points out something new or different about why UV works well for them.

    If you are new to UV, it’s a new tool written by Astral, the creators of Ruff.

    I like UV because it replaces, combines, or complements a bunch of Python tools into one tool and user developer experience without forcing a UV way of doing it. UV effectively solves the question, “Why do I need another Python tool?” to do everyday Python tasks.

    Some reason I like UV after using it for months:

    • It’s a faster pip and is really, really fast
    • It can install and manage Python versions
    • It can run and install Python scripts
    • It can run single-file Python scripts along with their dependencies
    • It can handle project lock files

    While some people don’t care about UV being fast, it’s shaved minutes off my CI builds and container rebuilds, which means it has also saved me money and energy resources.

    Overall thoughts on UV

    Oliver Andrich’s UV — I am (somewhat) sold takes the approach of only using UV to set up a new Python environment. Oliver uses UV to install Python, aliases to call Python, and UV tool install to set up a few global utilities.

    Using UV with Django

    Anže Pečar’s UV with Django shows how to use UV to set up a new project with Django.

    Switching from pyenv to UV

    Will Guaraldi Kahn-Greene’s Switching from pyenv to uv was relatable for me because I also use pyenv, but I plan to slowly migrate to using only UV. I’m already halfway there, but I will have pyenv for my legacy projects for years because many aren’t worth porting yet.

    Using UV and managing with Ansible

    Adam Johnson’s Python: my new uv setup for development taught me to use uv cache prune to clean up unused cache entries and shows how he manages his UV setup using Ansible.

    Some notes on UV

    Simon Willison’s Notes on UV is an excellent summary of Oliver’s notes.

    A parting UV tip

    If you are a pre-commit fan hoping for a version that supports UV, the pre-commit-uv project does just that. I started updating my justfile recipes to bake just lint to the following uv run command, which speeds up running and installing pre-commit significantly.

    $ uv run --with pre-commit-uv pre-commit run --all-files
    pre-commit-uv
    

    If you are attending DjangoCon US…

    If you are attending DjangoCon US and want to talk UV, Django, Django News, Django Packages, hit me up while you are there.

    I’ll be attending, volunteering, organizing, sponsoring, and sprinting around the venue in Durham, NC, for the next week starting this Friday.

    We still have online and in-person tickets, but not much longer!

    Thursday September 19, 2024
  • Django

    🚜 On Evolving Django's `auth.User` model

    I need to re-read/re-listen to Carlton’s Evolving Django’s auth.User Stack Report before I have a firm opinion, but I saw his Mastodon post and I thought it was worth sharing some initial thoughts.

    On my first read, I fell into the camp that wants django-unique-user-email and full name to be the defaults on Django’s built-in User. I have wanted this forever, and my clients do, too.

    Django should still allow me to create a custom user, but the documentation could be better. Django’s documentation is a topic for another day, but I’m confused by it and find it less and less helpful. I notice the developers I work with get lost in them, too. Docs may be a good DjangoCon US topic if you want to discuss it in a few weeks.

    I would love the default User to have a unique email address, full name, and a short name. Optionally, a username is an email address setting. Please don’t take away the ability to have a custom User model. (Update: Carlton is not making a case for removing a custom User model, but others have made that argument. No, thank you.)

    Carlton’s “Action points” conclusion seems reasonable to me.

    Tuesday September 10, 2024
  • Django

    ,

    Python

    🚫 Stop scheduling security updates and deprecating major features over holidays

    I know people outside the US 🙄 at this, but please stop releasing major security updates and backward incompatible changes over major US, international, and religious holidays.

    Given that major security updates are embargoed and scheduled weeks and months in advance, it’s essential to coordinate and avoid conflicts. A simple check of the calendar before scheduling announcements can prevent such issues.

    Even if you give everyone two weeks' notice, aka what GitHub just did, wait to schedule them for release over a holiday weekend.

    Historically, the Python and Django communities have also been guilty of this, so I’m not just finger-pointing at GitHub. We can all do better here.

    Update: 100% unrelated to this: Django security releases issued: 5.1.1, 5.0.9, and 4.2.16. Thank you, Natalia (and Sarah) for scheduling this after the US is back from a major holiday.

    Tuesday September 3, 2024
  • Django

    ,

    Python

    ⬆️ Which Django and Python versions should I be using today?

    Django 5.1 was released, and I was reminded of the article I wrote earlier this year about Choosing the Right Python and Django Versions for Your Projects.

    While I encouraged you to wait until the second, third, or even fourth patch release of Django and Python before upgrading, I received a bit of pushback. One interesting perspective claimed that if everyone waits to upgrade, we don’t find critical bugs until the later versions. While that may be plausible, I don’t believe that the dozens of people who read my blog will be swayed by my recommendation to wait for a few patch releases.

    I could have emphasized the potential risks of not testing early. Please start testing during the alpha and release candidate phase so that when Django 5.1 is released, your third-party applications will be ready and working on launch day, minimizing the risk of last-minute issues.

    Today, I tried to upgrade Django Packages to run on Django 5.1 to see if our test suite would run on Django 5.1, and it very quickly failed in CI due to at least one package not supporting 5.1 yet. Even if it had passed, I’m 90% sure another package would have failed because that’s the nature of running a new major Django or Python release on day one. Even if the third-party package is ready, the packaging ecosystem needs time to catch up.

    Which version of Django should I use today?

    I’m sticking with Django 5.0 until Django 5.1’s ecosystem has caught up. I plan to update the third-party packages I help maintain to have Django 5.1 support. After a few patch releases of Django 5.1 have come out and the ecosystem has time to catch up, I will try to migrate again.

    Which version of Python should I use today?

    I’m starting new projects on Python 3.12, with a few legacy projects still being done on Python 3.11. While I am adding Django 5.1 support, I plan to add Python 3.13 support in my testing matrixes to prepare everything for Python 3.13’s release this fall.

    Office hours

    I plan to spend some of my Office Hours this week working on Django 5.1 and Python 3.13 readiness for projects I maintain. Please join me if you have a project to update or would like some light-hearted banter to end your week.

    Wednesday August 7, 2024
  • Django

    ,

    Python

    🐘 Django Migration Operations aka how to rename Models

    Renaming a table in Django seems more complex than it is. Last week, a client asked me how much pain it might be to rename a Django model from Party to Customer. We already used the model’s verbose_name, so it has been referencing the new name for months.

    Renaming the model should be as easy as renaming the model while updating any foreign key and many-to-many field references in other models and then running Django’s make migrations sub-command to see where we are at.

    The main issue with this approach is that Django will attempt to create a new table first, update model references, and then drop the old table.

    Unfortunately, Django will either fail mid-way through this migration and roll the changes back or even worse, it may complete the migration only for you to discover that your new table is empty.

    Deleting data is not what we want to happen.

    As it turns out, Django supports a RenameModel migration option, but it did not prompt me to ask if we wanted to rename Party to Customer.

    I am also more example-driven, and the Django docs don’t have an example of how to use RenameModel. Thankfully, this migration operation is about as straightforward as one can imagine: class RenameModel(old_model_name, new_model_name)

    I re-used the existing migration file that Django created for me. I dropped the CreateModel and DeleteModel operations, added a RenameField operation, and kept the RenameField operations which resulted in the following migration:

    from django.db import migrations
    
    
    class Migration(migrations.Migration):
    
        dependencies = [
            ('resources', '0002_alter_party_in_the_usa'),
        ]
    
        operations = [
            migrations.RenameModel('Party', 'Customer'),
            migrations.RenameField('Customer', 'party_number', 'customer_number'),
            migrations.RenameField('AnotherModel', 'party', 'customer'),
        ]
    

    The story’s moral is that you should always check and verify that your Django migrations will perform as you expect before running them in production. Thankfully, we did, even though glossing over them is easy.

    I also encourage you to dive deep into the areas of the Django docs where there aren’t examples. Many areas of the docs may need examples or even more expanded docs, and they are easy to gloss over or get intimidated by.

    You don’t have to be afraid to create and update your migrations by hand. After all, Django migrations are Python code designed to give you a jumpstart. You can and should modify the code to meet your needs. Migration Operations have a clean API once you dig below the surface and understand what options you have to work with.

    Monday July 15, 2024
  • Django

    🧰 More fun with Django Extensions using `shell_plus` and `graph_models`

    Yesterday, I wrote about Django Extensions show_urls management command because it’s useful. I have Mastodon posted/tooted about it [previously](https://mastodon.social/@webology/110271223054909764, but I didn’t expect it to possibly lead to it being added to Django, and yet here we are. My favorite byproduct of blogging is when someone talks about something they like, and someone asks, “What if” or “Why doesn’t?” and then they get inspired to look into it and contribute. This post might have led to one new contribution to Django. 🎉

    Several people shared that they also liked Django Extensions shell_plus and graph_models management commands.

    I don’t use shell_plus often, but I bake it into my Just workflows for clients who do. I tend to forget about it, and I spend so much time using pytest.set_trace() and testing.

    If you haven’t used graph_models, I use it in most of my client projects. I generate SVG files with it and add them to an ERD section of their docs, which helps discuss models and onboard new developers. It’s a nice-to-have feature and is a small lift with a huge payoff. This code is also easy to copy and paste from project to project.

    Sunday July 7, 2024
  • Django

    ,

    Python

    Django Extensions is useful even if you only use show_urls

    Yes, Django Extensions package is worth installing, especially for its show_urls command, which can be very useful for debugging and understanding your project’s URL configurations.

    Here’s a short example of how to use it because I sometimes want to include a link to the Django Admin in a menu for staff users, and I am trying to remember what name I need to reference to link to it.

    First, you will need to install it via:

    pip install django-extensions
    
    # or if you prefer using uv like me:
    uv pip install django-extensions
    

    Next, you’ll want to add django_extensions to your INSTALLED_APPS in your settings.py file:

    INSTALLED_APPS = [
        ...
        "django_extensions",
    ]
    

    Finally, to urn the show_urls management command you may do some by running your manage.py script and passing it the following option:

    $ python -m manage show_urls
    

    Which will give this output:

    $ python -m manage show_urls | grep admin
    ...
    /admin/	django.contrib.admin.sites.index	admin:index
    /admin/<app_label>/	django.contrib.admin.sites.app_index	admin:app_list
    /admin/<url>	django.contrib.admin.sites.catch_all_view
    # and a whole lot more...
    

    In this case, I was looking for admin:index which I can now add to my HTML document this menu link/snippet:

    ... 
    <a href="{% url 'admin:index' %}">Django Admin</a>
    ... 
    

    What I like about this approach is that I can now hide or rotate the url pattern I’m using to get to my admin website, and yet Django will always link to the correct one.

    Saturday July 6, 2024
  • Django

    🤖 More Blocking Bots with Django ❌

    Tonight, I ran across Robb Knight’s Blocking Bots with Nginx, which fits well with two pieces I have already written about.

    Check out my 🤖 Super Bot Fight 🥊 article, which develops a Django-friendly middleware for blocking AI UserAgents so they never get to your content.

    My 🤖 On Robots.txt article shared more of my research on the robots.txt standard.

    I forgot I wrote the middleware, so I rewrote it tonight before rediscovering it while looking for both articles to share. That was fun, so I’m not writing more tonight. 😬

    Friday June 14, 2024
  • Django

    🧱 Django ModelForm Template starting point

    Recently, I have been doing a lot of Django formwork. I start with a basic template like form.as_div or form|crispy until it grows uncomfortable.

    Today, I was bouncing between two projects, and I noticed I was working on the tasks that had grown uncomfortable to the point that I dreaded working on the templates.

    While I enjoy working with Django’s template system, I was putting off these tasks, and all they had in common was finishing some of the form and template work.

    I couldn’t quite understand why this was such a mental blocker, so I stopped working, disconnected, and mowed my yard. Thankfully, that did the trick.

    As I finished mowing, I realized that I was struggling to complete these tasks because I was overwhelmed by needing to dump all the form fields into a template.

    Once I realized why I was feeling this resistance, I realized I needed to focus on solving this issue to move on.

    I remembered Daniel Roy Greenfeld’s Rapidly creating smoke tests for Django views from a few weeks ago, where he made a management command to print out a bunch of smoke tests.

    I decided to try the same technique by passing a string path to a Django ModelForm and printing out my form template.

    Edit: micro.blog did not like the templates in templates from my Python script, so I had to swap out the inline example with a gist. Sorry about that.

    This template could be better, but it was good enough. I tested it on a few of the forms I’m using on Django News Jobs, and it’s an improvement over what I started with.

    Something was in the water because when I checked our company Slack, Frank Wiles showed me his new make-management-command project, which takes a similar approach to creating the folders and files needed to create a new management command.

    Saturday June 8, 2024
  • Django

    🔋 My django-startproject project updates

    This morning, I updated my django-startproject to keep up with the latest Django, Docker Compose, ruff, pre-commit, and other versions. This project includes the bare minimum number of batteries I use in my Django projects.

    I also included a justfile, some common recipes, and workflows I use in my personal and client projects. I usually start with these recipes, and then I customize them for the project and the client.

    While these batteries and workflows work for me, I think there is something to be said for starting with Django’s default project_template and app_template to create your own opinionated startproject and startapp templates.

    Create your own start* templates and check out what other people are building. You might learn something that changes your perspective or saves you time.

    Sunday May 26, 2024
  • Django

    ✨ What's new in Django 5.1

    With today’s Django 5.1 alpha 1 release, picking just one favorite feature is hard. Django 5.1 is scheduled for release this August.

    I highly recommend reading the release notes on everything included in the next release.

    Here are a few of my favorite features that are solid quality-of-life improvements.

    ModelAdmin.list_display

    django.contrib.admin

    ModelAdmin.list_display now supports using __ lookups to list fields from related models.

    Many times a year, I forget that list_display does not support the dunder __ lookup, which leads to adding a property on ModelAdmin instead. FINALLY, Django supports this and I’m thrilled.

    The querystring template tag

    {% querystring %} template tag

    Django 5.1 introduces the {% querystring %} template tag, simplifying the modification of query parameters in URLs, making it easier to generate links that maintain existing query parameters while adding or changing specific ones.

    This is one of those template tags that I routinely add to my projects because it’s so helpful. I’m thrilled to no longer need it.

    Views require authentication by default

    Middleware to require authentication by default

    The new LoginRequiredMiddleware redirects all unauthenticated requests to a login page. Views can allow unauthenticated requests by using the new `login_not_required() decorator.

    Django now ships with a LoginRequiredMiddleware middleware, which adds authentication to all pages by default.

    I’m happy to see this because >90% of the apps I build require authentication by default, and it’d be easier/less code to mark views that do not need required auth. Plus, it feels more secure to have a way to default all views to using auth than to forget to decorate a view that should not be visible.

    I’m not sure how this impacts using third-party apps with views yet, but I suspect there will be a reasonable solution.

    One more thing…

    There are dozens and dozens of new features in the Django 5.1 release notes. If you spot any gems or have any favorite features, please let me know.

    Wednesday May 22, 2024
  • Django

    🎒 Everyone struggles with Django's static files

    Josh Thomas did a great job documenting and walking us through how he prefers to set up static files in his Django projects last week in his How I organize staticfiles in my Django projects article.

    Josh recommends the following config and naming convention:

    # settings.py
    
    # django.contrib.staticfiles
    STATIC_ROOT = BASE_DIR / "staticfiles"
    
    STATIC_URL = "/static/"
    
    STATICFILES_DIRS = [
        BASE_DIR / "static" / "dist",
        BASE_DIR / "static" / "public",
    ]
    

    Overall, this is very similar to what I do, but I settled into calling my STATICFILES_DIRS folder assets or frontend. After seeing Josh’s example, I changed this value to staticfiles to match the setting variable more closely.

    Updated config

    # settings.py
    
    # django.contrib.staticfiles
    
    # INPUT: Where to look for static files
    STATICFILES_DIRS = [
        BASE_DIR / "staticfiles" / "dist",
        BASE_DIR / "staticfiles" / "public",
    ]
    
    # OUTPUT: Where to put and look for static files to serve
    STATIC_ROOT = BASE_DIR / "static"
    
    # SERVE: Where to serve static files
    STATIC_URL = "/static/"
    

    This also changes our .gitignore to match our new settings. Since all of our files will be collected by Django and placed into the static folder, we can tell git to ignore this folder.

    We can also ignore the staticfiles/dist/ folder if we have an asset building pipeline and need a place to store the intermediate files.

    #.gitignore
    static/
    staticfiles/dist/
    # Optionally, to universally ignore all 'dist' directories:
    # dist
    
    Tuesday April 30, 2024
  • Django

    ,

    Python

    🤖 Super Bot Fight 🥊

    In March, I wrote about my robots.txt research and how I started proactively and defensively blocking AI Agents in my 🤖 On Robots.txt. Since March, I have updated my Django projects to add more robots.txt rules.

    Earlier this week, I ran across this Blockin’ bots. blog post and this example, the mod_rewrite rule blocks AI Agents via their User-Agent strings.

    <IfModule mod_rewrite.c>
    RewriteEngine on
    RewriteBase /
    # block “AI” bots
    RewriteCond %{HTTP_USER_AGENT} (AdsBot-Google|Amazonbot|anthropic-ai|Applebot|AwarioRssBot|AwarioSmartBot|Bytespider|CCBot|ChatGPT|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|DataForSeoBot|Diffbot|FacebookBot|FacebookBot|Google-Extended|GPTBot|ImagesiftBot|magpie-crawler|omgili|Omgilibot|peer39_crawler|PerplexityBot|YouBot) [NC]
    RewriteRule ^ – [F]
    </IfModule>
    

    Since none of my projects use Apache, and I was short on time, I decided to leave this war to the bots.

    Django Middleware

    I asked ChatGPT to convert this snippet to a piece of Django Middleware called Super Bot Fight. After all, if we don’t have time to keep up with bots, then we could leverage this technology to help fight against them.

    In theory, this snippet passed my eyeball test and was good enough:

    # middleware.py
    
    from django.http import HttpResponseForbidden
    
    # List of user agents to block
    
    BLOCKED_USER_AGENTS = [
        "AdsBot-Google",
        "Amazonbot",
        "anthropic-ai",
        "Applebot",
        "AwarioRssBot",
        "AwarioSmartBot",
        "Bytespider",
        "CCBot",
        "ChatGPT",
        "ChatGPT-User",
        "Claude-Web",
        "ClaudeBot",
        "cohere-ai",
        "DataForSeoBot",
        "Diffbot",
        "FacebookBot",
        "Google-Extended",
        "GPTBot",
        "ImagesiftBot",
        "magpie-crawler",
        "omgili",
        "Omgilibot",
        "peer39_crawler",
        "PerplexityBot",
        "YouBot",
    ]
    
    class BlockBotsMiddleware:
    
        def __init__(self, get_response):
            self.get_response = get_response
    
        def __call__(self, request):
            # Check the User-Agent against the blocked list
            user_agent = request.META.get("HTTP_USER_AGENT", "")
            if any(bot in user_agent for bot in BLOCKED_USER_AGENTS):
                return HttpResponseForbidden("Access denied")
            response = self.get_response(request)
            return response
    

    To use this middleware, you would update your Django settings.py to add it to your MIDDLEWARE setting.

    # settings.py
    
    MIDDLEWARE = [
        ...
        "middleware.BlockBotsMiddleware",
        ...
    ]
    

    Tests?

    If this middleware works for you and you care about testing, then these tests should also work:

    
    import pytest
    
    from django.http import HttpRequest
    from django.test import RequestFactory
    
    from middleware import BlockBotsMiddleware
    
    @pytest.mark.parametrize("user_agent, should_block", [
        ("AdsBot-Google", True),
        ("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", False),
        ("ChatGPT-User", True),
        ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3", False),
    ])
    def test_user_agent_blocking(user_agent, should_block):
        # Create a request factory to generate request instances
        factory = RequestFactory()
        request = factory.get('/', HTTP_USER_AGENT=user_agent)
    
        # Middleware setup
        middleware = BlockBotsMiddleware(get_response=lambda request: HttpResponse())
        response = middleware(request)
    
        # Check if the response should be blocked or allowed
        if should_block:
            assert response.status_code == 403, f"Request with user agent '{user_agent}' should be blocked."
        else:
            assert response.status_code != 403, f"Request with user agent '{user_agent}' should not be blocked."
    
    

    Enhancements

    To use this code in production, I would normalize the user_agent and BLOCKED_USER_AGENTS variables to be case-insensitive.

    I would also consider storing my list of user agents in a Django model or using a project like django-robots instead of a hard-coded Python list.

    Thursday April 18, 2024
  • Django

    🙋 How often do you use Django's startproject and startapp?

    I saw this Mastodon post last week and thought it worth sharing.

    Weekend #django question/poll

    How often do you use startproject & startapp or something else?

    These commands vary by the client or project I’m working on.

    I use startproject one to two times a month.

    I use startapp one to a dozen times a week.

    I have used copier and cookiecutter, and both have their place, but I only use them occasionally.

    I create more new projects and apps than the average developer because I’m curious and spin up projects quickly to try out ideas. I prefer this over every experiment being part of a mono project because I will throw most of these away.

    and how many years since you started using django?

    I have been using Django since 2006.

    Saturday April 13, 2024
  • Django

    ,

    Python

    🚜 Refactoring and fiddling with Django migrations for pending pull requests 🐘

    One of Django’s most powerful features is the ORM, which includes a robust migration framework. One of Django’s most misunderstood features is Django migrations because it just works 99% of the time.

    Even when working solo, Django migrations are highly reliable, working 99.9% of the time and offering better uptime than most web services you may have used last week.

    The most common stumbling block for developers of all skill levels is rolling back a Django migration and prepping a pull request for review.

    I’m not picky about pull requests or git commit history because I default to using the “Squash and merge” feature to turn all pull request commits into one merge commit. The merge commit tells me when, what, and why something changed if I need extra context.

    I am pickier about seeing >2 database migrations for any app unless a data migration is involved. It’s common to see 4 to 20 migrations when someone works on a database feature for a week. Most of the changes tend to be fiddly, where someone adds a field, renames the field, renames it again, and then starts using it, which prompts another null=True change followed by a blank=True migration.

    For small databases, none of this matters.

    For a database with 10s or 100s of millions of records, these small changes can cause minutes of downtime per migration, which amounts to a throwaway change. While there are ways to mitigate most migration downtime situations, that’s different from my point today.

    I’m also guilty of being fiddly with my Django model changes because I know I can delete and refactor them before requesting approval. The process I use is probably worth sharing because once every new client comes up.

    Let’s assume I am working on Django News Jobs, and I am looking over my pull request one last time before I ask someone to review it. That’s when I noticed four migrations that could quickly be rebuilt into one, starting with my 0020* migration in my jobs app.

    The rough steps that I would do are:

    # step 1: see the state of our migrations
    $ python -m manage showmigrations jobs
    jobs
     [X] 0001_initial
     ...
     [X] 0019_alter_iowa_versus_unconn
     [X] 0020_alter_something_i_should_delete
     [X] 0021_alter_uconn_didnt_foul
     [X] 0022_alter_nevermind_uconn_cant_rebound
     [X] 0023_alter_iowa_beats_uconn
     [X] 0024_alter_south_carolina_sunday_by_four
    
    # step 2: rollback migrations to our last "good" state
    $ python -m manage migrate jobs 0019
    
    # step 3: delete our new migrations
    $ rm jobs/migrations/002*
    
    # step 4: rebuild migrations 
    python -m manage makemigrations jobs 
    
    # step 5: profit 
    python -m manage migrate jobs
    

    95% of the time, this is all I ever need to do.

    Occasionally, I check out another branch with conflicting migrations, and I’ll get my local database in a weird state.

    In those cases, check out the --fake (“Mark migrations as run without actually running them.") and --prune (“Delete nonexistent migrations from the django_migrations table.") options. The fake and prune operations saved me several times when my django_migrations table was out of sync, and I knew that SQL tables were already altered.

    What not squashmigrations?

    Excellent question. Squashing migrations is wonderful if you care about keeping every or most of the operations each migration is doing. Most of the time, I do not, so I overlook it.

    Saturday April 6, 2024
  • Django

    ,

    Python

    ⛳ Syncing Django Waffle feature flags

    The django-waffle feature flag library is helpful for projects where we want to release and test new features in production and have a controlled rollout. I also like using feature flags for resource-intensive features on a website that we want to toggle off during high-traffic periods. It’s a nice escape hatch to fall back on if we need to turn off a feature and roll out a fix without taking down your website.

    While Waffle is a powerful tool, I understand the challenge of keeping track of feature flags in both code and the database. It’s a pain point that many of us have experienced.

    Waffle has a WAFFLE_CREATE_MISSING_FLAGS=True setting that we can use to tell Waffle to create any missing flags in the database should it find one. While this helps discover which flags our application is using, we need to figure out how to clean up old flags in the long term.

    The pattern I landed on combines storing all our known feature flags and a note about what they do in our main settings file.

    # settings.py
    ... 
    
    WAFFLE_CREATE_MISSING_FLAGS=True
    
    WAFFLE_FEATURE_FLAGS = {
       "flag_one": "This is a note about flag_one",
       "flag_two": "This is a note about flag_two",
    }
    

    We will use a management command to sync every feature flag we have listed in our settings file, and then we will clean up any missing feature flags.

    # management/commands/sync_feature_flags.py
    import djclick as click
    
    from django.conf import settings
    from waffle.models import Flag
    
    
    @click()
    def command():
        # Create flags that don't exist
        for name, note in settings.WAFFLE_FEATURE_FLAGS.items():
            flag, created = Flag.objects.update_or_create(
                name=name, defaults={"note": note}
            )
            if created:
                print(f"Created flag {name} ({flag.pk})")
    
        # Delete flags that are no longer registered in settings
        for flag in Flag.objects.exclude(name__in=settings.FEATURE_FLAGS.keys()):
            flag.delete()
            print(f"Deleted flag {flag.name} ({flag.pk})")
    
    

    We can use the WAFFLE_CREATE_MISSING_FLAGS settings as a failsafe to create any flags we might have accidently missed. They will stick out because they will not have a note associated with them.

    This pattern is also helpful in solving similar problems for scheduled tasks, which might also store their schedules in the database.

    Check out this example in the Django Styleguide for how to sync Celery’s scheduled tasks.

    Friday April 5, 2024
  • Django

    ,

    Python

    ⬆️ The Upgrade Django project

    Upgrade Django is a REVSYS project we created six years ago and launched three years ago.

    The goal of Upgrade Django was to create a resource that made it easy to see at a glance which versions of the Django web framework are maintained and supported. We also wanted to catalog every release and common gotchas and link to helpful information like release notes, blog posts, and the tagged git branch on GitHub.

    We also wanted to make it easier to tell how long a given version of Django would be supported and what phase of its release cycle it is in.

    Future features

    We have over a dozen features planned, but it’s a project that primarily serves its original purpose.

    One feature on my list is that I’d love to see every backward incompatible change between two Django versions. This way, if someone knows their website is running on Django 3.2, they could pick Django 4.2 or Django 5.0 version and get a comprehensive list with links to everything they need to upgrade between versions.

    Projects like Upgrade Django are fun to work on because once you collect a bunch of data and start working with it, new ways of comparing and presenting the information become more apparent.

    If you have ideas for improving Upgrade Django that would be useful to your needs, we’d love to hear about them.

    Thursday April 4, 2024
  • Django

    Django Chat recording today

    I was on the Django Chat podcast today, but since the episode will not come out for a few more months, I am sharing a few links for upcoming events and deadlines.

    DjangoCon US CFP is open through April 24th. I plan to publish my list of ideas soon. If you contact me, I have a gist to share if you would like an advanced preview.

    DEFNA is seeking a new director/treasurer through April 9th. Our board seats only become available every few years, so this is an excellent opportunity to contribute to a non-profit and to help the Django community.

    Djangonaut Space 2024 Session 2 applications open on April 15th and close on May 13th.

    We covered many topics, and I’m curious about what makes the final cut. It was also a long day, so today is my short post. We covered everything today.

    Tuesday March 26, 2024
  • Django

    ,

    Python

    Things I can never remember how to do: Django Signals edition

    I am several weeks into working on a project with my colleague, Lacey Henschel. Today, while reviewing one of her pull requests, I was reminded how to test a Django Signal via mocking.

    Testing Django signals is valuable to me because I need help remembering how to test a signal, and even with lots of effort, it never works. So bookmark this one, friends. It works.

    Thankfully, she wrote it up in one of her TIL: How I set up django-activity-stream, including a simple test

    https://mastodon.social/@lacey@hachyderm.io

    Monday March 25, 2024
  • Django

    ,

    Python

    On scratching itches with Python

    Python is such a fantastic glue language. Last night, while watching March Madness basketball games, I had a programming itch I wanted to scratch.

    I dusted off a demo I wrote several years ago. It used Python’s subprocess module, which strings together a bunch of shell commands to perform a git checkout, run a few commands, and then commit the results. The script worked, but I struggled to get it fully working in a production environment.

    To clean things up and as an excuse to try out a new third-party package, I converted the script to use:

    • GitPython - GitPython is a Python library used to interact with Git repositories.

    • Shelmet - A shell power-up for working with the file system and running subprocess commands.

    • Django Q2 - A multiprocessing distributed task queue for Django based on Django-Q.

    Using Django might have been overkill, but having a Repository model to work with felt nice. Django Q2 was also overkill, but if I put this app into production, I’ll want a task queue, and Django Q2 has a manageable amount of overhead.

    GitPython was a nice improvement over calling git commands directly because their API makes it easier to see which files were modified and to check against existing branch names. I was happy with the results after porting my subprocess commands to the GitPython API.

    The final package I used is a new package called Shelmet, which was both a nice wrapper around subprocess plus they have a nice API for file system operations in the same vein as Python’s Pathlib module.

    Future goals

    I was tempted to cobble together a GitHub bot, but I didn’t need one. I might dabble with the GitHub API more to fork a repo, but for now, this landed in a better place, so when I pick it back up again in a year, I’m starting in a good place.

    If you want to write a GitHub bot, check out Mariatta’s black_out project.

    Saturday March 23, 2024
  • Django

    ,

    Python

    Automated Python and Django upgrades

    Recently, I have been maintaining forks for several projects that are no longer maintained. Usually, these are a pain to update, but I have found a workflow that takes the edge off by leveraging pre-commit.

    My process:

    • Fork the project on GitHub to whichever organization I work with or my personal account.
    • Check out a local copy of my forked copy with git.
    • Install pre-commit
    • Create a .pre-commit-config.yaml with ZERO formatting or lint changes. This file will only include django-upgrade and pyupgrade hooks.

    We skip the formatters and linters to avoid unnecessary changes if we want to open a pull request in the upstream project. If the project isn’t abandoned, we will want to do that.

    • For django-upgrade, change the—-target-version option to target the latest version of Django I’m upgrading to, which is currently 5.0.
    • For pyupgrade, update the python settings under default_language_version to the latest version of Python that I’m targetting. Currently, that’s 3.12.

    The django-upgrade and pyupgrade projects attempt to run several code formatters and can handle most of the more tedious upgrade steps.

    • Run pre-commit autoupdate to ensure we have the latest version of our hooks.
    • Run pre-commit run --all-files to run pyupgrade and django-upgrade on our project.
    • Run any tests contained in the project and review all changes.
    • Once I’m comfortable with the changes, I commit them all via git and push them upstream to my branch.

    Example .pre-commit-config.yaml config

    From my experience, less is more with this bane bones .pre-commit-config.yaml config file.

    # .pre-commit-config.yaml
    
    default_language_version:
      python: python3.12
    
    repos:
      - repo: https://github.com/asottile/pyupgrade
        rev: v3.15.1
        hooks:
          - id: pyupgrade
    
      - repo: https://github.com/adamchainz/django-upgrade
        rev: 1.16.0
        hooks:
          - id: django-upgrade
            args: [--target-version, "5.0"]
    

    If I’m comfortable that the project is abandoned, I’ll add ruff support with a more opinionated config to ease my maintenance burden going forward.

    Friday March 22, 2024
  • Django

    ,

    Python

    On environment variables and dotenv files

    Brett Cannon recently vented some frustrations about .env files.

    I still hate .env files and their lack of a standard

    https://mastodon.social/@brettcannon@fosstodon.org/112056455108582204

    Brett’s thread and our conversation reminded me that my rule for working with dotenv files is to have my environment load them instead of my Python app trying to read from the .env file directly.

    What is a .env (dotenv) file?

    A .env (aka dotenv) is a file that contains a list of key-value pairs in the format of {key}=value.

    At a basic level, this is what a bare minimum .env file might look for in a Django project.

    # .env
    DEBUG=true
    SECRET_KEY=you need to change this
    

    My go-to library for reading ENV variables is environs. While the environs library can read directly from a dotenv file, don’t do that. I never want my program to read from a file in production because I don’t want a physical file with all of my API keys and secrets.

    Most hosting providers, like Fly.io, have a command line interface for setting these key-value pairs in production to avoid needing a physical dotenv file.

    Instead, we should default to assuming that the ENV variables will bet in our environment, and we should fall back to either a reasonable default value or fail loudly.

    Using the environs library, my Django settings.py file tends to look like this:

    # settings.py
    import environs
    
    env = environs.Env()
    
    # this will default to False if not set.
    DEBUG = env.bool("DJANGO_DEBUG", default=False)
    
    # this will error loudly if not set
    SECRET_KEY = env.str("SECRET_KEY")
    
    # everything else... 
    

    I lean on Docker Compose for local development when I’m building web apps because I might have three to five services running. Compose can read a dotenv file and register them into environment variables.

    .envrc files aren’t .env files

    On my macOS, when I’m not developing in a container, I use the direnv application to read an .envrc file which is very similar to a dotenv file.

    A .envrc is very similar to a .env file, but to register the values into memory, you have to use Bash’s export convention. If you don’t specify export, the environment variables won’t be available in your existing Bash environment.

    # .envrc
    export DEBUG=true
    export SECRET_KEY=you need to change this
    

    I’m a fan of direnv because the utility ensures that my environment variables are only set while I am in the same folder or sub-folders that contain the .envrc file. If I move to a different folder location or project, direnv will automatically unload every environment variable that was previously set.

    This has saved me numerous times over the years when I have run a command that might upload a file to s3 and ensure that I’m not uploading to the wrong account because an environment variable is still set from another project.

    Clients are generally understanding, but overriding static media for one client with another client’s files is not a conversation I want to have with any client.

    direnv is excellent insurance against forgetting to unset an environment variable.

    Seeding a .env file

    I prefer to ship an example .env.example file in my projects with reasonable defaults and instructions for copying them over.

    # .env.example
    DEBUG=true
    SECRET_KEY=you need to change this
    

    If you are a casey/just justfile user, I like to ship a just bootstrap recipe that checks if a .env file already exists. If the .env file does not exist, it will copy the example in place.

    My bootstrap recipe typically looks like this:

    # justfile
    bootstrap *ARGS:
        #!/usr/bin/env bash
        set -euo pipefail
    
        if [ ! -f ".env" ]; then
            echo ".env created"
            cp .env.example .env
        fi
    

    How do we keep dotenv files in sync?

    One pain point when working with dotenv files is keeping new environment variables updated when a new variable has been added.

    Thankfully, modenv is an excellent utility that can do precisely this. I run modenv check and will compare the .env* files in the existing folder. It will tell us which files are missing an environment variable when it exists in one but not one of the other files.

    I use modenv check -f to sync up any missing keys with a blank value. This works well to sync up any new environment variables added to our .env.example file with our local .env file.

    Alternatives

    I recently wrote about Using Chamber with Django and managing environment variables, which dives into using Chamber, another tool for managing environment variables.

    If you are working with a team, the 1Password CLI’s op run command is an excellent way to share environment variables securely. The tool is straightforward and can be integrated securely with local workflows and CI with just a few steps.

    Wednesday March 13, 2024
  • Django

    ,

    Python

    How to test with Django, parametrize, and lazy fixtures

    This article is a follow-up to my post on How to test with Django and pytest fixtures.

    Here are some notes on how I prefer to test views for a Django application with authentication using pytest-lazy-fixture.

    Fixtures

    pytest-django has a django_user_model fixture/shortcut, which I recommend using to create valid Django user accounts for your project.

    This example assumes that there are four levels of users. We have anonymous (not authenticated), “user,” staff, and superuser levels of permission to work with. Both staff and superusers follow the Django default pattern and have the is_staff and is_superuser boolean fields set appropriately.

    # users/tests/fixtures.py
    import pytest
    
    
    @pytest.fixture
    def password(db) -> str:
        return "password"
    
    
    @pytest.fixture
    def staff(db, django_user_model, faker, password):
        return django_user_model.objects.create_user(
            email="staff@example.com",
            first_name=faker.first_name(),
            is_staff=True,
            is_superuser=False,
            last_name=faker.last_name(),
            password=password,
        )
    
    
    @pytest.fixture()
    def superuser(db, django_user_model, faker, password):
        return django_user_model.objects.create_user(
            email="superuser@example.com",
            first_name=faker.first_name(),
            is_staff=True,
            is_superuser=True,
            last_name=faker.last_name(),
            password=password,
        )
    
    
    @pytest.fixture()
    def user(db, django_user_model, faker, password):
        return django_user_model.objects.create_user(
            email="user@example.com",
            first_name=faker.first_name(),
            last_name=faker.last_name(),
            password=password,
        )
    
    

    Testing our views with different User roles

    We will assume that our website has some working Category pages that can only viewed by staff or superusers. The lazy_fixture library allows us to pass the name of a fixture using parametrize along with the expected status_code that our view should return.

    If you have never seen parametrize, it is a nice pytest convention that will re-run the same test multiple times while passing a list of parameters into the test to be evaluated.

    The tp function variable is a django-test-plus fixture.

    user, staff, and superuser are fixtures we created above.

    # categories/tests/test_views.py
    import pytest
    
    from pytest import param
    from pytest_lazyfixture import lazy_fixture
    
    
    def test_category_noauth(db, tp):
        """
        GET 'admin/categories/'
        """
        url = tp.reverse("admin:category-list")
    
        # Does this view work with auth?
        response = tp.get(url)
        tp.response_401(response)
    
    
    @pytest.mark.parametrize(
        "testing_user,status_code",
        [
            param(lazy_fixture("user"), 403),
            param(lazy_fixture("staff"), 200),
            param(lazy_fixture("superuser"), 200),
        ],
    )
    def test_category_with_auth(db, tp, testing_user, password, status_code):
        """
        GET 'admin/categories/'
        """
        url = tp.reverse("admin:category-list")
    
        # Does this view work with auth?
        tp.client.login(username=testing_user.email, password=password)
        response = tp.get(url)
        assert response.status_code == status_code
    

    Notes

    Please note: These status codes are more typical for a REST API. So I would adjust any 40x status codes accordingly.

    My goal in sharing these examples is to show that you can get some helpful testing in with a little bit of code, even if the goal isn’t to dive deep and cover everything.

    Updates

    To make my example more consistent, I updated @pytest.mark.django_db() to use a db fixture. Thank you, Ben Lopatin, for the feedback.

    Thursday March 7, 2024
  • Django

    ,

    Python

    Importing data with Django Ninja's ModelSchema

    I have recently been playing with Django Ninja for small APIs and for leveraging Schema. Specifically, ModelSchema is worth checking out because it’s a hidden gem for working with Django models, even if you aren’t interested in building a Rest API.

    Schemas are very useful to define your validation rules and responses, but sometimes you need to reflect your database models into schemas and keep changes in sync. https://django-ninja.dev/guides/response/django-pydantic/

    One challenge we face is importing data from one legacy database into a new database with a different structure. While we can map old fields to new fields using a Python dictionary, we also need more control over what the data looks like coming back out.

    Thankfully, ModelSchema is built on top of Pydantic’s BaseModel and supports Pydantic’s Field alias feature.

    This allows us to create a ModelSchema based on a LegacyCategory model, and we can build out Field(alias="...") types to change the shape of how the data is returned.

    We can then store the result as a Python dictionary and insert it into our new model. We can also log a JSON representation of the instance to make debugging easier. See Serializing Outside of Views for an overview of how the from_orm API works.

    To test this, I built a proof of concept Django management command using django-click, which loops through all our legacy category models and prints them.

    # management/commands/demo_model_schema.py
    import djclick as click
    
    from ninja import ModelSchema
    from pydantic import Field
    
    from legacy.models import LegacyCategory
    from future.models import Category
    
    
    class LegacyCategorySchema(ModelSchema):
        name: str = Field(alias="cat_name")
        description: str = Field(alias="cat_description")
        active: bool = Field(alias="cat_is_active")
    
        class Meta:
            fields = ["id"]
            model = Category
    
    
    @click.command()
    def main():
        categories = LegacyCategory.objects.all()
        for category in categories:
            data = LegacyCategorySchema.from_orm(category).dict()
            print(data)
            # save to a database or do something useful here
    

    More resources

    If you are curious about what Django Ninja is about, I recommend starting with their CRUD example: Final Code, and working backward. This will give you a good idea of what a finished CRUD Rest API looks like with Django Ninja.

    Wednesday March 6, 2024