• Django

    ,

    Python

    🐘 Django Migration Operations aka how to rename Models

    Renaming a table in Django seems more complex than it is. Last week, a client asked me how much pain it might be to rename a Django model from Party to Customer. We already used the model’s verbose_name, so it has been referencing the new name for months.

    Renaming the model should be as easy as renaming the model while updating any foreign key and many-to-many field references in other models and then running Django’s make migrations sub-command to see where we are at.

    The main issue with this approach is that Django will attempt to create a new table first, update model references, and then drop the old table.

    Unfortunately, Django will either fail mid-way through this migration and roll the changes back or even worse, it may complete the migration only for you to discover that your new table is empty.

    Deleting data is not what we want to happen.

    As it turns out, Django supports a RenameModel migration option, but it did not prompt me to ask if we wanted to rename Party to Customer.

    I am also more example-driven, and the Django docs don’t have an example of how to use RenameModel. Thankfully, this migration operation is about as straightforward as one can imagine: class RenameModel(old_model_name, new_model_name)

    I re-used the existing migration file that Django created for me. I dropped the CreateModel and DeleteModel operations, added a RenameField operation, and kept the RenameField operations which resulted in the following migration:

    from django.db import migrations
    
    
    class Migration(migrations.Migration):
    
        dependencies = [
            ('resources', '0002_alter_party_in_the_usa'),
        ]
    
        operations = [
            migrations.RenameModel('Party', 'Customer'),
            migrations.RenameField('Customer', 'party_number', 'customer_number'),
            migrations.RenameField('AnotherModel', 'party', 'customer'),
        ]
    

    The story’s moral is that you should always check and verify that your Django migrations will perform as you expect before running them in production. Thankfully, we did, even though glossing over them is easy.

    I also encourage you to dive deep into the areas of the Django docs where there aren’t examples. Many areas of the docs may need examples or even more expanded docs, and they are easy to gloss over or get intimidated by.

    You don’t have to be afraid to create and update your migrations by hand. After all, Django migrations are Python code designed to give you a jumpstart. You can and should modify the code to meet your needs. Migration Operations have a clean API once you dig below the surface and understand what options you have to work with.

    Monday July 15, 2024
  • Django

    🧰 More fun with Django Extensions using `shell_plus` and `graph_models`

    Yesterday, I wrote about Django Extensions show_urls management command because it’s useful. I have Mastodon posted/tooted about it [previously](https://mastodon.social/@webology/110271223054909764, but I didn’t expect it to possibly lead to it being added to Django, and yet here we are. My favorite byproduct of blogging is when someone talks about something they like, and someone asks, “What if” or “Why doesn’t?” and then they get inspired to look into it and contribute. This post might have led to one new contribution to Django. 🎉

    Several people shared that they also liked Django Extensions shell_plus and graph_models management commands.

    I don’t use shell_plus often, but I bake it into my Just workflows for clients who do. I tend to forget about it, and I spend so much time using pytest.set_trace() and testing.

    If you haven’t used graph_models, I use it in most of my client projects. I generate SVG files with it and add them to an ERD section of their docs, which helps discuss models and onboard new developers. It’s a nice-to-have feature and is a small lift with a huge payoff. This code is also easy to copy and paste from project to project.

    Sunday July 7, 2024
  • Django

    ,

    Python

    Django Extensions is useful even if you only use show_urls

    Yes, Django Extensions package is worth installing, especially for its show_urls command, which can be very useful for debugging and understanding your project’s URL configurations.

    Here’s a short example of how to use it because I sometimes want to include a link to the Django Admin in a menu for staff users, and I am trying to remember what name I need to reference to link to it.

    First, you will need to install it via:

    pip install django-extensions
    
    # or if you prefer using uv like me:
    uv pip install django-extensions
    

    Next, you’ll want to add django_extensions to your INSTALLED_APPS in your settings.py file:

    INSTALLED_APPS = [
        ...
        "django_extensions",
    ]
    

    Finally, to urn the show_urls management command you may do some by running your manage.py script and passing it the following option:

    $ python -m manage show_urls
    

    Which will give this output:

    $ python -m manage show_urls | grep admin
    ...
    /admin/	django.contrib.admin.sites.index	admin:index
    /admin/<app_label>/	django.contrib.admin.sites.app_index	admin:app_list
    /admin/<url>	django.contrib.admin.sites.catch_all_view
    # and a whole lot more...
    

    In this case, I was looking for admin:index which I can now add to my HTML document this menu link/snippet:

    ... 
    <a href="{% url 'admin:index' %}">Django Admin</a>
    ... 
    

    What I like about this approach is that I can now hide or rotate the url pattern I’m using to get to my admin website, and yet Django will always link to the correct one.

    Saturday July 6, 2024
  • Django

    🤖 More Blocking Bots with Django ❌

    Tonight, I ran across Robb Knight’s Blocking Bots with Nginx, which fits well with two pieces I have already written about.

    Check out my 🤖 Super Bot Fight 🥊 article, which develops a Django-friendly middleware for blocking AI UserAgents so they never get to your content.

    My 🤖 On Robots.txt article shared more of my research on the robots.txt standard.

    I forgot I wrote the middleware, so I rewrote it tonight before rediscovering it while looking for both articles to share. That was fun, so I’m not writing more tonight. 😬

    Friday June 14, 2024
  • Django

    🧱 Django ModelForm Template starting point

    Recently, I have been doing a lot of Django formwork. I start with a basic template like form.as_div or form|crispy until it grows uncomfortable.

    Today, I was bouncing between two projects, and I noticed I was working on the tasks that had grown uncomfortable to the point that I dreaded working on the templates.

    While I enjoy working with Django’s template system, I was putting off these tasks, and all they had in common was finishing some of the form and template work.

    I couldn’t quite understand why this was such a mental blocker, so I stopped working, disconnected, and mowed my yard. Thankfully, that did the trick.

    As I finished mowing, I realized that I was struggling to complete these tasks because I was overwhelmed by needing to dump all the form fields into a template.

    Once I realized why I was feeling this resistance, I realized I needed to focus on solving this issue to move on.

    I remembered Daniel Roy Greenfeld’s Rapidly creating smoke tests for Django views from a few weeks ago, where he made a management command to print out a bunch of smoke tests.

    I decided to try the same technique by passing a string path to a Django ModelForm and printing out my form template.

    Edit: micro.blog did not like the templates in templates from my Python script, so I had to swap out the inline example with a gist. Sorry about that.

    This template could be better, but it was good enough. I tested it on a few of the forms I’m using on Django News Jobs, and it’s an improvement over what I started with.

    Something was in the water because when I checked our company Slack, Frank Wiles showed me his new make-management-command project, which takes a similar approach to creating the folders and files needed to create a new management command.

    Saturday June 8, 2024
  • Django

    🔋 My django-startproject project updates

    This morning, I updated my django-startproject to keep up with the latest Django, Docker Compose, ruff, pre-commit, and other versions. This project includes the bare minimum number of batteries I use in my Django projects.

    I also included a justfile, some common recipes, and workflows I use in my personal and client projects. I usually start with these recipes, and then I customize them for the project and the client.

    While these batteries and workflows work for me, I think there is something to be said for starting with Django’s default project_template and app_template to create your own opinionated startproject and startapp templates.

    Create your own start* templates and check out what other people are building. You might learn something that changes your perspective or saves you time.

    Sunday May 26, 2024
  • Django

    ✨ What's new in Django 5.1

    With today’s Django 5.1 alpha 1 release, picking just one favorite feature is hard. Django 5.1 is scheduled for release this August.

    I highly recommend reading the release notes on everything included in the next release.

    Here are a few of my favorite features that are solid quality-of-life improvements.

    ModelAdmin.list_display

    django.contrib.admin

    ModelAdmin.list_display now supports using __ lookups to list fields from related models.

    Many times a year, I forget that list_display does not support the dunder __ lookup, which leads to adding a property on ModelAdmin instead. FINALLY, Django supports this and I’m thrilled.

    The query_string template tag

    {% query_string %} template tag

    Django 5.1 introduces the {% query_string %} template tag, simplifying the modification of query parameters in URLs, making it easier to generate links that maintain existing query parameters while adding or changing specific ones.

    This is one of those template tags that I routinely add to my projects because it’s so helpful. I’m thrilled to no longer need it.

    Views require authentication by default

    Middleware to require authentication by default

    The new LoginRequiredMiddleware redirects all unauthenticated requests to a login page. Views can allow unauthenticated requests by using the new `login_not_required() decorator.

    Django now ships with a LoginRequiredMiddleware middleware, which adds authentication to all pages by default.

    I’m happy to see this because >90% of the apps I build require authentication by default, and it’d be easier/less code to mark views that do not need required auth. Plus, it feels more secure to have a way to default all views to using auth than to forget to decorate a view that should not be visible.

    I’m not sure how this impacts using third-party apps with views yet, but I suspect there will be a reasonable solution.

    One more thing…

    There are dozens and dozens of new features in the Django 5.1 release notes. If you spot any gems or have any favorite features, please let me know.

    Wednesday May 22, 2024
  • Django

    🎒 Everyone struggles with Django's static files

    Josh Thomas did a great job documenting and walking us through how he prefers to set up static files in his Django projects last week in his How I organize staticfiles in my Django projects article.

    Josh recommends the following config and naming convention:

    # settings.py
    
    # django.contrib.staticfiles
    STATIC_ROOT = BASE_DIR / "staticfiles"
    
    STATIC_URL = "/static/"
    
    STATICFILES_DIRS = [
        BASE_DIR / "static" / "dist",
        BASE_DIR / "static" / "public",
    ]
    

    Overall, this is very similar to what I do, but I settled into calling my STATICFILES_DIRS folder assets or frontend. After seeing Josh’s example, I changed this value to staticfiles to match the setting variable more closely.

    Updated config

    # settings.py
    
    # django.contrib.staticfiles
    
    # INPUT: Where to look for static files
    STATICFILES_DIRS = [
        BASE_DIR / "staticfiles" / "dist",
        BASE_DIR / "staticfiles" / "public",
    ]
    
    # OUTPUT: Where to put and look for static files to serve
    STATIC_ROOT = BASE_DIR / "static"
    
    # SERVE: Where to serve static files
    STATIC_URL = "/static/"
    

    This also changes our .gitignore to match our new settings. Since all of our files will be collected by Django and placed into the static folder, we can tell git to ignore this folder.

    We can also ignore the staticfiles/dist/ folder if we have an asset building pipeline and need a place to store the intermediate files.

    #.gitignore
    static/
    staticfiles/dist/
    # Optionally, to universally ignore all 'dist' directories:
    # dist
    
    Tuesday April 30, 2024
  • Django

    ,

    Python

    🤖 Super Bot Fight 🥊

    In March, I wrote about my robots.txt research and how I started proactively and defensively blocking AI Agents in my 🤖 On Robots.txt. Since March, I have updated my Django projects to add more robots.txt rules.

    Earlier this week, I ran across this Blockin’ bots. blog post and this example, the mod_rewrite rule blocks AI Agents via their User-Agent strings.

    <IfModule mod_rewrite.c>
    RewriteEngine on
    RewriteBase /
    # block “AI” bots
    RewriteCond %{HTTP_USER_AGENT} (AdsBot-Google|Amazonbot|anthropic-ai|Applebot|AwarioRssBot|AwarioSmartBot|Bytespider|CCBot|ChatGPT|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|DataForSeoBot|Diffbot|FacebookBot|FacebookBot|Google-Extended|GPTBot|ImagesiftBot|magpie-crawler|omgili|Omgilibot|peer39_crawler|PerplexityBot|YouBot) [NC]
    RewriteRule ^ – [F]
    </IfModule>
    

    Since none of my projects use Apache, and I was short on time, I decided to leave this war to the bots.

    Django Middleware

    I asked ChatGPT to convert this snippet to a piece of Django Middleware called Super Bot Fight. After all, if we don’t have time to keep up with bots, then we could leverage this technology to help fight against them.

    In theory, this snippet passed my eyeball test and was good enough:

    # middleware.py
    
    from django.http import HttpResponseForbidden
    
    # List of user agents to block
    
    BLOCKED_USER_AGENTS = [
        "AdsBot-Google",
        "Amazonbot",
        "anthropic-ai",
        "Applebot",
        "AwarioRssBot",
        "AwarioSmartBot",
        "Bytespider",
        "CCBot",
        "ChatGPT",
        "ChatGPT-User",
        "Claude-Web",
        "ClaudeBot",
        "cohere-ai",
        "DataForSeoBot",
        "Diffbot",
        "FacebookBot",
        "Google-Extended",
        "GPTBot",
        "ImagesiftBot",
        "magpie-crawler",
        "omgili",
        "Omgilibot",
        "peer39_crawler",
        "PerplexityBot",
        "YouBot",
    ]
    
    class BlockBotsMiddleware:
    
        def __init__(self, get_response):
            self.get_response = get_response
    
        def __call__(self, request):
            # Check the User-Agent against the blocked list
            user_agent = request.META.get("HTTP_USER_AGENT", "")
            if any(bot in user_agent for bot in BLOCKED_USER_AGENTS):
                return HttpResponseForbidden("Access denied")
            response = self.get_response(request)
            return response
    

    To use this middleware, you would update your Django settings.py to add it to your MIDDLEWARE setting.

    # settings.py
    
    MIDDLEWARE = [
        ...
        "middleware.BlockBotsMiddleware",
        ...
    ]
    

    Tests?

    If this middleware works for you and you care about testing, then these tests should also work:

    
    import pytest
    
    from django.http import HttpRequest
    from django.test import RequestFactory
    
    from middleware import BlockBotsMiddleware
    
    @pytest.mark.parametrize("user_agent, should_block", [
        ("AdsBot-Google", True),
        ("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", False),
        ("ChatGPT-User", True),
        ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3", False),
    ])
    def test_user_agent_blocking(user_agent, should_block):
        # Create a request factory to generate request instances
        factory = RequestFactory()
        request = factory.get('/', HTTP_USER_AGENT=user_agent)
    
        # Middleware setup
        middleware = BlockBotsMiddleware(get_response=lambda request: HttpResponse())
        response = middleware(request)
    
        # Check if the response should be blocked or allowed
        if should_block:
            assert response.status_code == 403, f"Request with user agent '{user_agent}' should be blocked."
        else:
            assert response.status_code != 403, f"Request with user agent '{user_agent}' should not be blocked."
    
    

    Enhancements

    To use this code in production, I would normalize the user_agent and BLOCKED_USER_AGENTS variables to be case-insensitive.

    I would also consider storing my list of user agents in a Django model or using a project like django-robots instead of a hard-coded Python list.

    Thursday April 18, 2024
  • Django

    🙋 How often do you use Django's startproject and startapp?

    I saw this Mastodon post last week and thought it worth sharing.

    Weekend #django question/poll

    How often do you use startproject & startapp or something else?

    These commands vary by the client or project I’m working on.

    I use startproject one to two times a month.

    I use startapp one to a dozen times a week.

    I have used copier and cookiecutter, and both have their place, but I only use them occasionally.

    I create more new projects and apps than the average developer because I’m curious and spin up projects quickly to try out ideas. I prefer this over every experiment being part of a mono project because I will throw most of these away.

    and how many years since you started using django?

    I have been using Django since 2006.

    Saturday April 13, 2024
  • Django

    ,

    Python

    🚜 Refactoring and fiddling with Django migrations for pending pull requests 🐘

    One of Django’s most powerful features is the ORM, which includes a robust migration framework. One of Django’s most misunderstood features is Django migrations because it just works 99% of the time.

    Even when working solo, Django migrations are highly reliable, working 99.9% of the time and offering better uptime than most web services you may have used last week.

    The most common stumbling block for developers of all skill levels is rolling back a Django migration and prepping a pull request for review.

    I’m not picky about pull requests or git commit history because I default to using the “Squash and merge” feature to turn all pull request commits into one merge commit. The merge commit tells me when, what, and why something changed if I need extra context.

    I am pickier about seeing >2 database migrations for any app unless a data migration is involved. It’s common to see 4 to 20 migrations when someone works on a database feature for a week. Most of the changes tend to be fiddly, where someone adds a field, renames the field, renames it again, and then starts using it, which prompts another null=True change followed by a blank=True migration.

    For small databases, none of this matters.

    For a database with 10s or 100s of millions of records, these small changes can cause minutes of downtime per migration, which amounts to a throwaway change. While there are ways to mitigate most migration downtime situations, that’s different from my point today.

    I’m also guilty of being fiddly with my Django model changes because I know I can delete and refactor them before requesting approval. The process I use is probably worth sharing because once every new client comes up.

    Let’s assume I am working on Django News Jobs, and I am looking over my pull request one last time before I ask someone to review it. That’s when I noticed four migrations that could quickly be rebuilt into one, starting with my 0020* migration in my jobs app.

    The rough steps that I would do are:

    # step 1: see the state of our migrations
    $ python -m manage showmigrations jobs
    jobs
     [X] 0001_initial
     ...
     [X] 0019_alter_iowa_versus_unconn
     [X] 0020_alter_something_i_should_delete
     [X] 0021_alter_uconn_didnt_foul
     [X] 0022_alter_nevermind_uconn_cant_rebound
     [X] 0023_alter_iowa_beats_uconn
     [X] 0024_alter_south_carolina_sunday_by_four
    
    # step 2: rollback migrations to our last "good" state
    $ python -m manage migrate jobs 0019
    
    # step 3: delete our new migrations
    $ rm jobs/migrations/002*
    
    # step 4: rebuild migrations 
    python -m manage makemigrations jobs 
    
    # step 5: profit 
    python -m manage migrate jobs
    

    95% of the time, this is all I ever need to do.

    Occasionally, I check out another branch with conflicting migrations, and I’ll get my local database in a weird state.

    In those cases, check out the --fake (“Mark migrations as run without actually running them.") and --prune (“Delete nonexistent migrations from the django_migrations table.") options. The fake and prune operations saved me several times when my django_migrations table was out of sync, and I knew that SQL tables were already altered.

    What not squashmigrations?

    Excellent question. Squashing migrations is wonderful if you care about keeping every or most of the operations each migration is doing. Most of the time, I do not, so I overlook it.

    Saturday April 6, 2024
  • Django

    ,

    Python

    ⛳ Syncing Django Waffle feature flags

    The django-waffle feature flag library is helpful for projects where we want to release and test new features in production and have a controlled rollout. I also like using feature flags for resource-intensive features on a website that we want to toggle off during high-traffic periods. It’s a nice escape hatch to fall back on if we need to turn off a feature and roll out a fix without taking down your website.

    While Waffle is a powerful tool, I understand the challenge of keeping track of feature flags in both code and the database. It’s a pain point that many of us have experienced.

    Waffle has a WAFFLE_CREATE_MISSING_FLAGS=True setting that we can use to tell Waffle to create any missing flags in the database should it find one. While this helps discover which flags our application is using, we need to figure out how to clean up old flags in the long term.

    The pattern I landed on combines storing all our known feature flags and a note about what they do in our main settings file.

    # settings.py
    ... 
    
    WAFFLE_CREATE_MISSING_FLAGS=True
    
    WAFFLE_FEATURE_FLAGS = {
       "flag_one": "This is a note about flag_one",
       "flag_two": "This is a note about flag_two",
    }
    

    We will use a management command to sync every feature flag we have listed in our settings file, and then we will clean up any missing feature flags.

    # management/commands/sync_feature_flags.py
    import djclick as click
    
    from django.conf import settings
    from waffle.models import Flag
    
    
    @click()
    def command():
        # Create flags that don't exist
        for name, note in settings.WAFFLE_FEATURE_FLAGS.items():
            flag, created = Flag.objects.update_or_create(
                name=name, defaults={"note": note}
            )
            if created:
                print(f"Created flag {name} ({flag.pk})")
    
        # Delete flags that are no longer registered in settings
        for flag in Flag.objects.exclude(name__in=settings.FEATURE_FLAGS.keys()):
            flag.delete()
            print(f"Deleted flag {flag.name} ({flag.pk})")
    
    

    We can use the WAFFLE_CREATE_MISSING_FLAGS settings as a failsafe to create any flags we might have accidently missed. They will stick out because they will not have a note associated with them.

    This pattern is also helpful in solving similar problems for scheduled tasks, which might also store their schedules in the database.

    Check out this example in the Django Styleguide for how to sync Celery’s scheduled tasks.

    Friday April 5, 2024
  • Django

    ,

    Python

    ⬆️ The Upgrade Django project

    Upgrade Django is a REVSYS project we created six years ago and launched three years ago.

    The goal of Upgrade Django was to create a resource that made it easy to see at a glance which versions of the Django web framework are maintained and supported. We also wanted to catalog every release and common gotchas and link to helpful information like release notes, blog posts, and the tagged git branch on GitHub.

    We also wanted to make it easier to tell how long a given version of Django would be supported and what phase of its release cycle it is in.

    Future features

    We have over a dozen features planned, but it’s a project that primarily serves its original purpose.

    One feature on my list is that I’d love to see every backward incompatible change between two Django versions. This way, if someone knows their website is running on Django 3.2, they could pick Django 4.2 or Django 5.0 version and get a comprehensive list with links to everything they need to upgrade between versions.

    Projects like Upgrade Django are fun to work on because once you collect a bunch of data and start working with it, new ways of comparing and presenting the information become more apparent.

    If you have ideas for improving Upgrade Django that would be useful to your needs, we’d love to hear about them.

    Thursday April 4, 2024
  • Django

    Django Chat recording today

    I was on the Django Chat podcast today, but since the episode will not come out for a few more months, I am sharing a few links for upcoming events and deadlines.

    DjangoCon US CFP is open through April 24th. I plan to publish my list of ideas soon. If you contact me, I have a gist to share if you would like an advanced preview.

    DEFNA is seeking a new director/treasurer through April 9th. Our board seats only become available every few years, so this is an excellent opportunity to contribute to a non-profit and to help the Django community.

    Djangonaut Space 2024 Session 2 applications open on April 15th and close on May 13th.

    We covered many topics, and I’m curious about what makes the final cut. It was also a long day, so today is my short post. We covered everything today.

    Tuesday March 26, 2024
  • Django

    ,

    Python

    Things I can never remember how to do: Django Signals edition

    I am several weeks into working on a project with my colleague, Lacey Henschel. Today, while reviewing one of her pull requests, I was reminded how to test a Django Signal via mocking.

    Testing Django signals is valuable to me because I need help remembering how to test a signal, and even with lots of effort, it never works. So bookmark this one, friends. It works.

    Thankfully, she wrote it up in one of her TIL: How I set up django-activity-stream, including a simple test

    https://mastodon.social/@lacey@hachyderm.io

    Monday March 25, 2024
  • Django

    ,

    Python

    On scratching itches with Python

    Python is such a fantastic glue language. Last night, while watching March Madness basketball games, I had a programming itch I wanted to scratch.

    I dusted off a demo I wrote several years ago. It used Python’s subprocess module, which strings together a bunch of shell commands to perform a git checkout, run a few commands, and then commit the results. The script worked, but I struggled to get it fully working in a production environment.

    To clean things up and as an excuse to try out a new third-party package, I converted the script to use:

    • GitPython - GitPython is a Python library used to interact with Git repositories.

    • Shelmet - A shell power-up for working with the file system and running subprocess commands.

    • Django Q2 - A multiprocessing distributed task queue for Django based on Django-Q.

    Using Django might have been overkill, but having a Repository model to work with felt nice. Django Q2 was also overkill, but if I put this app into production, I’ll want a task queue, and Django Q2 has a manageable amount of overhead.

    GitPython was a nice improvement over calling git commands directly because their API makes it easier to see which files were modified and to check against existing branch names. I was happy with the results after porting my subprocess commands to the GitPython API.

    The final package I used is a new package called Shelmet, which was both a nice wrapper around subprocess plus they have a nice API for file system operations in the same vein as Python’s Pathlib module.

    Future goals

    I was tempted to cobble together a GitHub bot, but I didn’t need one. I might dabble with the GitHub API more to fork a repo, but for now, this landed in a better place, so when I pick it back up again in a year, I’m starting in a good place.

    If you want to write a GitHub bot, check out Mariatta’s black_out project.

    Saturday March 23, 2024
  • Django

    ,

    Python

    Automated Python and Django upgrades

    Recently, I have been maintaining forks for several projects that are no longer maintained. Usually, these are a pain to update, but I have found a workflow that takes the edge off by leveraging pre-commit.

    My process:

    • Fork the project on GitHub to whichever organization I work with or my personal account.
    • Check out a local copy of my forked copy with git.
    • Install pre-commit
    • Create a .pre-commit-config.yaml with ZERO formatting or lint changes. This file will only include django-upgrade and pyupgrade hooks.

    We skip the formatters and linters to avoid unnecessary changes if we want to open a pull request in the upstream project. If the project isn’t abandoned, we will want to do that.

    • For django-upgrade, change the—-target-version option to target the latest version of Django I’m upgrading to, which is currently 5.0.
    • For pyupgrade, update the python settings under default_language_version to the latest version of Python that I’m targetting. Currently, that’s 3.12.

    The django-upgrade and pyupgrade projects attempt to run several code formatters and can handle most of the more tedious upgrade steps.

    • Run pre-commit autoupdate to ensure we have the latest version of our hooks.
    • Run pre-commit run --all-files to run pyupgrade and django-upgrade on our project.
    • Run any tests contained in the project and review all changes.
    • Once I’m comfortable with the changes, I commit them all via git and push them upstream to my branch.

    Example .pre-commit-config.yaml config

    From my experience, less is more with this bane bones .pre-commit-config.yaml config file.

    # .pre-commit-config.yaml
    
    default_language_version:
      python: python3.12
    
    repos:
      - repo: https://github.com/asottile/pyupgrade
        rev: v3.15.1
        hooks:
          - id: pyupgrade
    
      - repo: https://github.com/adamchainz/django-upgrade
        rev: 1.16.0
        hooks:
          - id: django-upgrade
            args: [--target-version, "5.0"]
    

    If I’m comfortable that the project is abandoned, I’ll add ruff support with a more opinionated config to ease my maintenance burden going forward.

    Friday March 22, 2024
  • Django

    ,

    Python

    On environment variables and dotenv files

    Brett Cannon recently vented some frustrations about .env files.

    I still hate .env files and their lack of a standard

    https://mastodon.social/@brettcannon@fosstodon.org/112056455108582204

    Brett’s thread and our conversation reminded me that my rule for working with dotenv files is to have my environment load them instead of my Python app trying to read from the .env file directly.

    What is a .env (dotenv) file?

    A .env (aka dotenv) is a file that contains a list of key-value pairs in the format of {key}=value.

    At a basic level, this is what a bare minimum .env file might look for in a Django project.

    # .env
    DEBUG=true
    SECRET_KEY=you need to change this
    

    My go-to library for reading ENV variables is environs. While the environs library can read directly from a dotenv file, don’t do that. I never want my program to read from a file in production because I don’t want a physical file with all of my API keys and secrets.

    Most hosting providers, like Fly.io, have a command line interface for setting these key-value pairs in production to avoid needing a physical dotenv file.

    Instead, we should default to assuming that the ENV variables will bet in our environment, and we should fall back to either a reasonable default value or fail loudly.

    Using the environs library, my Django settings.py file tends to look like this:

    # settings.py
    import environs
    
    env = environs.Env()
    
    # this will default to False if not set.
    DEBUG = env.bool("DJANGO_DEBUG", default=False)
    
    # this will error loudly if not set
    SECRET_KEY = env.str("SECRET_KEY")
    
    # everything else... 
    

    I lean on Docker Compose for local development when I’m building web apps because I might have three to five services running. Compose can read a dotenv file and register them into environment variables.

    .envrc files aren’t .env files

    On my macOS, when I’m not developing in a container, I use the direnv application to read an .envrc file which is very similar to a dotenv file.

    A .envrc is very similar to a .env file, but to register the values into memory, you have to use Bash’s export convention. If you don’t specify export, the environment variables won’t be available in your existing Bash environment.

    # .envrc
    export DEBUG=true
    export SECRET_KEY=you need to change this
    

    I’m a fan of direnv because the utility ensures that my environment variables are only set while I am in the same folder or sub-folders that contain the .envrc file. If I move to a different folder location or project, direnv will automatically unload every environment variable that was previously set.

    This has saved me numerous times over the years when I have run a command that might upload a file to s3 and ensure that I’m not uploading to the wrong account because an environment variable is still set from another project.

    Clients are generally understanding, but overriding static media for one client with another client’s files is not a conversation I want to have with any client.

    direnv is excellent insurance against forgetting to unset an environment variable.

    Seeding a .env file

    I prefer to ship an example .env.example file in my projects with reasonable defaults and instructions for copying them over.

    # .env.example
    DEBUG=true
    SECRET_KEY=you need to change this
    

    If you are a casey/just justfile user, I like to ship a just bootstrap recipe that checks if a .env file already exists. If the .env file does not exist, it will copy the example in place.

    My bootstrap recipe typically looks like this:

    # justfile
    bootstrap *ARGS:
        #!/usr/bin/env bash
        set -euo pipefail
    
        if [ ! -f ".env" ]; then
            echo ".env created"
            cp .env.example .env
        fi
    

    How do we keep dotenv files in sync?

    One pain point when working with dotenv files is keeping new environment variables updated when a new variable has been added.

    Thankfully, modenv is an excellent utility that can do precisely this. I run modenv check and will compare the .env* files in the existing folder. It will tell us which files are missing an environment variable when it exists in one but not one of the other files.

    I use modenv check -f to sync up any missing keys with a blank value. This works well to sync up any new environment variables added to our .env.example file with our local .env file.

    Alternatives

    I recently wrote about Using Chamber with Django and managing environment variables, which dives into using Chamber, another tool for managing environment variables.

    If you are working with a team, the 1Password CLI’s op run command is an excellent way to share environment variables securely. The tool is straightforward and can be integrated securely with local workflows and CI with just a few steps.

    Wednesday March 13, 2024
  • Django

    ,

    Python

    How to test with Django, parametrize, and lazy fixtures

    This article is a follow-up to my post on How to test with Django and pytest fixtures.

    Here are some notes on how I prefer to test views for a Django application with authentication using pytest-lazy-fixture.

    Fixtures

    pytest-django has a django_user_model fixture/shortcut, which I recommend using to create valid Django user accounts for your project.

    This example assumes that there are four levels of users. We have anonymous (not authenticated), “user,” staff, and superuser levels of permission to work with. Both staff and superusers follow the Django default pattern and have the is_staff and is_superuser boolean fields set appropriately.

    # users/tests/fixtures.py
    import pytest
    
    
    @pytest.fixture
    def password(db) -> str:
        return "password"
    
    
    @pytest.fixture
    def staff(db, django_user_model, faker, password):
        return django_user_model.objects.create_user(
            email="staff@example.com",
            first_name=faker.first_name(),
            is_staff=True,
            is_superuser=False,
            last_name=faker.last_name(),
            password=password,
        )
    
    
    @pytest.fixture()
    def superuser(db, django_user_model, faker, password):
        return django_user_model.objects.create_user(
            email="superuser@example.com",
            first_name=faker.first_name(),
            is_staff=True,
            is_superuser=True,
            last_name=faker.last_name(),
            password=password,
        )
    
    
    @pytest.fixture()
    def user(db, django_user_model, faker, password):
        return django_user_model.objects.create_user(
            email="user@example.com",
            first_name=faker.first_name(),
            last_name=faker.last_name(),
            password=password,
        )
    
    

    Testing our views with different User roles

    We will assume that our website has some working Category pages that can only viewed by staff or superusers. The lazy_fixture library allows us to pass the name of a fixture using parametrize along with the expected status_code that our view should return.

    If you have never seen parametrize, it is a nice pytest convention that will re-run the same test multiple times while passing a list of parameters into the test to be evaluated.

    The tp function variable is a django-test-plus fixture.

    user, staff, and superuser are fixtures we created above.

    # categories/tests/test_views.py
    import pytest
    
    from pytest import param
    from pytest_lazyfixture import lazy_fixture
    
    
    def test_category_noauth(db, tp):
        """
        GET 'admin/categories/'
        """
        url = tp.reverse("admin:category-list")
    
        # Does this view work with auth?
        response = tp.get(url)
        tp.response_401(response)
    
    
    @pytest.mark.parametrize(
        "testing_user,status_code",
        [
            param(lazy_fixture("user"), 403),
            param(lazy_fixture("staff"), 200),
            param(lazy_fixture("superuser"), 200),
        ],
    )
    def test_category_with_auth(db, tp, testing_user, password, status_code):
        """
        GET 'admin/categories/'
        """
        url = tp.reverse("admin:category-list")
    
        # Does this view work with auth?
        tp.client.login(username=testing_user.email, password=password)
        response = tp.get(url)
        assert response.status_code == status_code
    

    Notes

    Please note: These status codes are more typical for a REST API. So I would adjust any 40x status codes accordingly.

    My goal in sharing these examples is to show that you can get some helpful testing in with a little bit of code, even if the goal isn’t to dive deep and cover everything.

    Updates

    To make my example more consistent, I updated @pytest.mark.django_db() to use a db fixture. Thank you, Ben Lopatin, for the feedback.

    Thursday March 7, 2024
  • Django

    ,

    Python

    Importing data with Django Ninja's ModelSchema

    I have recently been playing with Django Ninja for small APIs and for leveraging Schema. Specifically, ModelSchema is worth checking out because it’s a hidden gem for working with Django models, even if you aren’t interested in building a Rest API.

    Schemas are very useful to define your validation rules and responses, but sometimes you need to reflect your database models into schemas and keep changes in sync. https://django-ninja.dev/guides/response/django-pydantic/

    One challenge we face is importing data from one legacy database into a new database with a different structure. While we can map old fields to new fields using a Python dictionary, we also need more control over what the data looks like coming back out.

    Thankfully, ModelSchema is built on top of Pydantic’s BaseModel and supports Pydantic’s Field alias feature.

    This allows us to create a ModelSchema based on a LegacyCategory model, and we can build out Field(alias="...") types to change the shape of how the data is returned.

    We can then store the result as a Python dictionary and insert it into our new model. We can also log a JSON representation of the instance to make debugging easier. See Serializing Outside of Views for an overview of how the from_orm API works.

    To test this, I built a proof of concept Django management command using django-click, which loops through all our legacy category models and prints them.

    # management/commands/demo_model_schema.py
    import djclick as click
    
    from ninja import ModelSchema
    from pydantic import Field
    
    from legacy.models import LegacyCategory
    from future.models import Category
    
    
    class LegacyCategorySchema(ModelSchema):
        name: str = Field(alias="cat_name")
        description: str = Field(alias="cat_description")
        active: bool = Field(alias="cat_is_active")
    
        class Meta:
            fields = ["id"]
            model = Category
    
    
    @click.command()
    def main():
        categories = LegacyCategory.objects.all()
        for category in categories:
            data = LegacyCategorySchema.from_orm(category).dict()
            print(data)
            # save to a database or do something useful here
    

    More resources

    If you are curious about what Django Ninja is about, I recommend starting with their CRUD example: Final Code, and working backward. This will give you a good idea of what a finished CRUD Rest API looks like with Django Ninja.

    Wednesday March 6, 2024
  • Django

    ,

    Python

    New `django-startproject` update

    I updated my django-startproject project today to support the latest versions of Django, Python, Compose, and other tools I’m a fan of. I use django-startproject to spin up projects that need some batteries quickly, but not every battery.

    Features:

    • Django 5.0
    • Python 3.12
    • Docker Compose 3
    • Adds casey/just recipes/workflows (Just is a command runner, not a build tool)
    • Adds uv support

    uv is the newest addition, which is a Python package installer and pip-tools replacement. It’s not a 100% drop-in replacement for pip and pip-tools, but it cuts my build times in half, and I have yet to hit any significant show-stoppers.

    Saturday March 2, 2024
  • Django

    ,

    Python

    Using Django Q2

    I’m long overdue to write about how Django Q2 has become part of my development toolkit. As the maintained successor to Django Q, Django Q2 extends Django to handle background tasks and scheduled jobs.

    Django Q2 is flexible in managing tasks, whether sending out daily emails or performing hourly tasks like checking RSS feeds. The project works seamlessly with Django, making it one of the more straightforward background task solutions to integrate into your projects.

    Using Django Q2 involves passing a method or a string reference to a method to an async_task() function, which will run in the background.

    One feature of Django Q2 that particularly impresses me is its adaptability to various databases. Whether your project uses the default Django database or something more scalable like Redis, Django Q2 fits perfectly. This flexibility means that a database queue suffices without any hiccups for most of my projects, even those that are small to medium.

    Unlike other task queues that require managing multiple processes or services, Django Q2 keeps it simple. The only necessity is to have the qcluster management command running, which is a breeze compared to other task queues because you only need to run one service to handle everything.

    Django Q2’s flexibility, ease of use, and seamless integration with Django make it an excellent tool to reach for when you need background tasks.

    Tuesday February 27, 2024
  • Django

    ,

    Python

    Fetch the contents of a URL with Django service

    For the last few months, I have been using the cooked.wiki recipe-saving website, which initially impressed me because of how easy the website’s API is to use.

    To use the service, all one has to do is prepend any website that contains a food recipe with https://cooked.wiki/, and you get the recipe without a coming-of-age discovery story.

    This is a fun pattern, so I wrote my own in Django to illustrate how to build a Django view, which accepts a URL like http://localhost:8000/https://httpbin.org/get/ where https://httpbin.org/get/ will be fetched and the contents stored for processing.

    # views.py 
    import httpx
    
    from django.http import HttpResponse
    from urllib.parse import urlparse
    
    def fetch_content_view(request, url: str) -> HttpResponse:
        # Ensure the URL starts with http:// or https://
        parsed_url = urlparse(url)
        if parsed_url.scheme in ("http", "https"):
            try:
                response = httpx.get(url)
    
                # Check for HTTP request errors
                httpx.raise_for_status()
                content = httpx.content
    
                # TODO: do something with content here...
                assert content
                return HttpResponse(f"{url=}")
    
            except httpx.RequestException as e:
                return HttpResponse(f"Error fetching the requested URL: {e}", status=500)
    
        else:
            return HttpResponse("Invalid URL format.", status=400)
    
    # urls.py
    
    from django.urls import path
    from . import views
    
    urlpatterns = [
        # other URL patterns here...
        ...
        
        path("<path:url>/", views.fetch_content_view, name="fetch_content"),
    ]
    

    If you create your fetch the contents of a URL-like service, please consider putting it behind authentication to avoid someone discovering it and using it to DDOS someone’s website. I recommend throttling the view to prevent overloading a website by spamming requests to it.

    Updated: I updated the example to switch from the python-requests references to the HTTPX library.

    Saturday February 24, 2024
  • Django

    ,

    Python

    Scratching Itches with Python and ChatGPT

    A few times a week over the last several months, I have paired with ChatGPT to work on Python scripts that solve problems that I would otherwise have spent less time on. Some might feel too niche or even too tedious that I would otherwise not take the time to work on. Most of the time, these are scratching an itch and solving a problem on my mind.

    Because of the time constraint, I have been impressed with the results. I usually spend 10 to 15 minutes prompting ChatGPT, then I spend 10 to 15 minutes refactoring the script, adding Typer, and refining the code. Sometimes, this involved copying all or parts of my script and pasting it back into ChatGPT to have it refine or refactor some section of code.

    YouTube Playlist to Markdown file

    My YouTube playlist to markdown script is helpful for quickly getting a list of video URLs and titles back from a YouTube playlist. I have used this for DjangoCon US and a few other conferences to help collect links for social media and a few times for the Django News Newsletter.

    ChatGPT even documented the process, including links for how to set permissions for the YouTube API.

    Use Playwright to pull data out of the Django Admin

    I have database access for most projects, but I recently needed to export a list of RSS feeds from the admin of a Django website. ChatGPT could quickly write a Playwright script to log in to the website, access a list page, and pull the feed field from the detail page. The script generated a JSON feed and could understand pagination and how to page through the links.

    GitHub Issues and Pull Request templates

    For the Awesome Django project, I asked ChatGPT to generate GitHub Issues and Pull Request templates based on my criteria. Once the templates were complete, I prompted ChatGPT to help me write a script that uses the GitHub API to read a pull request and validate the answers filled out while adding some other contextual data that makes it easier to verify the request.

    Modeling HTML with PyDantic

    I am still trying to figure out what to do with the project, but I asked ChatGPT to use Pydantic to create a class that could represent HTML tags. Once I was happy with the API, I asked ChatGPT to represent all HTML tags. After a few more prompts, I could read, write, and represent an HTML document using this script using Pydantic.

    Outro

    I’m still determining how useful these scripts are, but I have enjoyed these quick sessions to write a one-off script or to solve problems that come up a few times a year that never seemed worth the time spent trying to write it from scratch.

    Thursday February 22, 2024
  • Django

    ,

    Python

    Using Chamber with Django and managing environment variables

    One of my favorite hosting setups is getting a cheap slice from Digital Ocean or your favorite provider, installing Docker Compose and Tailscale, and then fire-walling everything off except for port 443.

    Whenever I want to host a new project, I copy a docker-compose.yml file to the server, and then I start it with `docker compose up -d'.

    I run Watchtower in Docker on the server, which looks for new Docker images from GitHub Packages, pulls them, and restarts any updated containers.

    I can update my projects, git push changes, and GitHub Actions will build and store a new container image for me.

    My main pain point was juggling environment variables until someone pointed out Chamber, which manages environment variables well.

    Since creating this setup, I have shared several GitHub Gists with curious friends, and my goal of this post is to serve as more of an overview of options than it is to be a comprehensive guide to using Chamber.

    Prerequisites

    You’ll need an AWS account, some essential Docker and Compose knowledge, and to follow Chamber’s Installing instructions.

    Setting up my environment

    Ironically, my goal of eliminating individual environment variables led me to need four environment variables to bootstrap Chamber itself. The environment variables I’m using:

    • AWS_ACCESS_KEY_ID
    • AWS_REGION
    • AWS_SECRET_ACCESS_KEY
    • CHAMBER_KMS_KEY_ALIAS=aws/ssm

    Dockerfile Setup

    To make running Chamber running more straightforward, I used the segment/chamber Docker image, copied the /bin/chamber binary into my image, and configured it to run it as a ENTRYPOINT.

    FROM segment/chamber:2.14 AS chamber
    
    FROM python:3.11-slim-buster AS dev
    
    FROM dev AS production
    ...
    COPY --from=chamber /chamber /bin/chamber
    ENTRYPOINT ["/bin/chamber", "exec", "django-news.com/production", "--"]
    

    I prefer to namespace these variables based on the project and the environment I’m referencing, like django-news.com/production.

    I am using a Docker entrypoint so that my secrets/environment variables work by default, whether running the image or overriding the default command, so I may shell into my container.

    Docker Compose Setup

    services:
      web:
        entrypoint: /bin/chamber exec django-news.com/production --
        command: gunicorn config.wsgi --bind 0.0.0.0:8000
        ...
    

    Please note that the entrypoint line is optional if you set it in your ENTRYPOINT setting in your DOCKERFILE.

    Using Chamber

    Now that you have seen how we use Chamber in Docker and Docker Compose, this is how we get things into Chamber.

    Listing our projects

    $ chamber list-services
    
    $ chamber list django-news.com/production
    

    Write new settings

    $ chamber write django-news.com/production DJANGO_DEBUG true
    

    Delete an existing setting

    $ chamber delete django-news.com/production DJANGO_DEBUG
    

    Export our settings into a dotenv (.env) file

    $ chamber export --format=dotenv django-news.com/production
    

    Consuming an env variable from Django

    The environs project is my go-to for parsing environment variables. Here is an example of how to toggle Django’s debug mode.

    # settings.py
    import environs
    
    env = environs.Env()
    
    DEBUG = env.bool("DJANGO_DEBUG", default=False)
    

    Conclusion

    I’m happy to manage my environment variables from the command line without syncing files. Using Chamber with KMS increased my monthly AWS bill by $0.01, which is money well spent for the flexibility of using Chamber.

    Alternatives

    I had a good experience using the 1Password CLI for a recent client project to share and load secrets into the environment. If you are working with a team, consider checking it out for your team in case it’s a good fit. Check out their Load secrets into the environment docs.

    Wednesday February 21, 2024