• Django

    ,

    Python

    🗓️ December 21, 2024, is Volunteer Responsibility Amnesty Day

    December 21 is Volunteer Responsibility Amnesty Day, one of two days every year that allows you to build out your responsibility inventory and decide what you have time for.

    Volunteer Responsibility Amnesty Day is about checking with yourself, and ending the commitments you need to end – maybe by taking a break, or by rotating it on to someone else, or by sunsetting a project.

    It’s one of the best ideas in tech because it creates space to de-commit from projects we might not otherwise have time for, but we feel like we’d be letting people down by stepping back from. It’s a wonderful excuse to step back from projects that have been weighing on your mind.

    What I’m stepping down from

    This year, I decided it was time to step down from the Paramiko, Fabric, and Invoke Triage Teams because I had not been active there in years. I volunteered to help triage issues that were always a bit over my head.

    I joined when Jeff Forcier needed some extra hands, and it might have helped get another maintainer signed up who could help out more. At PyCon US this year, I got to hang out with Jeff. I gave him a heads-up that I felt like I was excess baggage to the project because my good intentions were no longer turning into helping out the project. I have much respect for Jeff (#teamjeff) and all three of these projects.

    If you have free cycles and can help with these projects, please check out the open issues and see if you can help. I use all three beneficial projects in one way or another every week.

    What you can do

    Suppose you are signing up for projects but are not fulfilling your commitment. In that case, I encourage you to use Volunteer Responsibility Amnesty Day as motivation to step down from whatever project or responsibility you may have.

    I also want to call out some members of the Python and Django communities. If you are signing up to help with projects you know you won’t have time for, please step back and encourage someone else to help with those projects and roles. If you know that famous face you see plastered on every website and they are friends, please ask them if they know about Volunteer Responsibility Amnesty Day and share your list. That nudge may help them decide if they are taking on too much.

    Friday December 20, 2024
  • Django

    ,

    Python

    🤷 Why do the Django and Python communities use so many Google Forms?

    Last week, I wrote but didn’t publish some notes about why the Django and Python communities use so many Google Forms.

    The simple answer is that Google Forms are quick and easy to set up, and data can be easily exported via Google Sheets or a CSV file.

    Getting data in and out of Django isn’t hard, but why isn’t it as easy as using Google Forms?

    I’d love to see a Django version of Google Forms that we can all use.

    Has anyone solved this problem?

    From a technical perspective, Django ships with a JSONField, which can store any form data we want without having to migrate databases. Turning a flat JSONField into CSV is already a solved problem.

    Please note: I am not looking for a Django forms wrapper or your opinions about why Django should use your fancy Django forms tool.

    Tuesday December 17, 2024
  • Django

    ,

    Python

    New project to shorten django-admin to django because we are not monsters

    One of the biggest mysteries in Django is why I have to run django-admin from my terminal instead of just running django. Confusingly, django-admin has nothing to do with Django’s admin app.

    If you have ever wondered why and wanted to type django from your terminal, my new project, django-cli-no-admin solves this problem for you.

    I looked at several package names on PyPI, including django-cli, which I liked the best (someone is name squatting this package.)

    I gave up and went with django-cli-no-admin for lack of a better name.

    # new school
    uv pip install django-cli-no-admin
    
    # old school
    pip install django-cli-no-admin
    
    # to use it...
    django --version 
    

    This tool aliases Django’s django-admin script does but we shorted the name by 50%:

    [project.scripts]
    django = "django.core.management:execute_from_command_line"
    

    Should Django adopt this?

    Yes. But we can leave django-admin alone since we have ~20 years of history referencing it.

    How long has this lived in your head?

    Almost two decades.

    Where is the code?

    https://github.com/jefftriplett/django-cli-no-admin

    Saturday December 14, 2024
  • Weeknotes

    ,

    Django

    ,

    Python

    ,

    Today I Learned

    📓 Weeknotes for Week 48: November 25 to December 1

    Family

    We drove up to Chicago to see family for five days. It’s a 7.25-hour drive if not stopping were an option, but we usually have to stop at least two or three times to re-fuel, eat, and have a few bathroom breaks. Both of my kids are pro-travelers. Give them an iPad with an offline cache of their favorite Disney and Netflix movies and series, plus some snacks, and they are good to go for 8 to 16 hours. On our last trip back from Omaha, they complained that it was too short because we didn’t stop on our 2.5-hour drive back.

    We take turns driving, and through the magic of tethering, I’m surprised that I can comfortably use my laptop from the car for half the trip.

    Seeing family was good. There are five kids, ranging from two to nine, but this year everyone is out of diapers, everyone can communicate their needs, and everyone plays together nicely.

    We decided to avoid dinner prep drama for Thanksgiving and go out for Mexican food and margaritas. This was an excellent idea. The staff was super friendly, the food was excellent, the margaritas were the right amount of salty and sweet, and everyone got to pick something they enjoyed. There was no food prep or cleanup stress. Overall, our bill for 10 people plus a very generous tip, even after a service fee, was much less than it costs to try to feed 10 people a traditional spread.

    Work

    It was a short, two-day workweek for me. I helped a client with an inventory project running a week or two behind on their side. The timing wasn’t great because it all landed on my lap the day before I was heading out of town, and it took a few days to run. I hate k8s even more than before.

    My main client is missing a tool or two to nail the pulse. I have thought about this because everything takes less time to complete but more time to work on the problem, so things are stretched out. The holiday break was the mental reset I needed to know how to manage this for the next month until our winter break.

    Community Work

    I skipped all Office Hours this week, but it was a busier few days for my DSF CoC WG card. We reset/rebooted the WG a month ago and have more members and communication. It’s taking more time to reset and settle into a healthy normal.

    Side projects

    Side Quests

    Calendar Agent

    I spent some free time over the break working on an app that lets ChatGPT control my work calendar. I got the basics up quickly, and my Calendar Agent (Bot?) can look at what’s scheduled on my calendar and create new events for me. The hardest part was navigating Google permissions (I hate them) so that I could access my calendar through Python. Once I got that setup, I used Claude Projects to help me write most of the code.

    I ran into an issue with tool calling because I wanted to let the Agent query the time for a day before creating the event. I shifted this logic into my create event function as a fix and realized I was way overthinking it. Now, I have an interesting agent. Still, I have yet to figure out how to run it outside my terminal, which isn’t helpful.

    Entertainment

    I mostly watched football and basketball this week, but I started a few shows from jefftriplett.com/now/

    New gear

    Since it was Black Friday, I did some shopping, but not as much as in previous years. I noticed Mastodon vibes are anti-Black Friday, whereas Blue Sky and X are more about saving money and getting a deal. More of my friends shared deals over Slack and Discord than in previous years.

    I picked up a five-pack of 32 GB microSD drives for my NanoKVMs, which I hope will be delivered next week. I also bought the family an Air Fryer convection oven combo unit, which may replace our old toaster and be helpful.

    Next week

    We get to start our LEGO Advent Calendars. I started this tradition with my son when he was two or three. My daughter will be three in a few weeks, so she is now old enough to join in the tradition, too. She requested the Disney Princess calendar, and my son has the Star Wars set. We skipped the traditional set because one can only have so many Santa Clauses.

    Sunday December 1, 2024
  • Python

    ,

    UV

    ,

    Today I Learned

    🤷 UV does everything or enough that I'm not sure what else it needs to do

    UV feels like one of those old infomercials where it solves everything, which is where we have landed in the Python world.

    I have had several discussions with friends about UV, and even when we talk about it during my weekly(ish) office hours, the list has grown to an ever-growing number of options.

    UV started as a quicker way of installing Python packages, and now it’s easier to tell people that UV does everything and to focus on what it doesn’t do.

    My favorite feature is that UV can now bootstrap a project to run on a machine that does not previously have Python installed, along with installing any packages your application might require.

    Here is my incomplete list of what UV does today:

    • uv pip install replaces pip install
    • uv venv replaces python -m venv
    • uv pip compile replaces pip-tools compile
    • uv pip sync replaces pip-tools sync
    • uv run replaces pipx
    • uv tool run replaces pipx
    • uv python replaces pyenv, asdf, mise, and several other like-minded tools
    • uv build - Build your Python package for pypi
    • uv publish - Upload your Python package to pypi
    • astral-sh/setup-uv brings UV to GitHub Actions
    • ghcr.io/astral-sh/uv:latest brings UV and Python to Docker

    I copied these four from uv --help, which feels like poetry features.

    • uv add - Add dependencies to the project
    • uv remove - Remove dependencies from the project
    • uv sync - Update the project’s environment
    • uv lock - Update the project’s lockfile

    So what doesn’t UV do?

    UV does a lot, but it still needs to do everything.

    • UV doesn’t run custom scripts defined in our pyproject.toml like npm-run-script allows. Thank you to @command_tab for jogging my memory.
    • UV doesn’t convert my non-UV-based projects to UV. Converting is more about prefixing and replacing my commands to switch over.
    • UV doesn’t manage, and bump version numbers like the BumpVer, and others do.
    • UV doesn’t manage pre-commit like hooks. This is a long shot, but I’d love to see support via pyproject.toml.
    • UV doesn’t replace Python, nor should it.
    Sunday November 3, 2024
  • Python

    ,

    Docker

    ,

    UV

    ,

    Today I Learned

    📓 My notes on publishing a Python package with UV and building a custom GitHub Action for files-to-claude-xml

    My new Python application files-to-claude-xml is now on PyPI, which means they are packaged and pip installable. My preferred way of running files-to-claude-xml is via UV’s tool run, which will install it if it still needs to be installed and then execute it.

    $ uv tool run files-to-claude-xml --version
    

    Publishing on PyPi with UV

    UV has both build and publish commands, so I took them for a spin today.

    uv build just worked, and a Python package was built.

    When I tried uv publish, it prompted me for some auth settings for which I had to log in to PyPI to create a token.

    I added those to my local ENV variables I manage with direnv.

    export UV_PUBLISH_PASSWORD=<your-PyPI-token-here>
    export UV_PUBLISH_USERNAME=__token__
    

    Once both were set and registered, uv publish published my files on PyPI.

    GitHub Action

    To make files-to-claude-xml easier to run on GitHub, I created a custom action to build a _claude.xml from the GitHub repository.

    To use this action, I wrote this example workflow, which runs from files-to-claude-xml-example

    name: Convert Files to Claude XML
    
    
    on:
      push
    
    
    jobs:
      convert-to-xml:
        runs-on: ubuntu-latest
        steps:
        - uses: actions/checkout@v4
        - name: Convert files to Claude XML
          uses: jefftriplett/files-to-claude-xml-action@main
          with:
            files: |
              README.md
              main.py          
            output: '_claude.xml'
            verbose: 'true'
        - name: Upload XML artifact
          uses: actions/upload-artifact@v4
          with:
            name: claude-xml
            path: _claude.xml
    

    My GitHub action is built with a Dockerfile, which installs files-to-claude-xml.

    # Dockerfile
    FROM ghcr.io/astral-sh/uv:bookworm-slim
    
    
    ENV UV_LINK_MODE=copy
    
    
    RUN --mount=type=cache,target=/root/.cache/uv \
        --mount=type=bind,source=uv.lock,target=uv.lock \
        --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
        uv sync --frozen --no-install-project
    
    
    WORKDIR /app
    
    
    ENTRYPOINT ["uvx", "files-to-claude-xml"]
    

    To turn a GitHub repository into a runnable GitHub Action, an action.yml file needs to exist in the repository. This file describes the input arguments and which Dockerfile or command to run.

    # action.yml
    name: 'Files to Claude XML'
    description: 'Convert files to XML format for Claude'
    inputs:
      files:
        description: 'Input files to process'
        required: true
        type: list
      output:
        description: 'Output XML file path'
        required: false
        default: '_claude.xml'
      verbose:
        description: 'Enable verbose output'
        required: false
        default: 'false'
      version:
        description: 'Display the version number'
        required: false
        default: 'false'
    runs:
      using: 'docker'
      image: 'Dockerfile'
      args:
        - ${{ join(inputs.files, ' ') }}
        - --output
        - ${{ inputs.output }}
        - ${{ inputs.verbose == 'true' && '--verbose' || '' }}
        - ${{ inputs.version == 'true' && '--version' || '' }}
    

    Overall, this works. Claude’s prompting helped me figure it out, which felt fairly satisfying given the goal of files-to-claude-xml.

    Wednesday October 16, 2024
  • Django

    ,

    Python

    ,

    LLM

    🤖 I released files-to-claude-xml and new development workflows

    After months of using and sharing this tool via a private gist, I finally carved out some time to release files-to-claude-xml.

    Despite my social media timeline declaring LLMs dead earlier today, I have used Claude Projects and Artifacts.

    My workflow is to copy a few files into a Claude Project and then create a new chat thread where Claude will help me write tests or build out a few features.

    My files-to-claude-xml script grew out of some research I did where I stumbled on their Essential tips for long context prompts which documents how to get around some file upload limits which encourages uploading one big file using Claude’s XML-like format.

    With files-to-claude-xml, I build a list of files that I want to import into a Claude Project. Then, I run it to generate a _claude.xml file, which I drag into Claude. I create a new conversation thread per feature, then copy the finished artifacts out of Claude once my feature or thread is complete.

    After the feature is complete, I delete the _claude.xml file from my project and replace it with an updated copy after I re-run files-to-claude-xml.

    Features on the go

    One bonus of using Claude Projects is that once everything is uploaded, I can use the Claude iOS app as a sort-of notes app and development tool. I can start parallel conversation threads and have it work on new ideas and features. Once I get back to my desktop, I can pull these chat conversations up, and if I like the direction of the feature, I might use them. If not, I have wasted no time or effort on them. This also serves as a nice ToDo list.

    New workflows

    I am working on side projects further using this methodology. Sometimes, I would like to work on something casually while watching Netflix, but my brain shuts off from coding during the day. Instead of feeling bad that I haven’t added share links to a website or some feature I meant to add last week, I can pair Claude to work on it with me.

    I can also get more done with my lunch hours on projects like DjangoTV than I could have otherwise. Overall, I’m happy to have an on-demand assistant to pair with and work on new features and ideas.

    It’s also quicker to try out new ideas and projects that I would have needed to make time for.

    Alternatives

    Simon Willison wrote files-to-prompt, which I think is also worth trying. I contributed to the discussion, feedback, and document structure for the --cxml feature.

    I wrote files-to-claude-xml before Simon had cxml support and hoped to not release my version.

    However, after trying it out on several projects, my ignore/exclude list grew more significant than the files that I wanted to include in my project to send to Claude. I found it easier to generate a list of files to pass to mine instead of maintaining a long list to exclude.

    Saturday October 12, 2024
  • Django

    ,

    Python

    ,

    UV

    ⚙️ UV with GitHub Actions to run an RSS to README project

    For my personal GitHub profile, I list my activities, affiliations, and the latest updates from some of my projects.

    Historically, I have used JasonEtco/rss-to-readme GitHub Action to fetch a few RSS feeds or two and to update my README a few times a day.

    Overall, I’m happy with this setup. I used it on the Django News GitHub Organization to pull in newsletter issues, jobs, and the latest videos from our various projects. When I tried to install rss-to-readme in our repo, I was getting node12 errors. (Have I mentioned how much I loathe node/npm?).

    Instead of forking rss-to-readme and trying to figure out how to upgrade it, I used this as an excuse to “pair program” with Claude. We quickly built out a prototype using Python and the feedparser library.

    I would share the chat log, but it’s mostly me trying out a few different ways to invoke it before I settle on the finished approach. See the source code over on GitHub if you are curious: https://github.com/django-news/.github/blob/main/fetch-rss.py

    Once I had a working Python script that could fetch an RSS file and modify the README, I decided to run/deploy it using UV to see how minimal I could build out the GitHub Action.

    GitHub Action

    To run our fetch-rss.py script, we have four steps:

    1. actions/checkout Get a git checkout of our project.
    2. astral-sh/setup-uv Setup UV also installs Pythons for us. As a bonus, we enabled UV’s cache support, which will run much faster in the future unless we change something in our fetch-rss.py file.
    3. Run uv run fetch-rss.py ... to fetch our RSS feeds and write them to disk. uv run installs any dependencies and caches them before our fetch-rss.py runs.
    4. stefanzweifel/git-auto-commit-action If our README.md file has changed, save our changes and commit them back to git and into our README.

    Our schedule.yml GitHub Action workflow runs twice daily or whenever we push a new change to our repo. We also set workflow_dispatch, which gives us a button to run the script manually.

    # .github/workflows/schedule.yml
    name: Update README
    
    on:
      push:
        branches:
          - main
      schedule:
        # Once a day at 12 AM
        - cron: 0 12 * * *
      workflow_dispatch:
    
    jobs:
      update:
        runs-on: ubuntu-latest
    
        permissions:
          contents: write
    
        steps:
          - uses: actions/checkout@v4
    
          - name: Install uv
            uses: astral-sh/setup-uv@v3
            with:
              enable-cache: true
              cache-dependency-glob: |
                            *.py
    
          - name: Fetch our Feeds
            run: |
              # Fetch latest Django News Newsletter entries
              uv run fetch-rss.py \
                  --section=news \
                  --readme-path=profile/README.md \
                  https://django-news.com/issues.rss          
    
          - uses: stefanzweifel/git-auto-commit-action@v5
            with:
              commit_message: ":pencil: Updates README"
    

    Results

    Overall, I’m pleased with this solution. If I wanted to spend more time on it or re-use this workflow, I might turn it into a GitHub Action workflow so that we can call: django-news/rss-to-readme to use in other projects. For now, this is fine.

    I’m happy with the astral-sh/setup-uv and uv run steps because they save me from having to set up Python and then install our project dependencies as separate steps.

    I normally shy away from running Python workflows like this in GitHub Actions because they involve a lot of slow steps. This entire workflow takes 16 to 20 seconds to run, which feels fast to me.

    Saturday October 5, 2024
  • Django

    ,

    Python

    🎉 Announcing DjangoTV

    Friends, I’m launching a half-finished website idea that I have been playing around with for a while. djangotv.com is focused on promoting and searching Django videos to make discovery easier.

    I wanted to launch DjangoTV before I left DjangoCon US 2024, so I’m technically announcing it from the airport. Last year, I launched Django News Jobs during the Django sprints because I was annoyed by the state of the various Django job boards.

    After a year of hearing people complain about Django and Python YouTube videos not getting enough views, I decided to address the problem by building out a website to help organize and promote them.

    DjangoTV is not competing with pyvideo.org. PyVideo is a fantastic resource for the Python community, and one of my goals is to make it easier to get Django content on PyVideo, too.

    DjangoTV is incomplete, and I have many ideas to improve it, including backfilling some of our older conferences, adding presenters, and adding some other metadata to make it a useful community resource.

    I focused on DjangoCon videos to bootstrap the project, and we’ll slowly expand our archives. I have been extracting good quotes, adding transcriptions, and adding better summaries. Please expect this to change a lot. I even launched the website with DEBUG turned on for a few days before I turned it off.

    Your feedback is crucial to us. If you have any ideas or suggestions, please don’t hesitate to share them.

    Most importantly, names change, people change, and bodies change. It’s essential to me that we respect each other and handle this with care. If you have a concern or need us to pull a video, please don’t hesitate to reach out; we will take care of it.

    Friday September 27, 2024
  • Django

    ,

    Python

    ,

    Today I Learned

    🧳 DjangoCon US, Black Python Devs Leadership Summit, and Django Girls Durham

    I’m heading to Durham, NC, for seven days of DjangoCon US this Friday. This is my 10th year volunteering and the 9th year that DEFNA, the non-profit I co-founded, has run a DjangoCon US event. Here is an overview of the week.

    Black Python Devs Leadership Summit (Saturday)

    I’m attending and speaking on a discussion panel on Saturday at the Black Python Devs Leadership Summit. Tickets are free, and they will be streaming online in the afternoon. Donations are accepted and appreciated.

    Django Girls Durham (Saturday)

    Django Girls are hosting a Django workshop and teaching beginners a crash course on building their first website using Django.

    DjangoCon US Tutorials (Sunday)

    On Sunday morning, I’ll be volunteering and helping out at the tutorials. In the afternoon, we have a tradition of stuffing swag bags, which takes a big group and is a fun way to kick off the conference. You do not need a tutorial ticket or an organizer to help out. Ask at the registration desk, and they can direct you to when and where we are doing this.

    Django Social meetup (Sunday)

    My company REVSYS is sponsoring a DjangoSocial Raleigh/Durham Pre-DjangoCon Special meetup on Sunday evening before the conference kicks off. The meetup will be great for meeting other attendees the night before the conference.

    DjangoCon US Talks (Monday through Wednesday)

    The talks are great, but the busiest three days of the conference are also the busiest. There is always a lot going on, from sun up to sun down.

    DjangoCon US Sprints (Thursday and Friday)

    The sprints are one of my favorite parts of the conference. In past years, I have been so exhausted by the sprints that it’s hard to sit down and focus. It’s one of the best times to discuss Django and the Django ecosystem. If you have a project or want to find a project to help with, the sprints are great for getting your feet wet.

    Outro

    Tickets are still available if you live near Durham and want to attend. Both events have online and in-person options, so there is no pressure to make last-minute travel plans.

    If you live around Durham and want to meet up, please reach out. Let’s see if we can meet for coffee.

    Friday September 20, 2024
  • Django

    ,

    Python

    ,

    UV

    ,

    Today I Learned

    🤠 UV Roundup: Five good articles and a pre-commit tip

    I have written quite a bit about UV on my micro blog, and I am happy to see more and more people adopt it. I have stumbled on so many good articles recently that I wanted to share them because every article points out something new or different about why UV works well for them.

    If you are new to UV, it’s a new tool written by Astral, the creators of Ruff.

    I like UV because it replaces, combines, or complements a bunch of Python tools into one tool and user developer experience without forcing a UV way of doing it. UV effectively solves the question, “Why do I need another Python tool?” to do everyday Python tasks.

    Some reason I like UV after using it for months:

    • It’s a faster pip and is really, really fast
    • It can install and manage Python versions
    • It can run and install Python scripts
    • It can run single-file Python scripts along with their dependencies
    • It can handle project lock files

    While some people don’t care about UV being fast, it’s shaved minutes off my CI builds and container rebuilds, which means it has also saved me money and energy resources.

    Overall thoughts on UV

    Oliver Andrich’s UV — I am (somewhat) sold takes the approach of only using UV to set up a new Python environment. Oliver uses UV to install Python, aliases to call Python, and UV tool install to set up a few global utilities.

    Using UV with Django

    Anže Pečar’s UV with Django shows how to use UV to set up a new project with Django.

    Switching from pyenv to UV

    Will Guaraldi Kahn-Greene’s Switching from pyenv to uv was relatable for me because I also use pyenv, but I plan to slowly migrate to using only UV. I’m already halfway there, but I will have pyenv for my legacy projects for years because many aren’t worth porting yet.

    Using UV and managing with Ansible

    Adam Johnson’s Python: my new uv setup for development taught me to use uv cache prune to clean up unused cache entries and shows how he manages his UV setup using Ansible.

    Some notes on UV

    Simon Willison’s Notes on UV is an excellent summary of Oliver’s notes.

    A parting UV tip

    If you are a pre-commit fan hoping for a version that supports UV, the pre-commit-uv project does just that. I started updating my justfile recipes to bake just lint to the following uv run command, which speeds up running and installing pre-commit significantly.

    $ uv run --with pre-commit-uv pre-commit run --all-files
    pre-commit-uv
    

    If you are attending DjangoCon US…

    If you are attending DjangoCon US and want to talk UV, Django, Django News, Django Packages, hit me up while you are there.

    I’ll be attending, volunteering, organizing, sponsoring, and sprinting around the venue in Durham, NC, for the next week starting this Friday.

    We still have online and in-person tickets, but not much longer!

    Thursday September 19, 2024
  • Django

    ,

    Python

    ,

    Office Hours

    📅 Office Hours Fall Update

    This Friday, I am hosting Office Hours before I travel to DjangoCon US (organizer) and the Black Python Devs Leadership Summit (speaker) in Durham, NC.

    This Friday will be my last session before a two-week break, but I will resume Office Hours again on October 4th.

    • Friday, September 13th, 2024, 2:30 pm to 4:30 pm
    • Friday, September 20th, 2024 - No office hours
    • Friday, September 27th, 2024 - No office hours
    • Friday, October 4th, 2024, 2:30 pm to 4:30 pm

    High-level details

    ℹ️ Anyone can join office hours. Many join because they work remotely, miss seeing faces, and miss the random conversations when a small group hangs out.

    ℹ️ Our office hours are a collaborative space where we can discuss our ongoing projects, catch up, and work together to wrap up our week on a productive note.

    🙏 As always, everyone is welcome to join, whether you’re a regular attendee or joining for the first time. If you are curious, reach out.

    ✅ If you need any additional details, feel free to send me a message or check out the gist from our previous sessions, where you’ll always find the most recent Zoom link ⚠️

    Thursday September 12, 2024
  • Django

    ,

    Python

    🚫 Stop scheduling security updates and deprecating major features over holidays

    I know people outside the US 🙄 at this, but please stop releasing major security updates and backward incompatible changes over major US, international, and religious holidays.

    Given that major security updates are embargoed and scheduled weeks and months in advance, it’s essential to coordinate and avoid conflicts. A simple check of the calendar before scheduling announcements can prevent such issues.

    Even if you give everyone two weeks' notice, aka what GitHub just did, wait to schedule them for release over a holiday weekend.

    Historically, the Python and Django communities have also been guilty of this, so I’m not just finger-pointing at GitHub. We can all do better here.

    Update: 100% unrelated to this: Django security releases issued: 5.1.1, 5.0.9, and 4.2.16. Thank you, Natalia (and Sarah) for scheduling this after the US is back from a major holiday.

    Tuesday September 3, 2024
  • Django

    ,

    Python

    ,

    UV

    🚜 Using Claude 3.5 Sonnet to refactor one of Brian Okken's Python projects

    Brian Okken posted and published his Top pytest Plugins script and then a follow-up post, Finding the top pytest plugins, which was pretty cool.

    I have written a few throw-away scripts, which William Vincent wrote about and updated a few times in the Top 10 Django Third-Party Packages (2024) and The 10 Most-Used Django Packages (2024).

    These efforts are powered by Hugo van Kemenade’s excellent Top PyPI Packages.

    This inspired me to fork Brian’s top-pytest-plugins project, which I updated to support passing in other package names like “django” to get a rough estimate of monthly package downloads.

    The refactored project is jefftriplett/top-python-packages.

    Please note: Looking at the package name doesn’t scale as well for projects that have their own Trove classifiers. For a project like pytest, it works well. Many of the top packages may not even have Django in their name for a project like Django. Some projects may even actively discourage a project from using their project in their package’s name for trademark reasons. So, YMMV applies here.

    Prompts

    I added uv run support, which I have written about a lot lately.

    I also copied the top_pytest.py file into a Claude 3.5 Sonnet session, and I let it handle the whole refactor. It even handled adding the PEP 723 new package dependencies without me asking it to.

    In case it’s useful to anyone, here are my prompts:

    ## Prompt:
    Please update this script to use a rich table.
    
    
    ## Prompt:
    Please update the table styles to be ascii so I can copy and paste it into a markdown doc
    
    
    ## Prompt:
    Please remove the description column
    
    
    ## Prompt:
    Please change all PyTest and pytest references to Django and django
    
    
    ## Prompt:
    Please add back `if 'django' in project.lower() and 'django' != project.lower():`
    
    
    ## Prompt:
    please remove the \*# Export to markdown section. I can just pipe the output \*
    
    
    ## Prompt:
    Please add the typer library.
    
    
    ## Prompt:
    Please remove days and limit
    
    
    ## Prompt:
    Please refactor the script to allow me to pass the package name instead of django. You can default to django though.
    
    
    This way I can pass pytest or flask or other projects.
    
    
    ## Prompt:
    Please change the default Table box type to MARKDOWN
    

    Outro

    I don’t usually write about Claude or prompts, but the tool has been handy lately.

    If you have had some similar successes, let me know. I have been exploring some rabbit holes, and it’s changing the way I approach solving problems.

    Sunday August 25, 2024
  • Django

    ,

    Python

    ,

    UV

    ,

    Today I Learned

    📓 UV Run Django Notes

    I wanted to know how hard it would be to turn one of my django-startproject projects into a uv run friendly project. As it turns out, it worked, and the steps were more than reasonable.

    Before the PEP 723’ing…

    I started with a fairly vanilla manage.py that Django will give you after running python -m manage startproject.

    """Django's command-line utility for administrative tasks."""
    
    import os
    import sys
    
    
    def main():
        """Run administrative tasks."""
        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
        try:
            from django.core.management import execute_from_command_line
        except ImportError as exc:
            raise ImportError(
                "Couldn't import Django. Are you sure it's installed and "
                "available on your PYTHONPATH environment variable? Did you "
                "forget to activate a virtual environment?"
            ) from exc
        execute_from_command_line(sys.argv)
    
    
    if __name__ == "__main__":
        main()
    

    shebang

    Then we add #!/usr/bin/env -S uv run to the top of our manage.py file.

    Next, we make our manage.py executable and try to run it.

    $ chmod +x manage.py
    $ ./manage.py
    ModuleNotFoundError: No module named 'django'
    

    Our script ran, but Python couldn’t find Django. To tell our script to install Django, we can use uv add—- script to add it.

    $ uv add --script manage.py django
    Updated `manage.py`
    $ ./manage.py
    ...
    
    Type 'manage.py help <subcommand>' for help on a specific subcommand.
    
    Available subcommands:
    
    [django]
        check
        compilemessages
        createcachetable
        dbshell
        diffsettings
        dumpdata
        flush
        inspectdb
        loaddata
        makemessages
        makemigrations
        migrate
        optimizemigration
        runserver
        sendtestemail
        shell
        showmigrations
        sqlflush
        sqlmigrate
        sqlsequencereset
        squashmigrations
        startapp
        startproject
        test
        testserver
    Note that only Django core commands are listed as settings are not properly configured (error: No module named 'environs').
    

    Django worked as expected this time, but Python could not find a few third-party libraries I like to include in my projects.

    To add these, I passed the other four to uv add --script which will add them to the project.

    $ uv add --script manage.py django-click "environs[django]" psycopg2-binary whitenoise
    Updated `manage.py`
    ...
    $ ./manage.py
    ...
    

    Our Django app’s manage.py works when we run it.

    After the PEP 723’ing…

    After we installed our dependencies in our manage.py file, they were added to the top of the file between the /// blocks.

    #!/usr/bin/env -S uv run
    # /// script
    # requires-python = ">=3.10"
    # dependencies = [
    #     "django",
    #     "django-click",
    #     "environs[django]",
    #     "psycopg2-binary",
    #     "whitenoise",
    # ]
    # ///
    """Django's command-line utility for administrative tasks."""
    
    import os
    import sys
    
    
    def main():
        """Run administrative tasks."""
        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
        try:
            from django.core.management import execute_from_command_line
        except ImportError as exc:
            raise ImportError(
                "Couldn't import Django. Are you sure it's installed and "
                "available on your PYTHONPATH environment variable? Did you "
                "forget to activate a virtual environment?"
            ) from exc
        execute_from_command_line(sys.argv)
    
    
    if __name__ == "__main__":
        main()
    
    Friday August 23, 2024
  • Python

    ,

    UV

    🐍 Python UV run with shebangs

    This UV shebang trick that Simon Willison linked up is a nice pattern, and I plan to rebuild some of my one-off scripts in my dotfiles using it.

    Here is a demo that will print “hello python” using the Python Branding colors using the Rich library while letting UV install and manage rich for you.

    #!/usr/bin/env -S uv run
    # /// script
    # requires-python = ">=3.10"
    # dependencies = [
    #     "rich",
    # ]
    # ///
    
    from rich.console import Console
    from rich.theme import Theme
    
    python_theme = Theme(
        {
            "pyyellow": "#ffde57",
            "pyblue": "#4584b6",
        }
    )
    
    console = Console(theme=python_theme)
    
    console.print("[pyyellow]hello[/pyyellow] [pyblue]python[/pyblue]", style="on #646464")
    

    Assuming you have UV installed, and you save and chmod +x this file as hello-python.py, then you should be able to run it via ./hello-python.py.

    I suspect I can more easily bootstrap new machines using this trick without fewer worries about polluting my global system packages.

    Thursday August 22, 2024
  • Python

    ,

    UV

    🐍 UV Updates and PEP 723: Simplifying Python Packaging and Scripting

    The uv: Unified Python packaging update brings fresh air to the Python community, with several improvements streamlining the development process. One exciting addition is an early preview of PEP 723, also known as Single-file scripts.

    The Single-file scripts feature particularly caught my attention due to its potential to simplify the distribution and execution of small Python projects. Streamlining the process is highly appealing to someone who frequently creates GitHub Gists and shares them privately and publicly.

    With this new feature, I can now instruct users to run uv run main.py without explaining what a venv or virtualenv is, plus a long list of requirements that need to be passed to pip install.

    I had the opportunity to test this feature over lunch today. While adding libraries to the script was straightforward, I encountered a few hurdles when I forgot to invoke uv run in my virtual environment (venv). This makes sense, given that it’s a new habit, but it highlights the importance of adapting to changes in our development workflow.

    Overall, the UV: Unified Python packaging update and the introduction of Single-file scripts mark a significant step in simplifying Python development. As developers become more familiar with these improvements, we expect increased adoption and smoother collaboration on small-scale projects.

    Bonus Example

    I looked through some of my recent visits, and one I recently shared with a few conference organizer friends was a one-off script I used to read several YouTube video JSON files that I’m using to bootstrap another project. It was the first time I used DuckDB to make quick work of reading data from a bunch of JSON files using SQL.

    Overall, I was happy with DuckDB and what PEP 723 might bring to the future of Python apps, even if my example only does a little.

    # To run this application, use:
    #   uv run demo-duckdb.py
    #
    # /// script
    # requires-python = ">=3.10"
    # dependencies = [
    #     "duckdb",
    #     "rich",
    #     "typer",
    # ]
    # ///
    import duckdb
    import typer
    
    from rich import print
    
    
    def main():
        result = duckdb.sql("SELECT id,snippet FROM read_json('json/*.json')").fetchall()
    
        for row in result:
            id, snippet = row
            print("-" * 80)
            print(f"{id=}")
            print(f"{snippet['channelTitle']=}")
            print(f"{snippet['title']=}")
            print(f"{snippet['publishedAt']=}")
            print(snippet["description"])
            print(snippet["thumbnails"].get("maxres") or snippet.get("standard"))
            print()
    
    
    if __name__ == "__main__":
        typer.run(main)
    

    Overall, the future is bright with UV and PEP 723 may bring us. I’m excited to have more one-file Python apps that are easier to share and run with others.

    PEP 723 also opens the door to turning a one-file Python script into a runnable Docker image that doesn’t even need Python on the machine or opens the door for Beeware and Briefcase to build standalone apps.

    Wednesday August 21, 2024
  • Django

    ,

    Python

    ,

    Office Hours

    💼 Office Hours this Friday, August 9th

    I am hosting Office Hours this Friday, August 9th, 2024, from 2:30 pm to 4:30 pm Central Time.

    💼 I plan to upgrade some projects to work with Django 5.1 and possibly Python 3.13 as time permits. I have a few Django Packages tasks to finish too.

    ℹ️ Anyone can join office hours.

    ℹ️ Our office hours are a collaborative space where we can discuss our ongoing projects, catch up, and work together to wrap up our week on a productive note.

    🙏 As always, everyone is welcome to join, whether you’re a regular attendee or joining for the first time.

    ✅ If you need any additional details, feel free to send me a message or check out the gist from our previous sessions, where you’ll find the Zoom link ⚠️

    I look forward to seeing everyone.

    Thursday August 8, 2024
  • Django

    ,

    Python

    ,

    Today I Learned

    ⬆️ Which Django and Python versions should I be using today?

    Django 5.1 was released, and I was reminded of the article I wrote earlier this year about Choosing the Right Python and Django Versions for Your Projects.

    While I encouraged you to wait until the second, third, or even fourth patch release of Django and Python before upgrading, I received a bit of pushback. One interesting perspective claimed that if everyone waits to upgrade, we don’t find critical bugs until the later versions. While that may be plausible, I don’t believe that the dozens of people who read my blog will be swayed by my recommendation to wait for a few patch releases.

    I could have emphasized the potential risks of not testing early. Please start testing during the alpha and release candidate phase so that when Django 5.1 is released, your third-party applications will be ready and working on launch day, minimizing the risk of last-minute issues.

    Today, I tried to upgrade Django Packages to run on Django 5.1 to see if our test suite would run on Django 5.1, and it very quickly failed in CI due to at least one package not supporting 5.1 yet. Even if it had passed, I’m 90% sure another package would have failed because that’s the nature of running a new major Django or Python release on day one. Even if the third-party package is ready, the packaging ecosystem needs time to catch up.

    Which version of Django should I use today?

    I’m sticking with Django 5.0 until Django 5.1’s ecosystem has caught up. I plan to update the third-party packages I help maintain to have Django 5.1 support. After a few patch releases of Django 5.1 have come out and the ecosystem has time to catch up, I will try to migrate again.

    Which version of Python should I use today?

    I’m starting new projects on Python 3.12, with a few legacy projects still being done on Python 3.11. While I am adding Django 5.1 support, I plan to add Python 3.13 support in my testing matrixes to prepare everything for Python 3.13’s release this fall.

    Office hours

    I plan to spend some of my Office Hours this week working on Django 5.1 and Python 3.13 readiness for projects I maintain. Please join me if you have a project to update or would like some light-hearted banter to end your week.

    Wednesday August 7, 2024
  • Python

    ,

    Ollama

    ,

    Today I Learned

    🦙 Ollama Tool Calling Loose Notes

    I spent a few hours this week working with the Ollama project and trying to get tool calling to work with the LangChain library.

    Tool calling is a way to expose Python functions to a language model that allows them to be called. This will enable models to perform more complex actions and even call the outside world for more information.

    I haven’t used LangChain before, and I found the whole process frustrating. The docs were full of errors. I eventually figured it out, but I was limited to one tool call per prompt, which felt broken.

    Earlier today, I was telling a colleague about it, and when we got back from grabbing coffee, I thought I would check the Ollama Discord channel to see if anyone else had figured it out. To my surprise, they added and released Tool support last night, which allowed me to ditch LangChain altogether.

    The Ollama project’s tool calling example was just enough to help get me started.

    I struggled with the function calling syntax, but after digging a bit deeper, I found this example from OpenAI’s Function calling docs, which matches the format the Ollama project is following. I still don’t fully understand it, but I got more functions working and verified that I can make multiple tool calls within the same prompt.

    Meta’s Llama 3.1 model supports tool calling, and the two work quite well together. I am also impressed with Llama 3.1 and the large context window support. I’m running the 8B and 70B models on a Mac Studio, and they feel very close to the commercial APIs I have worked with, but I can run them locally.

    Embedding models

    Tonight, I tried out Ollama’s Embedding models example, and while I got it working, I still need to put practical data into it to give it a better test

    One more tip

    If you did not know Ollama can parse and return valid JSON, check out How to get JSON response from Ollama. It made my JSON parsing and responses much more reliable.

    Friday July 26, 2024
  • Python

    ,

    Ollama

    ,

    LLM

    ,

    Today I Learned

    🦙 Ollama Llama 3.1 Red Pajama

    For a few weeks, I told friends I was excited to see if the new Llama 3.1 release was as good as it was being hyped.

    Yesterday, Llama 3.1 was released, and I was impressed that the Ollama project published a release to Homebrew and had the models ready to use.

    ➜ brew install ollama
    
    ➜ ollama serve
    
    # (optionally) I run Ollama as a background service
    ➜ brew services start ollama
    
    # This takes a while (defaults to the llama3.1:8b model)
    ➜ ollama pull llama3.1:latest 
    
    # (optional) This takes a longer time
    ➜ ollama pull llama3.1:70b
    
    # (optional) This takes so long that I skipped it and ordered a CAT6 cable...
    # ollama pull llama3.1:405b
    

    To use chat with the model, you use the same ollama console command:

    ➜ ollama run llama3.1:latest
    >>> how much is 2+2?
    The answer to 2 + 2 is:
    4!```
    
    ## Accessing Ollama Llama 3.1 with Python
    
    The Ollama project has an [`ollama-python`](https://github.com/ollama/ollama-python) library, which I use to build applications. 
    
    My demo has a bit of flare because there are a few options, like `--stream,` that improve the quality of life while waiting for Ollama to return results. 
    
    ```python
    # hello-llama.py
    import typer
    
    from enum import Enum
    from ollama import Client
    from rich import print
    
    
    class Host(str, Enum):
        local = "http://127.0.0.1:11434"
        the_office = "http://the-office:11434"
    
    
    class ModelChoices(str, Enum):
        llama31 = "llama3.1:latest"
        llama31_70b = "llama3.1:70b"
    
    
    def main(
        host: Host = Host.local,
        local: bool = False,
        model: ModelChoices = ModelChoices.llama31,
        stream: bool = False,
    ):
        if local:
            host = Host.local
    
        client = Client(host=host.value)
    
        response = client.chat(
            model=model.value,
            messages=[
                {
                    "role": "user",
                    "content": \
                        "Please riff on the 'Llama Llama Red Pajama' book but using AI terms like the 'Ollama' server and the 'Llama 3.1' model."
                        "Instead of using 'Llama Llama', please use 'Ollama Llama 3.1'.",
                }
            ],
            stream=stream,
        )
    
        if stream:
            for chunk in response:
                print(chunk["message"]["content"], end="", flush=True)
            print()
    
    	else:
            print(f"[yellow]{response['message']['content']}[/yellow]")
    
    if __name__ == "__main__":
        typer.run(main)
    

    Some of my family’s favorite books are the late Anna Dewdney’s Llama Llama books. Please buy and support their work. I can’t read Llama 3.1 and Ollama without considering the “Llama Llama Red Pajama” book.

    To set up and run this:

    # Install a few "nice to have" libraries
    ➜ pip install ollama rich typer
    
    # Run our demo
    ➜ python hello-llama.py --stream
    
    Here's a riff on "Llama Llama Red Pajama" but with an AI twist:
    
    **Ollama Llama 3.1, Ollama Llama 3.1**
    Mama said to Ollama Llama 3.1,
    "Dinner's done, time for some learning fun!"
    But Ollama Llama 3.1 didn't wanna play
    With the data sets and algorithms all day.
    
    He wanted to go out and get some rest,
    And dream of neural nets that were truly blessed.
    But Mama said, "No way, young Ollama Llama 3.1,
    You need to train on some more NLP."
    
    Ollama Llama 3.1 got so mad and blue
    He shouted at the cloud, "I don't wanna do this too!"
    But then he remembered all the things he could see,
    On the Ollama server, where his models would be.
    
    So he plugged in his GPU and gave a happy sigh
    And trained on some texts, till the morning light shone high.
    He learned about embeddings and wordplay too,
    And how to chat with humans, that's what he wanted to do.
    
    **The end**
    

    Connecting to Ollama

    I have two Macs running Ollama and I use Tailscale to bounce between them from anywhere. When I’m at home upstairs it’s quicker to run a local instance. When I’m on my 2019 MacBook Pro it’s faster to connect to the office.

    The only stumbling block I ran into was needing to set a few ENV variables setup so that Ollama is listening on a port that I can proxy to. This was frustrating to figure out, but I hope it saves you some time.

    ➜ launchctl setenv OLLAMA_HOST 0.0.0.0:11434
    ➜ launchctl setenv OLLAMA_ORIGINS http://*
    
    # Restart the Ollama server to pick up on the ENV vars
    ➜ brew services restart ollama
    

    Simon Willison’s LLM tool

    I also like using Simon Willison’s LLM tool, which supports a ton of different AI services via third-party plugins. I like the llm-ollama library, which allows us to connect to our local Ollama instance.

    When working with Ollama, I start with the Ollama run command, but I have a few bash scripts that might talk to OpenAI or Claude 3.5, and it’s nice to keep my brain in the same tooling space. LLM is useful for mixing and matching remote and local models.

    To install and use LLM + llm-ollama + Llama 3.1.

    Please note that the Ollama server should already be running as previously outlined.

    # Install llm
    ➜ brew install llm
    
    # Install llm-ollama
    ➜ llm install llm-ollama
    
    # List all of models from Ollama
    ➜ llm ollama list-models
    
    # 
    ➜ llm -m llama3.1:latest "how much is 2+2?"
    The answer to 2 + 2 is:
    
    4
    

    Bonus: Mistral Large 2

    While I was working on this post, Mistral AI launched their Large Enough: Mistral Large 2 model today. The Ollama project released support for the model within minutes of its announcement.

    The Mistral Large 2 release is noteworthy because it outperforms Lllama 3.1’s 405B parameter model and is under 1/3 of the size. It is also the second GPT-4 class model release in the last two days.

    Check out Simon’s post for more details and another LLM plugin for another way to access it.

    Wednesday July 24, 2024
  • Weeknotes

    ,

    Django

    ,

    Python

    ,

    Today I Learned

    📓 Weeknotes for Week 28: July 8 to 14

    I’m running a week behind on this.

    This week was our first week back home without traveling in a month, and it felt good to be home. I had time to catch up on mowing the yard, and I treated the yard with an eco-safe mosquito repellent. Despite the hot weather, Sunday felt nice outside to be mosquito-free.

    I rolled my above-ground sprinkler system A few years ago, and I still need to install and run it this year. I wanted to get it this weekend, and here we are.

    Family

    I converted my daughter’s crib to a daybed over the weekend, and we have been using it for two nights and two naps without any issues. My son took to the board game Risk in Chicago, so I installed the iPad version and walked him through it. It was a pizza and tacos weekend because it was a long week.

    Work

    Occasionally, a project feels like you signed up for a race, but the distance keeps changing whenever you are within sight of the finish line. A project we have been finishing up keeps growing.

    Community Work

    Side projects

    • Django News Newsletter: We shipped issue #241.

    • Django News Jobs: This week, we picked up more jobs that weren’t from one source. I need to write a tool to help maintain this, but it’s a manageable load.

    • I bought a new domain name for a project this weekend. More on that soon.

    Side Quests

    • I dusted off my YouTube-to-Frontmatter tool and added the ability to pull playlists from a given username. I wrote the files out as JSON and used DuckDB to query them, which worked amazingly well.

    • I wrote an Amazon product image downloader for a few blog posts. When the product API did not work, I punted and had ChatGPT write a playwright scraper. It was faster and much less frustrating. I need this for several projects.

    • I cleaned up my sitemaps research tool.

    • I tried out a screenshots-to-code project and ran some 00s-era websites through it that I wish still existed. If someone wants to give me a few years of funding, I think we can make the web not suck again.

    Writing

    2024-07-14🔥 Why I deleted Firefox from my machines this weekend I no longer trust or believe in Mozilla, so I deleted Firefox from my machines this weekend. ➜ brew …

    2024-07-13🦆 DuckDB may be the tool you didn’t know you were missing 🤔 I haven’t fully figured out DuckDB yet, but it’s worth trying out if you are a Python …

    2024-07-12🚜 macOS Bartender app to Ice app I upgraded my Macs to macOS Sonoma a few weeks ago. While everything has been uneventful, the …

    2024-07-11🎮 8BitDo Golden/Silver Limited Edition controllers My favorite third-party video game hardware company, 8BitDo, announced its 11th-anniversary limited …

    2024-07-10📅 Office Hours for July 12th Office Hours returns this Friday, July 12th, 2024, from 2:30 pm to 4:30 pm Central Time. ℹ️ Anyone …

    2024-07-09🔓 Sharing is Caring: How a Simple Sudo Question Led to Better Solutions One of the fun discoveries of blogging is finding your article in search results while trying to …

    2024-07-08📓 Weeknotes for Week 27: July 1 to 7 The last week was a blur between the holiday, travel, and cramming a lot of work. My notes this week …

    Entertainment

    📺 Vikings: Valhalla

    📺 The Marvels (2023) - This movie was better than people gave it credit for. It wasn’t my favorite, but it was fun to watch.

    📺 Defending Jacob - I skipped to the end of this series.

    📺 The Last Thing He Told Me - I skipped to the end of this series.

    📺 Presumed Innocent - I surprised myself that I’m still keeping up with this series, but there are only a few weeks left.

    📺 The Acolyte - We are ready for the last episode.

    📺 Atlas (2024) - I didn’t go into this movie with any expectations, and I immensely enjoyed it.

    Next week

    I’m solo-parenting next weekend. I’m looking forward to hanging out with my kids and another weekend of being home.

    Saturday July 20, 2024
  • Python

    ,

    Today I Learned

    🗳️ My thoughts on the PSF Election results

    A few weeks ago, I wrote about this year’s PSF Election, three proposed bylaws changes, and how I intended to vote. I’m happy that the membership overwhelmingly approved all three proposed bylaw changes. Here is this year’s results.

    Merging Contributing and Managing member classes

    This change is a good step toward consolidating two membership classes and a commitment to acknowledging that all community contributions are important, not just code contributions.

    Simplifying the voter affirmation process by treating past voting activity as intent to continue voting

    If you voted in last year’s election, there are fewer barriers to voting in the next election. With a 76% turnout this year, I suspect next year will still yield over a 50% voter turnout, and I suspect turnout will continue to be high.

    Allow for removal of Fellows by a Board vote in response to Code of Conduct violations, removing the need for a vote of the membership

    This one means the most to me. When I joined the board, our Code of Conduct was barely two paragraphs long and said little. We rewrote it and formed the PSF Code of Conduct workgroup. From today forward, we can appreciate that the Python Code of Conduct applies to everyone.

    Overall

    We also gained three new directors, including two returning directors. This election may be the first time we have had an election in which no one running from North America made it on the board. (Possibly Europe, too, but I didn’t dive as deep to verify that.) Either way, this is a noteworthy milestone.

    I’m proud of the Python community for embracing our Code of Conduct and membership changes. A few of these were overdue, but updating the voter affirmation process is an excellent proactive step and a shift for the board.

    I also want to thank Débora Azevedo, the PSF’s vice chair-elect and our outbound director. I was impressed with Débora when we served on the board together, and I thought she brought valuable insights. When she put her name forward to run for vice chair, I was impressed because it’s an intimidating group to put yourself out there, and I thought Débora managed it well.

    Resources

    Tuesday July 16, 2024
  • Django

    ,

    Python

    🐘 Django Migration Operations aka how to rename Models

    Renaming a table in Django seems more complex than it is. Last week, a client asked me how much pain it might be to rename a Django model from Party to Customer. We already used the model’s verbose_name, so it has been referencing the new name for months.

    Renaming the model should be as easy as renaming the model while updating any foreign key and many-to-many field references in other models and then running Django’s make migrations sub-command to see where we are at.

    The main issue with this approach is that Django will attempt to create a new table first, update model references, and then drop the old table.

    Unfortunately, Django will either fail mid-way through this migration and roll the changes back or even worse, it may complete the migration only for you to discover that your new table is empty.

    Deleting data is not what we want to happen.

    As it turns out, Django supports a RenameModel migration option, but it did not prompt me to ask if we wanted to rename Party to Customer.

    I am also more example-driven, and the Django docs don’t have an example of how to use RenameModel. Thankfully, this migration operation is about as straightforward as one can imagine: class RenameModel(old_model_name, new_model_name)

    I re-used the existing migration file that Django created for me. I dropped the CreateModel and DeleteModel operations, added a RenameField operation, and kept the RenameField operations which resulted in the following migration:

    from django.db import migrations
    
    
    class Migration(migrations.Migration):
    
        dependencies = [
            ('resources', '0002_alter_party_in_the_usa'),
        ]
    
        operations = [
            migrations.RenameModel('Party', 'Customer'),
            migrations.RenameField('Customer', 'party_number', 'customer_number'),
            migrations.RenameField('AnotherModel', 'party', 'customer'),
        ]
    

    The story’s moral is that you should always check and verify that your Django migrations will perform as you expect before running them in production. Thankfully, we did, even though glossing over them is easy.

    I also encourage you to dive deep into the areas of the Django docs where there aren’t examples. Many areas of the docs may need examples or even more expanded docs, and they are easy to gloss over or get intimidated by.

    You don’t have to be afraid to create and update your migrations by hand. After all, Django migrations are Python code designed to give you a jumpstart. You can and should modify the code to meet your needs. Migration Operations have a clean API once you dig below the surface and understand what options you have to work with.

    Monday July 15, 2024
  • Python

    🦆 DuckDB may be the tool you didn't know you were missing

    🤔 I haven’t fully figured out DuckDB yet, but it’s worth trying out if you are a Python dev who likes to work on data projects or gets frequently tasked with data import projects.

    DuckDB is a fast database engine that lets you read CSV, Parquet, and JSON files and query them using SQL. Instead of importing data into your database, DuckDB enables you to write SQL and run it against these file types.

    I have a YouTube to frontmatter project that can read a YouTube playlist and write out each video to a markdown file. I modified the export script to save the raw JSON output to disk.

    I used DuckDB to read a bunch of JSON files using the following script:

    import duckdb
    
    def main():
        result = duckdb.sql("SELECT id,snippet FROM read_json('data/*.json')").fetchall()
    
        for row in result:
            id, snippet = row
            print(f"{id=}")
            print(snippet["channelTitle"])
            print(snippet["title"])
            print(snippet["publishedAt"])
            print(snippet["description"])
            print()
    
    
    if __name__ == "__main__":
        main()
    

    This script accomplishes several things:

    • It reads over 650 JSON files in about one second.
    • It uses SQL to query the JSON data directly.
    • It extracts specific fields (id and snippet) from each JSON file.

    Performance and Ease of Use

    The speed at which DuckDB processes these files is remarkable. In traditional setups, reading and parsing this many JSON files could take significantly longer and require more complex code.

    When to Use DuckDB

    DuckDB shines in scenarios where you need to:

    • Quickly analyze data in files without a formal import process.
    • Perform SQL queries on semi-structured data (like JSON)
    • Process large datasets efficiently on a single machine.

    Conclusion

    DuckDB is worth trying out in your data projects. If you have a lot of data and you need help with what to do with it, being able to write SQL against hundreds of files is powerful and flexible.

    Saturday July 13, 2024