Automating PyPI Releases Using Poetry and Github Actions

In creating my first package on PyPI, I realised that many of the tasks I’d want to do on a subsequent publish would be repetitive. If I wanted to push out a simple fix, I wanted to automate away those tasks so I could focus on my code rather than a sequence of repetitive steps.

Github actions is a simple way to do this and has given me the ability to quickly push out changes without lots of manual work. I’ve used them a number of times for other clients so this is the first piece of work for myself I’ve been able to use them for.

You can find all the final code for the workflows in my repository here.

Workflows

The actions I want to use are defined through use of a workflow file written in yaml and stored in .github/workflows directory. These workflows are triggered by particular github events such as a push to a branch, merge to master or even a regular cron type event that runs on a schedule. In my case, I wanted to have tests run when a pull request was opened and when tag was created I wanted to publish to test.pypi.org.

Only if when I inspected the publish to test.pypi.org and was entirely happy would I want for it to be made publicly available, so I only published there on creation of a release from my tag. Doing things this way means that we publish to test on tags, but can group a collection of tags together for an official release. This means setting up three separate workflow files.

One feature I’d like to see at a later date is yaml partial support. Many workflow files can end up with duplication between them and it would be nice from a maintenance point of view to only have to update a single partial and have that updated in multiple workflows. You could acheivev this through a local build step but that’s not ideal.

If you read my earlier post, you can see I’m using poetry as my packaging tool for this project.

Below is my yaml file which is triggered on each PR to the repository:

name: CI
on: pull_request

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
      - name: Install Poetry
        run: curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
      - name: Add Poetry to path
        run: echo "${HOME}/.poetry/bin" >> $GITHUB_PATH
      - name: Install venv
        run: poetry install
      - name: Test
        run: poetry run pytest

As you can see this workflow is triggered by the pull request event. That means it will run each time theres an update to a PR. I’ve given this workflow the name CI.

Actions

The uses steps define which actions to use for this workflow. They refer to public repositories on github so you can visit each one of them to see what code will be executed when the step runs. The @ version indicates the version of the action that will be run and can be found by looking at the repository tags.

The first thing to do is use the checkout action to actually checkout the updated version of our code, ready to be used in the workflow. This isn’t something that happens by default so if you don’t make use of the action, you won’t be able to work with your code within the workflow.

The setup python action installs python ready for the workflow to make use of. In my case I just use the latest version of python that is installed with this plugin, but I could specify a particular version to be used if I wanted.

Custom Steps

The subsequent steps in the workflow install poetry, add it to the path and then install the project dependencies to a virtualenv. Once all this is done, we can finally run pytest with poetry to test that our project is working as expected.

Publishing Pre-Releases

The workflow file for when tags are created is much the same as the pull request one except for a couple of differences. Firstly, the event trigger is a when a tag is created:

name: Publish to Test PyPI
on:
  push:
    tags:
      - '*.*.*'

Additionally, I need to use a variable TEST_PYPI_API_KEY which is stored as a secret in the repository settings. You can see where to find this in the screenshot below:

Note: I could have stored these variables at an organisation level (and probably will update to do so), so they can be made available to any other projects I needed to publish to PyPI.

Poetry uses this variable to configure for test.pypi.org so that it can publish our build like so:

- run: poetry config repositories.testpypi https://test.pypi.org/legacy/
- run: poetry config pypi-token.testpypi ${{ secrets.TEST_PYPI_API_KEY }}
- name: Publish package
  run: poetry publish --build -r testpypi

Publishing Releases

In order to publish to the final build, I use all the same orignal steps as the pre release workflow, but respond to a release event from any tag that is created:

name: Publish to PyPI
on:
  release:
    types: [published]

The steps needed for configuring the official build publish are also much shorter and use the official PyPI API key rather than the test one.

- run: poetry config pypi-token.pypi "${{ secrets.PYPI_API_KEY }}"
- name: Publish package
  run: poetry publish --build

You can see all the final workflow files within the repository. Here’s an example of a run after a release has been created:

Now, whenever I want to package my library after adding a feature, I need not worry about remembering the exact set of commands to do so and let github do the work for me.