When starting a new project, it’s easy to get caught up in the excitement of starting the shiny new thing and forget about some basics. What starts out as a script slowly evolves into an application and before long you’ll find yourself repeating common steps for every project to get it to match personal preferences. If you happen to be working within on a team, you may have many repositories that should have the same setup and inconsistencies between them can quickly lead to large amounts of lost time to get them aligned.

Most of the things I want configured for a new project will always be the same. I use Python and Pipfile as the package manager for most of my projects. I’ve settled on black being my code formatter of choice for python and use pytest to run my tests. I’d want to be using git for version control and the code stored remotely with my preferred choice GitHub. It’s also likely that I’d want CI setup on a project which in my case would mean configuring GitHub actions.

Many of these things are able to be automated away, though are often left forgotten until we’re deep into work. The good news is there’s a way to remove some of this pain.

Templating Projects with Cookiecutter

Cookiecutter is a fantastic library which allows templating of project structure and content. It uses Jinja2 under the hood to allow replacement of variables within files and folder names. Here I use cookiecutter to create a template for my python project, but we could use it for any language or file type we wanted to write a template for.

Cookiecutter has great documentation itself, so here I’ll lead with what my project template looks like:

python-pipenv-github
├── cookiecutter.json
├── {{cookiecutter.project_name}}
│   ├── app
│   │   └── app.py
│   ├── .github
│   │   └── workflows
│   │       └── pull_request.yml
│   ├── Makefile
│   ├── Pipfile
│   ├── .python-version
│   ├── README.md
│   └── tests
│       └── test_something.py
└── .gitignore

Yes, you read that right - the directory name has curly braces in it. My terminal in VSCode has a hard time understanding this if I try and navigate to it.

The other key thing here is the content of cookiecutter.json, which defines the variables which are needed, along with some default values:

{
        "project_name": "someproject",
        "python_version": "3.8.6"
}

As an example, lets look at the Pipfile which uses the version variable:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
pytest = "*"
flake8 = "*"
black = "==20.8b1"

[packages]

[requires]
python_version = "{{cookiecutter.python_version}}"

Cookiecutter is run on a template folder or repo like so:

cookiecutter ../cookiecutter/python-pipenv-github

Any variables defined in the json file will be requested and replaced and used throughout its files and folders. That means our project directory name will be correctly named along with the python version which will be picked up and used in the Pipfile and github actions. You can also call cookiecutter on any repo that holds a cookiecutter template.

As this is a python project template, we can tailor the .gitignore content, so we filter out any rogue files like *.pyc or *.DS_Store.

Our project doesn’t include a Pipfile.lock, so we’ll get the latest version of any of the packages specified in our Pipfile when we execute a pipenv install --dev

Automating Dev Tasks with Makefiles

On a recent piece of work, the team made use of makefiles as an easy way to simplify common takes like formatting code or testing. Having to remember the exact tool or syntax is something that can take time. If we use makefile for each project by we by default only need to remember the targets we define.

Here’s the contents of the Makefile:

init:
        git init
        git add -A
        git commit -m "Initial commit"
        git branch -M main
        gh repo create
        @echo "push with: git push -u origin main"
install:
        pipenv install --dev
lint:
        pipenv run flake8
        pipenv run black --check .
test: lint
        pipenv run py.test
format:
        pipenv run black .

You can see that in my case I’m just calling the relevant pipenv commands, but I can group as many commands together for each make rule which simplifies things quite a bit if I were to have both backend and frontend code in the mix. Knowing that all the common commands are grouped in this way is pretty helpful if you are new to a repo and want to get familiar with the project too.

Additionally, I’ve included a make init target. This not only initialises my local repo with all my code, but uses the github cli to create it remotely ready for me to push back to.

The sum total of commands I need to remember to get started with the project is:

make install

..and then some time later after I’ve worked on actually creating some code:

make init

GitHub Action Workflows with Cookiecutter

The .github/workflows also has a bunch of content which we can template with cookiecutter. In my case, I can call my tests through make (along with their dependency lint tests). Cookiecutter is able to call the relevant actions to install and use the same version of python I’m using locally and execute my test commands when I open a PR:

name: CI
on: pull_request

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: {{cookiecutter.python_version}}
      - name: Install Pipenv
        run: pip install pipenv==2020.11.15
      - name: Install venv
        run: make install
      - name: Test
        run: make test

Conclusion

Automating away this much of my project setup is a massive win. I wish I’d taken the time to set up a template such as this for previous team projects as it would have saved us quite a bit of time.

What’s nice about this basic python template is that it can be used as the basis for creating other templates for more complex apps with additional common dependencies or workflows - a flask/django webapp for instance running within a container with gitlab as a remote host for the repo. It’s also really simple to switchout say Pipenv for poetry and use that instead.

The entire template is up on github and you can use it yourself with cookiecutter by calling:

cookiecutter https://github.com/iwootten/python-pipenv-github