Setup CI for django unit testing with github actions

Github actions allows to automate all your software workflows and build customizable CI/CD pipeline. This blog assumes you have some knowledge of Django but are new to github actions workflows.

Setup django's database settings to be read from environment

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': os.environ.get('MYSQL_DATABASE'),
        'USER': os.environ.get('MYSQL_USER'),
        'PASSWORD': os.environ.get('MYSQL_PASSWORD'),
        'HOST': os.environ.get('MYSQL_HOST'),
        'PORT':  os.environ.get('MYSQL_PORT'),
    }
}

Setup github workflow

1. Download the template from github marketplace

We are using a workflow template provided by github and modifying it to meet our needs.

Note: Github actions files are stored in your .github/workflows/<file-name>.yml and are to be committed.

# This workflow will install Python dependencies, run tests and lint with a single version of Python
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: Python application

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Set up Python 3.9
      uses: actions/setup-python@v2
      with:
        python-version: 3.9
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install flake8 pytest
        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
    - name: Lint with flake8
      run: |
        # stop the build if there are Python syntax errors or undefined names
        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
    - name: Test with pytest
      run: |
        pytest

Let first try to understand what is happening in the given template

name: Python application # name of the work flow

# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#on
on:
  push:                  # whenever there is push event
    branches: [ master ] # on master branch
  pull_request:          # whenever there is pull request created
    branches: [ master ] # with target branch as master

# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobs
jobs:      # jobs is reserved namespace
  build:   # a unique identifier for your job. This can be anything and translates to jobs.<job_id>

# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idruns-on
runs-on: ubuntu-latest    # The type of machine to run the job on

# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idsteps
steps:   # sequence of tasks called steps

You can define sequential steps to be run in the steps key

# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsuses
- uses: actions/checkout@v2  # official github provided action. Checks out code for processing

# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepswith
- name: Set up Python 3.9        # name for easy identification
  uses: actions/setup-python@v2  # official github provided action. Installs Python on your runs-on machine
  with:                          # with provides an ability to pass arguments to action
    python-version: 3.9          # python-version to be installed

Other steps in template are intuitive to understand.

2. Edit the template to setup `mysql` service

Define the services key under jobs.<job_id> as following

...

jobs:
  build:
    runs-on: ubuntu-latest

    env:
      MYSQL_PASSWORD: password

    services:
      mysql:
        image: mysql:8.0
        env:
          MYSQL_ROOT_PASSWORD: ${{ env.MYSQL_PASSWORD }}
        ports:
          - 3306:3306
...

If you have worked with docker the syntax would be very intuitive to understand. We are basically installing mysql:8.0 image and passing some required variable MYSQL_ROOT_PASSWORD for setup. The important parts here are the ports & options.

Port 3306 is exposed from the container service and it mapped to 3306 port on the host machine (github machine).

3. Setup health check for our mysql service

Containers takes sometime to get mysql ready for accepting connections. Therefore it is important that we wait for mysql service to become healthy before proceeding with steps

...

jobs:
  build:
    runs-on: ubuntu-latest

    env:
      MYSQL_PASSWORD: password

    services:
      mysql:
        image: mysql:8.0
        env:
          MYSQL_ROOT_PASSWORD: ${{ env.MYSQL_PASSWORD }}
        ports:
          - 3306:3306
      options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3
...

We are passing health-cmd in options to change the container healthy check command.
If mysqladmin ping does not respond within health-timeout we wait health-interval seconds to try again unless we have tried health-retries times. If mysqladmin ping responds without error. We establish that mysql inside the container is working perfectly and is ready to accept connections.

Lets just add more environment variables for mysql service creation, this will allow us to have same mysql user, port, host & database for django's database settings

...

runs-on: ubuntu-latest
    env:
      MYSQL_PASSWORD: password
      MYSQL_HOST: 127.0.0.1
      MYSQL_PORT: 3306
      MYSQL_DATABASE: test_database
      MYSQL_USER: mysql_user
      ENVIRONMENT: CI

    services:
      mysql:
        image: mysql:8.0
        env:
          MYSQL_ROOT_PASSWORD: ${{ env.MYSQL_PASSWORD }}
          MYSQL_PASSWORD: ${{ env.MYSQL_PASSWORD }}
          MYSQL_HOST: ${{ env.MYSQL_HOST }}
          MYSQL_PORT: ${{ env.MYSQL_PORT }}
          MYSQL_DATABASE: ${{ env.MYSQL_DATABASE }}
          MYSQL_USER: ${{ env.MYSQL_USER }}
        ports:
          - 3306:3306
        options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3

...

4. Setup permissions for mysql user

To avoid any permission errors like:

Got an error creating the test database: permission denied to create database

django.db.utils.OperationalError: (1045, “Access denied for user 'root'@'localhost'

The magic command to fix those error in our test environment is

GRANT ALL PRIVILEGES ON *.* TO '${{ env.MYSQL_USER }}'@'%'

Our complete file at .github/workflows/python-app.yml looks like this. Note the highlighted area in the below code.We want to allow our mysql user to have all rights to database.

# .github/workflows/python-app.yml

name: Python application

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest
    env:
      MYSQL_PASSWORD: password
      MYSQL_HOST: 127.0.0.1
      MYSQL_PORT: 3306
      MYSQL_DATABASE: test_database
      MYSQL_USER: mysql_user
      ENVIRONMENT: CI

    services:
      mysql:
        image: mysql:8.0
        env:
          MYSQL_HOST: ${{ env.MYSQL_HOST }}
          MYSQL_PORT: ${{ env.MYSQL_PORT }}
          MYSQL_DATABASE: ${{ env.MYSQL_DATABASE }}
          MYSQL_USER: ${{ env.MYSQL_USER }}
          MYSQL_PASSWORD: ${{ env.MYSQL_PASSWORD }}
          MYSQL_ROOT_PASSWORD: ${{ env.MYSQL_PASSWORD }}
        ports:
          - 3306:3306
        options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3

    steps:

    - name: Grant permission to mySQL and Flush privileges
      run: |
        mysql --host ${{ env.MYSQL_HOST }} --port ${{ env.MYSQL_PORT }} -uroot -p${{ env.MYSQL_PASSWORD }} -e "GRANT ALL PRIVILEGES ON *.* TO '${{ env.MYSQL_USER }}'@'%';FLUSH PRIVILEGES;"


    - uses: actions/checkout@v2

    - name: Set up Python 3.6
      uses: actions/setup-python@v2
      with:
        python-version: 3.6

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi

    - name: Run migrations
      run: python manage.py migrate

    - name: Test with django
      run: |
        python manage.py test

5. Commit and create PR

Open a PR to master with file .github/workflows/python-app.yml committed and you will see github action workflow picks it up and execute all the steps.

Contents: