Useful Github Actions Bots to Regulate my Python Package

Any automation engineer probably has half a dozen builds or pipelines that are maybe kind of working or you force override the authorization and merge builds in. With large scale software infrastructure it can get pretty crazy in terms of management. So let’s have a look at one of my packages, Global-Chem:

Lately, I’ve been the sole major developer on my team (a couple of undergraduates and other researchers for the science and teaching) but for the most part pretty alone on it. Which is why I know I needed bots to help me regulate the code as it scales. Continous Integration (CI) and Continous Deployment (CD) can be tricky to implement especially knowing which tool to choose. For about a decade, I have tried CircleCI, TravisCI, ConcouseCI, JENKINS with Katalon Studio/Selenium for front-end, Cypress for Javascript Testing Frameworks but recently Github Actions have been picking up and I want everything to be self-contained from my repository to a CI system so I figured this was a shot.

In that time, I have accumulated numerous pythonic workflows and figuring out what I need and want so here’s the list of what I am using and how is it implemented:

1. Github Actions with PyTest/Nose and Coveralls

name: GlobalChem API
on: [pull_request, push, workflow_dispatch]
jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: true
      matrix:
        os: ["ubuntu-latest", "windows-latest", "macos-latest"]
        python-version: ["3.7", "3.8", "3.9"]
steps:
      - name: Checkout source
        uses: actions/checkout@v2
- name: Setup python
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
          architecture: x64
- name: Install
        run: |
          python global_chem/setup.py install
          pip install python-coveralls
          pip install coveralls
          pip install coverage==4.5.4     
          pip install nose              
      - name: Run GlobalChem tests
        env:
          GITHUB_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
        run: |
          cd global_chem
          python -m nose --verbose --with-coverage -s -w tests/

I use this one religiously and it’s saved me a lot in terms of continuously testing my code and it’s deployment. One of the major bugs it was able to find because of the windows, mac os, and linux parallel deployment was a character encoding which when the downloads shot down I realized it was because of this particular test. And their full results can be found below

SS-42: Lost Support for the Windows Users · Issue #86 · Sulstice/global-chem

This action does really depend on how well you write your tests but it has come as a saviour for me down the road for testing especially for interoperability.

2. Pre-Commit

For large scale code management we use pre-commit with flake8 to perform static code analysis to make sure there is a regulation for the code quality being contributed in. This bot kind of works and it depends on how strict the user working the bot is. Here is an example of it failing on a pull request that isn’t formatted correctly:

name: pre-commit
on:
  pull_request:
  push:
    branches: [main]
jobs:
  pre-commit:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v2
    - uses: pre-commit/action@v2.0.3
      with:
        extra_args: flake8 --all-files
    - uses: mszostok/codeowners-validator@v0.7.1
      with:
        checks: "files,owners,duppatterns"
        experimental_checks: "notowned"
        github_access_token: "${{ secrets.OWNERS_VALIDATOR_GITHUB_SECRET }}"

I have found as I have more contributors to my code, I have to regulate code quality more stringently. To do that I actually need to install some rules on where flake8 should be expanded too, this also only is valid for python and more of my code base is starting to switch to a web server using Github as a database so other static analysis methods will be needed and I will be expanding this aspect. I still do code quality manually and that won’t scale.

3. Publish Package

When I was younger and writing my first python package, this task took the longest because setting up distribution your first time is tricky. I actually hope this github action bot also teaches others how to distribute their packages automatically. As we schedule releases for publication and official versions it’s tied to our Github on a release that is created and automatically deploys it to PyPi. One caveat, is that the setup.py file needs to be updated appropriately.

name: publish
on:
  release:
    types:
      - created
  workflow_dispatch:
jobs:
  publish:
runs-on: ubuntu-latest
steps:
    - uses: actions/checkout@v2
    - name: Set up Python "3.6"
      uses: actions/setup-python@v2
      with:
        python-version: "3.6"
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        if [ -f global_chem/requirements.txt ]; then pip install -r global_chem/requirements.txt; fi
        python global_chem/setup.py install
    - name: Build a binary wheel and a source tarball
      run: |
        pip install build
        cd global_chem
        python -m build --sdist --wheel --outdir dist/ .
    - name: Publish distribution 📦 to PyPI
      uses: pypa/gh-action-pypi-publish@master
      with:
        password: ${{ secrets.PYPI_API_TOKEN }}

I have rarely used this bot, but that’s because I have own one-liner mechanism on my local machine. I think moving forward with more developers joining the project this will help the scalability.

4. Translate README

I wanted to make my software and readme documentation available in all types of languages and not just english, my native tongue. So, I actually implemented this bot:

Now is it perfect? No. According to my friends/collaborators who have been translating my documentation for me said the grammar is weird but it does actually give them a starting point to work on. It also helped get the point across. I actually think this bot is more marketing material than useful but I did see a rise in more activity from china with the implementations of Chinese Simplified and Chinese Traditional:

name: Translate README
on:
  release:
    types:
      - created
  workflow_dispatch:
  
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Setup Node.js
        uses: actions/setup-node@v1
        with:
          node-version: 12.x
      # ISO Langusge Codes: https://cloud.google.com/translate/docs/languages  
      - name: Adding README - Chinese Simplified
        uses: dephraiim/translate-readme@main
        with:
          LANG: zh-CN
      - name: Adding README - Chinese Traditional
        uses: dephraiim/translate-readme@main
        with:
          LANG: zh-TW
      - name: Adding README - Hindi
        uses: dephraiim/translate-readme@main
        with:
          LANG: hi
      - name: Adding README - Arabic
        uses: dephraiim/translate-readme@main
        with:
          LANG: ar
      - name: Adding README - French
        uses: dephraiim/translate-readme@main
        with:
          LANG: fr

5. Pages Build and Deployment

This is my favourite bot, I have hackneyed this one like crazy where basically all I have to do is give an index.html file and github automatically deploys the pages to a static site tied to the repository. I don’t even write code.

Automatic deployment for Github Pages has been around on Github for awhile but recently the feature just matured. Because of this I have made tons of different random websites tied to a repository to host different visualizations, example:

https://sulstice.github.io/Faith/neural_net_layers/atom_type_layer/index.html