The open analytics template is a very lightweight and reusable automation pipeline for open analytics projects. The tech stack is built using an end-to-end open-source tooling, consisting four key components:
- GitHub API / GitLab API: We use the open API to pull data from open health repositoiries as
.jsonfiles that are flattened into
pandasdataframes for analysis.
- Plotly.py: An open source python graphing library is used to plot the repository data as tables and interactive charts.
- GitHub Actions: Used to orchestrate and automate the first two components on a schedule and commit those changes back to the project’s repository.
- GitHub.io Pages: We host and publish the results of our analysis to a static website that is re-built on every new commit.
The template is used to build and deploy the following projects:
- The Open-Source Health Statistics project was created by developers at NHSX and NHS Business Services Authority
An API (Application Program Interface) allows us to access web tools or data in the cloud. The Github / GitLab API’s are designed so that we can create and manage our repositories, branches, issues, pull requests programmatically. Typically you would need to sign into your own account to access these features, but some information is publicly available. In this project we are using the API to access publicly available information on open source repositories published by NHS and health related organisations.
We use the
urllib.request python library to access the API as follows:
Note: you can only make 60 calls per hour to the publich GitHub API, so we need to bear this in mind when looping through the API calls.
The outputs of the API call returns a
.json file from which we can flatten to a
We can then do some basic calculations to summerise these data. For example, count the number of repositores by each organisation.
We use Plotly to save the graph to standalone HTML files.
Example Ploty Graph
GitHub Actions are a way to automate workflows using a simple YAML syntax. It’s free on public repositories.
You must store workflow files in the
.github/workflows directory of your repository.
The name of your workflow. GitHub displays the names of your workflows on your repository’s actions page.
Required. The name of the GitHub event that triggers the workflow. For a list of available events, see Events that trigger workflows.
You can schedule a workflow to run at specific UTC times using POSIX cron syntax. Scheduled workflows run on the latest commit on the default or base branch.
This example triggers the workflow every day at 00:00 UTC:
A workflow run is made up of one or more jobs. Each job runs in a fresh instance of a virtual environment specified by
|Virtual environment||YAML workflow label|
|Windows Server 2019||
|Windows Server 2016||
|macOS Big Sur 11||
|macOS Catalina 10.15||
Checkout: This action checks-out your repository so the workflow can access it.
Setup python: This action sets up a Python environment for use in actions by installing and adding to PATH an available version of Python in this case python 3.8
Install dependancies: This GitHub Action installs Python package dependencies from a user-defined
requirements.txt file path with
In this case plotly, pandas, and pyYaml
Runs command-line programs using the operating system’s shell. run the run.py to get the latest data
Commit changes to files
Push changes to repo so github pages will re-build website
You can use a static site generator to build your site for you or publish any static files that you push to your repository as follows:
- On GitHub, navigate to your site’s repository, example: https://nhs-pycom.github.io/opensource-health-statistics/.
- In the root of the repository, create a new file called
index.mdthat contains the content for your site.
- Under your repository name, select
- In the left sidebar, select
- Select the branch from which to publish your page and select
- Your page will be deployed within 60 seconds
- To see your published site, under
GitHub Pages, select your site’s URL.