Contributing

🐛 Issues and bugs

The easiest way to contribute is to report issues or bugs that you might find while using feasibility-data. You can do this by creating a new issue on our GitHub repository.

✏️ Adding or modifying content

If you would like to contribute content, please check out our guidebook for more specific details on how we work and develop. It is a regularly evolving document, so is at various states of completion.

To contribute to feasibility-data, you first need to install uv and justfile. We use uv and justfile to manage our project, such as to run checks on the data package and build the website. Both the uv and justfile websites have a more detailed guide on using uv, but below are some simple instructions to get you started.

It’s easiest to first install uv and then install justfile with uv. Once you’ve installed uv, install justfile by running:

uv tool install rust-just

We keep all our development workflows in the justfile, so you can explore it to see what commands are available. To see a list of commands available, run:

just

As you contribute, make sure your changes will pass our tests by opening a terminal so that the working directory is the root of this project (feasibility-data/) and running:

just run-all

When committing changes, please try to follow Conventional Commits as Git messages. Using this convention allows us to be able to automatically create a release based on the commit message by using Cocogitto. If you don’t use Conventional Commits when making a commit, we will revise the pull request title to follow that format. That’s because we use squash merges when merging pull requests, so all other commits in the pull request will be squashed into one commit.

📁 Explanation of files and folders

This is a brief description of some of the files in this repository.

  • .copier-answers.yml: Contains the answers you gave when copying the project from the template. You should not modify this file directly.
  • .github/: Contains GitHub-specific files, such as issue and pull request templates, workflows, dependabot configuration, pull request templates, and a CODEOWNERS file.
  • _quarto.yml: Quarto configuration file for the website, including settings for the website, such as the theme, navigation, and other options.
  • _metadata.yml: Quarto metadata file for the website, including information about the project, such as the titles and GitHub names.
  • .gitignore: This ignore file tells Git which files to not track. Unless you know what you are doing, it’s best to not touch this file.
  • .pre-commit-config.yaml: Pre-commit configuration file for managing and running checks before each commit.
  • .config/: Contains configuration files for various tools used in the project, such as:
    • typos.toml: typos spell checker configuration file.
    • rumdl.toml: rumdl configuration file for formatting Markdown files in the project.
    • cog.toml: Cocogitto configuration file for managing versions.
    • cliff.toml: git-cliff configuration file for creating the changelog.
    • ruff.toml: Ruff configuration file for linting and formatting Python code.
  • .editorconfig: Editor configuration file for EditorConfig to maintain consistent coding styles across different editors and IDEs.
  • CITATION.cff: Structured citation metadata for your project when archived on Zenodo and used by GitHub to display the citation information on the repository page. This is used to add the metadata to Zenodo when a GitHub release has been uploaded to Zenodo.
  • justfile: just configuration file for scripting project tasks.
  • CHANGELOG.md: Changelog file for tracking changes in the project.

Flow of data

REDCap

The data flows directly from the REDCap API into raw/redcap/ as a CSV file with a timestamp appended to the filename. Every time the data is pulled from REDCap, a new CSV file is created in raw/redcap/ with the current timestamp.

Using code written in src/feasibility_data/data/redcap/<resource>.py, each raw CSV file is processed into a collection of staging/redcap/<resource>/<timestamp>.parquet files. There should be a 1-to-1 mapping between the raw CSV’s timestamp and the staged resource Parquet file’s timestamp.

If metadata drifts over time, errors will happen when processing the older raw CSV files using the newer metadata. This is expected and desirable behaviour as it:

  • Informs us that we need to update or resolve the older data to match the newer metadata.
  • Helps ensure transparency and a record of how the data has changed over time and how we’ve fixed it.
  • Ensures that all files in staging/ are aligned, as Sprout takes all files in staging/ and converts them into a single resource. So they must always match together.
  • Matches the behaviour of our pipelines from other sources. While REDCap stores data for up to 5 years, other sources of data for ON LiMiT have much shorter retention periods. So previously pulled raw data in this repository may be the only copy of that data available to us. Which means we need to us all raw data when processing into staging/ and eventually into resources/.

There are specific things to note about the REDCap data:

  • Fields ending in _id are primary/foreign keys.
  • Fields that contain admin are excluded from the data package.

When processing the data, each resource should (almost always) contain a participant_id and a visit_id field.

Layout of src/

Similar to how raw/ and staging/ are organized, the Python files within src/ are organized at the top level by data and metadata, then by source of the original data, and finally by the eventual resource name. The structure under src/feasibility_data/ is:

  • metadata/<source>/<resource>.py: Python files within this directory contain functions that are used to convert the raw dictionaries into the final datapackage.json metadata file. Functions within these modules can be named without needing to state the source or resource (as the module path already contains that information). For example, metadata/redcap/vas.py would contain the functions for processing the metadata for the VAS resource from the REDCap source.
  • data/<source>/<resource>.py: Same with the metadata files, but these contain functions for taking the original raw data and converting them into the staging/ folder. Unlike the metadata above, raw data goes into staging/ first before being processed into the final data resource as Sprout needs to run checks against the metadata before converting it into the final data resource.
  • In either the data/ or metadata/ directories, files named */core.py contain functions that do general processing tasks related to the parent folder name. For example, metadata/core.py contains functions for top-level metadata processing that is for general metadata, but not strictly tied to any given source or resource, such as data package-level metadata. Meanwhile, data/redcap/core.py contains functions for processing REDCap data that is not specific to any resource. This core.py file can be treated like the __init__.py file. We don’t use __init__.py files to store functions as the semantic meaning of __init__.py is to initialise the folder as part of the package. The semantic meaning of core.py is to be a collection of functions that are used in its parent source/resource folder.
  • common/: Contains functions that are used across all (or many) Python files, between metadata and data or between sources/resources. This is not the same as the **/core.py files that are specific to the particular source or resource. The names of the Python files within are not standardized, but they should be descriptive of the overall functionality they provide within. An advantage of keeping common functions in one location is that it makes it easier for us to identify if any of these functions belong in their own package.
  • build.py: This file lists all the functions (as pytask tasks) that are needed to take the raw data and raw dictionaries and turn it all into a final data package. We keep all tasks in this file to make it easier to track, review, and update the full build process in one location.

Similar to a Python package, all Python files must only contain functions and/or classes and not be called directly. Functions are kept small and focused, with a narrow scope and clear input and output (with type hints, ideally using custom types). The only exception is the build.py file that has the pytask tasks. This file is used to build up all the smaller functions into specific tasks. These tasks have input/output that matches the style of pytask and can be larger and more complex than the non-build functions.

Writing Python code

  • Each “public” function should be at the top of the module file, with “private” (prefixed with _) functions below them.
  • Classes, either public or private, go at the top of the file.