diff --git a/.github/workbench-docker-version.txt b/.github/workbench-docker-version.txt new file mode 100644 index 0000000000..8f51ea947e --- /dev/null +++ b/.github/workbench-docker-version.txt @@ -0,0 +1 @@ +v0.2.9 diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 7076ddd9f6..2017fc4630 100755 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -1,198 +1,313 @@ -# Carpentries Workflows +# Workbench Workflows Documentation -This directory contains workflows to be used for Lessons using the {sandpaper} -lesson infrastructure. Two of these workflows require R (`sandpaper-main.yaml` -and `pr-receive.yaml`) and the rest are bots to handle pull request management. +These GitHub Actions workflows manage and maintain the build processes for lessons using the Carpentries Workbench on GitHub. -These workflows will likely change as {sandpaper} evolves, so it is important to -keep them up-to-date. To do this in your lesson you can do the following in your -R console: + +## Managing Workflow Updates + +By using prebuilt Docker containers that are managed by the Carpentries core Workbench maintainers, these workflows are designed to be rarely updated. + +However, is important to be able to keep them up-to-date when appropriate. +You can do this locally using your own R and Workbench installation, or via the "04 Maintain: Update Workflow Files" (`update-workflows.yaml`) GitHub Action. + +### Updating locally + +In a terminal/git bash, navigate to the lesson folder where you want to update the workflows, e.g.: + +```bash +cd ~/lessons/shell-novice +``` + +Then, start an R session and: ```r # Install/Update sandpaper -options(repos = c(carpentries = "https://carpentries.r-universe.dev/", - CRAN = "https://cloud.r-project.org")) +options(repos = c(carpentries = "https://carpentries.r-universe.dev/", CRAN = "https://cloud.r-project.org")) install.packages("sandpaper") # update the workflows in your lesson library("sandpaper") -update_github_workflows() +sandpaper::update_github_workflows() +quit() +``` + +And then in a bash prompt/git bash terminal: + +```bash +$ git add .github/workflows +$ git commit -m "Manual update to docker workflows" +$ git push origin main +``` + +> [!NOTE] +> For non-renv lessons, this is all the setup you need! +> +> For renv-enabled lessons: +> - Cancel any "01 Maintain: Build and Deploy Site" workflow currently running +> - Run the "02 Maintain: Check for Updated Packages" workflow and merge any PR opened to update the renv lockfile +> - This should automatically run the "03 Maintain: Apply Package Cache" workflow to install packages and build the cache +> - A successful cache build should then trigger the "01 Maintain: Build and Deploy Site" workflow + +#### Updating to a specific workflow release or branch + +To test new Workbench workflow features or branches, or to pin to a specific release, the `branch` option can be supplied to `update_github_workflows()`. + +For a given branch: + +```r +sandpaper::update_github_workflows(branch="develop") +``` + +Or for a given release version: +```r +sandpaper::update_github_workflows(branch="1.0.1") ``` -Inside this folder, you will find a file called `sandpaper-version.txt`, which -will contain a version number for sandpaper. This will be used in the future to -alert you if a workflow update is needed. +### Updating using GitHub -What follows are the descriptions of the workflow files: +#### Official lessons -## Deployment +"Official" lessons are those in the lesson program repositories, Incubator, or Lab. +They need no extra setup as this is all managed for you as part of the Carpentries GitHub organisations. -### 01 Build and Deploy (sandpaper-main.yaml) +To update the workflows, either: +- wait for the scheduled run of the "04 Maintain: Update Workflow Files" at approximately midnight every Tuesday +- go to the Actions tab on GitHub, click "04 Maintain: Update Workflow Files" on the left, then "Run Workflow" on the right -This is the main driver that will only act on the main branch of the repository. -This workflow does the following: +Once complete, this will raise a PR with any changes to the workflows that are needed. +If you are happy with the changes made, you can merge the PR into your lesson repository. - 1. checks out the lesson - 2. provisions the following resources - - R - - pandoc - - lesson infrastructure (stored in a cache) - - lesson dependencies if needed (stored in a cache) - 3. builds the lesson via `sandpaper:::ci_deploy()` +#### Your own lessons -#### Caching +This presumes you: + - already have a lesson repository available on GitHub + - have enabled workflows in the lesson repo + - have set up a SANDPAPER_WORKFLOW personal access token (PAT) in the lesson repo -This workflow has two caches; one cache is for the lesson infrastructure and -the other is for the the lesson dependencies if the lesson contains rendered -content. These caches are invalidated by new versions of the infrastructure and -the `renv.lock` file, respectively. If there is a problem with the cache, -manual invaliation is necessary. You will need maintain access to the repository -and you can either go to the actions tab and [click on the caches button to find -and invalidate the failing cache](https://github.blog/changelog/2022-10-20-manage-caches-in-your-actions-workflows-from-web-interface/) -or by setting the `CACHE_VERSION` secret to the current date (which will -invalidate all of the caches). +To go through these steps, please follow the [Forking a Workbench Lesson](https://docs.carpentries.org/resources/curriculum/lesson-forks.html#forking-a-workbench-lesson-repository) +documentation. -## Updates +Once set up, run the "04 Maintain: Update Workflow Files" (`update-workflows.yaml`) action. -### Setup Information +This will raise a PR with any changes to the workflows that are needed. +If you are happy with the changes made, you can merge the PR into your lesson repository. -These workflows run on a schedule and at the maintainer's request. Because they -create pull requests that update workflows/require the downstream actions to run, -they need a special repository/organization secret token called -`SANDPAPER_WORKFLOW` and it must have the `public_repo` and `workflow` scope. -This can be an individual user token, OR it can be a trusted bot account. If you -have a repository in one of the official Carpentries accounts, then you do not -need to worry about this token being present because the Carpentries Core Team -will take care of supplying this token. +## Dependency Images for RMarkdown Lessons -If you want to use your personal account: you can go to - -to create a token. Once you have created your token, you should copy it to your -clipboard and then go to your repository's settings > secrets > actions and -create or edit the `SANDPAPER_WORKFLOW` secret, pasting in the generated token. +Lessons that use R packages and RMarkdown are built on GitHub using Docker images within these workflows. +These images are based on the [workbench-docker](https://github.com/carpentries/workbench-docker) image. +This base image only contains the Workbench packages to build lessons and not any extra R packages your lesson might need. -If you do not specify your token correctly, the runs will not fail and they will -give you instructions to provide the token for your repository. +Therefore, there are two steps to perform to resolve the required R packages in a lesson, and then generate a Docker image layer comprising those packages. -### 02 Maintain: Update Workflow Files (update-workflow.yaml) +The "02 Maintain: Check for Updated Packages" workflow checks for any used R packages and their versions, and if successful, opens a Pull Request to update your lesson's renv lockfile (`renv.lock`). +Once the PR is merged, the "03 Maintain: Apply Package Cache" workflow builds the dependency image layer with `docker build` and publishes it in your user or organisation account Packages area. -The {sandpaper} repository was designed to do as much as possible to separate -the tools from the content. For local builds, this is absolutely true, but -there is a minor issue when it comes to workflow files: they must live inside -the repository. +> [!NOTE] +> Caching is only relevant for lessons that use Rmd files and renv to manage R packages. +> If you are building basic markdown documents, caching will not apply to you, and the only +> workflow that needs to be run is "01 Maintain: Build and Deploy Site". -This workflow ensures that the workflow files are up-to-date. The way it work is -to download the update-workflows.sh script from GitHub and run it. The script -will do the following: +### Publishing dependency images -1. check the recorded version of sandpaper against the current version on github -2. update the files if there is a difference in versions +The two dependency workflows are separated to ensure that once you have a successful build with a working renv environment, the resulting dependency image is stored and will be reused by the Workbench Docker container. +This means that lesson builds will be faster and more consistent once a dependency image is created and reused. This is doubly important if you need very specific versions of R packages, i.e. "pinning". -After the files are updated, if there are any changes, they are pushed to a -branch called `update/workflows` and a pull request is created. Maintainers are -encouraged to review the changes and accept the pull request if the outputs -are okay. +This setup means that this dependency image layer is available indefinitely: +- to build your lesson, improving robustness and reproducibility +- for others to use as a complete environment in their local builds, teaching environments, or Codespaces -This update is run weekly or on demand. +If and when you want to perform an update to the dependency image, you can re-run the "02 Maintain: Check for Updated Packages" workflow and verify that your lesson still builds with any new packages or changes. +The "03 Maintain: Apply Package Cache" workflow will publish a new repository- or organisation-local GHCR image for the lesson. -### 03 Maintain: Update Package Cache (update-cache.yaml) +As such, the dependency image is invalidated by new versions of the `renv.lock` file. +This happens: + - if you update your lockfile locally by using the `sandpaper::update_cache()` and `sandpaper::manage_deps()` functions, and then push it to the lesson repository + - when you run the "02 Maintain: Check for Updated Packages" and there are new packages to install -For lessons that have generated content, we use {renv} to ensure that the output -is stable. This is controlled by a single lockfile which documents the packages -needed for the lesson and the version numbers. This workflow is skipped in -lessons that do not have generated content. +More information on managing local renv-based lesson dependencies can be found in the [Sandpaper packages vignettes](https://carpentries.github.io/sandpaper/articles/building-with-renv.html). -Because the lessons need to remain current with the package ecosystem, it's a -good idea to make sure these packages can be updated periodically. The -update cache workflow will do this by checking for updates, applying them in a -branch called `updates/packages` and creating a pull request with _only the -lockfile changed_. +#### Using different dependency image versions + +There are times when you may want to go back to a previous dependency image: + - if you run "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" and the cache generation fails for some reason + - if there is a new R package that produces incorrect or broken lesson output + - a new R version is released and downstream packages fail to build and install, and need updating by the package maintainer(s) + +Dependency images will have the following name format, where IMAGE is the workbench-docker image version, and HASHSUM is the `renv.lock` lockfile hash: + +``` +IMAGE HASHSUM +[ | ] [ | ] +v0.2.4_renv-2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4 +``` + +Copy the hashsum part of the desired image tag you want to use, e.g. `2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4`. + +Then either: + 1. Add a repository variable called CACHE_VERSION, and paste in the hash + 2. Run the "01 Maintain: Build and Deploy Site" manually, supplying the `Optional renv cache version override` input + +If you have no images listed, make sure to run the "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" workflows to publish a new dependency image. + +> [!NOTE] +> If you are maintaining an official lesson, dependency images are saved to the Carpentries lesson program organisation, e.g. `datacarpentry`, GHCR package namespace. +> Once a successful dependency image has been published, the build workflows will use it automatically when the matching Docker version tag exists, e.g. 'latest' or a specific version. +> +> If you are developing a lesson in your own repository, the dependency images are stored in your repository's GHCR package namespace. +> You can see available images by going to your user or organisation's Packages tab, e.g. `https://github.com/?tab=packages`. + +#### Pruning dependency image layers + +Through the lifetime of a lesson, an increasing number of dependency images will be generated. +As the images are public, GitHub currently charges no storage, or network ingress or egress fees to upload or download them. + +The "03 Maintain: Apply Package Cache" workflow will automatically check the number of published dependency images and keep each version published plus one underlying untagged digest (if any) for each version. +This means it is possible to roll back easily to a previous version or digest, but not end up with a huge number of dependency images to manage. + +In any case, you can manually delete any versions you wish by: +- going to your user or organisation's Packages tab, e.g. `https://github.com/?tab=packages`. +- selecting the dependency image you want to manage, e.g. `-deps` +- selecting `Package settings` on the right hand side +- deleting any images + +> [!NOTE] +> Deleting these images manually may break your builds. +> Only perform this action if you are sure you want to delete the generated dependency images. +> +> If you mistakenly delete an image, you can rebuild it by rerunning the "03 Maintain: Apply Package Cache" workflow. + + +## User Settings + +Input level variables are documented in the `carpentries/actions` repository READMEs for each composite action. + +Specific repository level variables can be set that will force particular options across all workflow runs. + +### 01 Maintain: Build and Deploy Site (docker_build_deploy.yaml) + +Repository-level variables for this workflow are: +- WORKBENCH_TAG + - The workbench-docker release version to use for a given build + - This can be set to a specific version number to force all builds to use a given container version + - Default is unset or `latest` +- BUILD_RESET + - Force a reset of previously build markdown files + - Setting this variable value to `true` will force sandpaper to delete any previously build markdown files + - Default is unset or `false` +- OMIT_PR_CONTAINER_VERSION_UPDATE + - Control raise behaviour of the workbench-docker version update PR + - When a new workbench Docker image version is detected, usually after a sandpaper, varnish, or pegboard update, its version number will be incremented + - If a newer version is available, a PR will be raised that updates the `.github/workbench-docker-version.txt` file + - To not raise this PR set this to `true` + - Default is unset or `false` +- AUTO_MERGE_WORKBENCH_VERSION_UPDATE + - Control merge behaviour of the workbench-docker version update PR + - If OMIT_PR_CONTAINER_VERSION_UPDATE is `true` this option does nothing + - To auto-merge this PR, set this to `true` + - Default is unset or `false` +- LANG_CODE + - Two-letter language code that triggers the use of Joel Nitta's {dovetail} package for lesson translation + - This is used in the internationalisation repos of the main Carpentry lesson programs + - Default is unset or `''` + +### 02 Maintain: Check for Updated Packages (update-cache.yaml) + +Repository-level variables for this workflow are: +- LOCKFILE_CACHE_GEN + - Passed to the `generate-cache` input of the [update-lockfile](https://github.com/carpentries/actions/tree/main/update-lockfile) action + - A temporary renv cache is generated when this workflow runs + - If this option is set to `false`, no temporary cache will be generated + - Default is `true` +- FORCE_RENV_INIT + - Passed to the `force-renv-init` input of the [update-lockfile](https://github.com/carpentries/actions/tree/main/update-lockfile) action + - renv initialises a cache based on a given lockfile + - If this lockfile is particularly old or packages have broken/unresolvable dependencies, then builds will fail + - If this option is set to `true`, a full renv reinitialisation will occur, "wiping the slate clean" + - This option is useful if you're using Bioconductor packages which often break when new Bioconductor releases happen + - Default is `false` +- UPDATE_PACKAGES + - Passed to the `update` input of the [update-lockfile](https://github.com/carpentries/actions/tree/main/update-lockfile) action + - If set to `false` only package hydration will happen and no package update checks will occur + - Default is `true` + +### 03 Maintain: Apply Package Cache (docker_apply_cache.yaml) + +Repository-level variables for this workflow are: +- WORKBENCH_TAG + - The workbench-docker release version to use for a given build + - This can be set to a specific version number to force all builds to use a given container version + - Default is unset or `latest` + +### 04 Maintain: Update Workflow Files (update-workflows.yaml) + +There are no repository variables for this workflow. -From here, the markdown documents will be rebuilt and you can inspect what has -changed based on how the packages have updated. ## Pull Request and Review Management -Because our lessons execute code, pull requests are a secruity risk for any -lesson and thus have security measures associted with them. **Do not merge any -pull requests that do not pass checks and do not have bots commented on them.** +Because our lessons execute code, pull requests are a security risk for any lesson and thus have security measures associated with them. +**Do not merge any pull requests that do not pass checks and do not have bots commented on them.** -This series of workflows all go together and are described in the following -diagram and the below sections: +This series of workflows all go together and are described in the following diagram and the below sections: ![Graph representation of a pull request](https://carpentries.github.io/sandpaper/articles/img/pr-flow.dot.svg) ### Pre Flight Pull Request Validation (pr-preflight.yaml) -This workflow runs every time a pull request is created and its purpose is to -validate that the pull request is okay to run. This means the following things: +This workflow runs every time a pull request is created and its purpose is to validate that the pull request is okay to run. +This means the following things: 1. The pull request does not contain modified workflow files -2. If the pull request contains modified workflow files, it does not contain - modified content files (such as a situation where @carpentries-bot will - make an automated pull request) -3. The pull request does not contain an invalid commit hash (e.g. from a fork - that was made before a lesson was transitioned from styles to use the - workbench). +2. If the pull request contains modified workflow files, it does not contain modified content files + (such as a situation where @carpentries-bot will make an automated pull request) +3. The pull request does not contain an invalid commit hash + (e.g. from a fork that was made before a lesson was transitioned from styles to use the Workbench). -Once the checks are finished, a comment is issued to the pull request, which -will allow maintainers to determine if it is safe to run the -"Receive Pull Request" workflow from new contributors. +Once the checks are finished, a comment is issued to the pull request, which will allow maintainers to determine if it is safe to run the "Receive Pull Request" workflow from new contributors. -### Receive Pull Request (pr-receive.yaml) +### Receive Pull Request (docker_pr_receive.yaml) -**Note of caution:** This workflow runs arbitrary code by anyone who creates a -pull request. GitHub has safeguarded the token used in this workflow to have no -priviledges in the repository, but we have taken precautions to protect against -spoofing. +**Note of caution:** This workflow runs arbitrary code by anyone who creates a pull request. +GitHub has safeguarded the token used in this workflow to have no privileges in the repository, but we have taken precautions to protect against spoofing. -This workflow is triggered with every push to a pull request. If this workflow -is already running and a new push is sent to the pull request, the workflow -running from the previous push will be cancelled and a new workflow run will be -started. +This workflow is triggered with every push to a pull request. +If this workflow is already running and a new push is sent to the pull request, the workflow running from the previous push will be cancelled and a new workflow run will be started. -The first step of this workflow is to check if it is valid (e.g. that no -workflow files have been modified). If there are workflow files that have been -modified, a comment is made that indicates that the workflow is not run. If -both a workflow file and lesson content is modified, an error will occurr. +The first step of this workflow is to check if it is valid (e.g. that no workflow files have been modified): +- If there are workflow files that have been modified, a comment is made that indicates that the workflow will not continue. +- If both a workflow file and lesson content is modified, an error will occur and the workflow will not continue. -The second step (if valid) is to build the generated content from the pull -request. This builds the content and uploads three artifacts: +The second step (if valid) is to build the generated content from the pull request. +This builds the content and uploads three artifacts: 1. The pull request number (pr) 2. A summary of changes after the rendering process (diff) 3. The rendered files (build) -Because this workflow builds generated content, it follows the same general -process as the `sandpaper-main` workflow with the same caching mechanisms. - -The artifacts produced are used by the next workflow. +The artifacts produced are used by the "Comment on Pull Request" workflow. ### Comment on Pull Request (pr-comment.yaml) -This workflow is triggered if the `pr-receive.yaml` workflow is successful. +This workflow is triggered if the `docker_pr_receive.yaml` workflow is successful. The steps in this workflow are: -1. Test if the workflow is valid and comment the validity of the workflow to the - pull request. -2. If it is valid: create an orphan branch with two commits: the current state - of the repository and the proposed changes. +1. Test if the workflow is valid and comment the validity of the workflow to the pull request. +2. If it is valid: create an orphan branch with two commits: the current state of the repository and the proposed changes. 3. If it is valid: update the pull request comment with the summary of changes -Importantly: if the pull request is invalid, the branch is not created so any -malicious code is not published. +Importantly: if the pull request is invalid, the branch is not created so any malicious code is not published. -From here, the maintainer can request changes from the author and eventually -either merge or reject the PR. When this happens, if the PR was valid, the -preview branch needs to be deleted. +From here, the maintainer can request changes from the author and eventually either merge or reject the PR. +When this happens, if the PR was valid, the preview branch needs to be deleted. ### Send Close PR Signal (pr-close-signal.yaml) -Triggered any time a pull request is closed. This emits an artifact that is the -pull request number for the next action +Triggered any time a pull request is closed. +This emits an artifact that is the pull request number for the next action. ### Remove Pull Request Branch (pr-post-remove-branch.yaml) -Tiggered by `pr-close-signal.yaml`. This removes the temporary branch associated with -the pull request (if it was created). +Triggered by `pr-close-signal.yaml`. +This removes the temporary branch associated with the pull request (if it was created). diff --git a/.github/workflows/docker_apply_cache.yaml b/.github/workflows/docker_apply_cache.yaml new file mode 100644 index 0000000000..9c44f5beca --- /dev/null +++ b/.github/workflows/docker_apply_cache.yaml @@ -0,0 +1,190 @@ +name: "03 Maintain: Apply Package Cache" +description: "Build and publish the lesson dependency image after a pull request has been merged or via manual trigger" +on: + workflow_dispatch: + inputs: + name: + description: 'Who triggered this build?' + required: true + default: 'Maintainer (via GitHub)' + force-dependency-image-rebuild: + description: 'Rebuild the dependency image layer even if one already exists?' + required: false + default: false + type: boolean + prune-keep-count: + description: 'How many existing dependency image layers to keep?' + required: false + default: 1 + type: number + pull_request: + types: + - closed + branches: + - main + +# queue cache runs +concurrency: + group: docker-apply-cache + cancel-in-progress: false + +jobs: + preflight: + name: "Preflight: PR or Manual Trigger?" + runs-on: ubuntu-latest + outputs: + do-apply: ${{ steps.check.outputs.merged_or_manual }} + steps: + - name: "Should we run cache application?" + id: check + run: | + if [[ "${{ github.event_name }}" == "workflow_dispatch" || + ("${{ github.ref }}" == "refs/heads/main" && "${{ github.event.action }}" == "closed" && "${{ github.event.pull_request.merged }}" == "true") ]]; then + echo "merged_or_manual=true" >> $GITHUB_OUTPUT + else + echo "This was not a manual trigger and no PR was merged. No action taken." + echo "merged_or_manual=false" >> $GITHUB_OUTPUT + fi + shell: bash + + check-renv: + name: "Check If We Need {renv}" + runs-on: ubuntu-latest + needs: preflight + if: needs.preflight.outputs.do-apply == 'true' + outputs: + renv-needed: ${{ steps.check-for-renv.outputs.renv-needed }} + renv-cache-hashsum: ${{ steps.check-for-renv.outputs.renv-cache-hashsum }} + steps: + - name: "Check for renv" + id: check-for-renv + uses: carpentries/actions/renv-checks@v1 + with: + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG || 'latest' }} + skip-cache-check: true + + no-renv-cache-used: + name: "No renv package dependency image needed" + runs-on: ubuntu-latest + needs: check-renv + if: needs.check-renv.outputs.renv-needed != 'true' + steps: + - name: "No dependency image needed" + run: echo "No renv dependency image needed for this lesson" + + update-renv-cache: + name: "Publish renv package dependency image" + runs-on: ubuntu-latest + needs: check-renv + if: needs.check-renv.outputs.renv-needed == 'true' + permissions: + contents: read + packages: write + steps: + - uses: actions/checkout@v6 + + - name: "Get Container Version Used" + id: wb-vers + uses: carpentries/actions/container-version@v1 + with: + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG }} + renv-needed: ${{ needs.check-renv.outputs.renv-needed }} + token: ${{ secrets.GITHUB_TOKEN }} + + - name: Log in to GHCR + uses: docker/login-action@v4 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Set dependency image tags + id: image + env: + IMAGE_OWNER: ${{ github.repository_owner }} + IMAGE_NAME: ${{ github.event.repository.name }} + WB_VERSION: ${{ steps.wb-vers.outputs.container-version }} + RENV_HASH: ${{ needs.check-renv.outputs.renv-cache-hashsum }} + run: | + set -euo pipefail + exact_image="ghcr.io/${IMAGE_OWNER}/${IMAGE_NAME}-deps:${WB_VERSION}_renv-${RENV_HASH}" + latest_image="ghcr.io/${IMAGE_OWNER}/${IMAGE_NAME}-deps:latest" + + # lowercaseify + echo "exact_image=${exact_image,,}" >> "$GITHUB_OUTPUT" + echo "latest_image=${latest_image,,}" >> "$GITHUB_OUTPUT" + shell: bash + + - name: Check for existing dependency image tag + id: image-exists + env: + EXACT_IMAGE: ${{ steps.image.outputs.exact_image }} + run: | + set -euo pipefail + if docker manifest inspect "${EXACT_IMAGE}" >/dev/null 2>&1; then + echo "exists=true" >> "$GITHUB_OUTPUT" + echo "## ⚠️ Dependency image already exists" >> $GITHUB_STEP_SUMMARY + echo "Dependency image already exists for this renv hash: ${EXACT_IMAGE}" >> $GITHUB_STEP_SUMMARY + echo "Dependency image already exists for this renv hash: ${EXACT_IMAGE}" + else + echo "exists=false" >> "$GITHUB_OUTPUT" + echo "No existing dependency image found for this renv hash: ${EXACT_IMAGE}" + fi + shell: bash + + - name: Build and push dependency image layer + id: build-push-deps-layer + if: | + steps.image-exists.outputs.exists != 'true' || + ( + github.event_name == 'workflow_dispatch' && + github.event.inputs.force-dependency-image-rebuild == 'true' + ) + uses: carpentries/actions/build-dependency-image@v1 + with: + workbench-tag: ${{ vars.WORKBENCH_TAG || 'latest' }} + github-token: ${{ secrets.GITHUB_TOKEN }} + github-repository: ${{ github.repository }} + github-sha: ${{ github.sha }} + exact-image: ${{ steps.image.outputs.exact_image }} + latest-image: ${{ steps.image.outputs.latest_image }} + build-context: ${{ github.workspace }} + + prune-dependency-images: + name: "Prune Dependency Images" + runs-on: ubuntu-latest + needs: check-renv + permissions: + contents: read + packages: write + steps: + - name: Prune any old dependency image layers + uses: carpentries/actions/prune-dependency-images@v1 + if: needs.check-renv.outputs.renv-needed == 'true' + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + owner: ${{ github.repository_owner }} + owner-type: ${{ github.event.repository.owner.type }} + repository: ${{ github.event.repository.name }} + package-name: ${{ github.event.repository.name }}-deps + keep-count: ${{ github.event.inputs.prune-keep-count }} + continue-on-error: true + + record-cache-result: + name: "Record Caching Status" + runs-on: ubuntu-latest + needs: [check-renv, update-renv-cache] + if: always() + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - name: "Record cache result" + run: | + echo "${{ needs.check-renv.outputs.renv-needed != 'true' || needs.update-renv-cache.result == 'success' }}" > ${{ github.workspace }}/apply-cache-result + shell: bash + + - name: "Upload cache result" + uses: actions/upload-artifact@v7 + with: + name: apply-cache-result + path: ${{ github.workspace }}/apply-cache-result diff --git a/.github/workflows/docker_build_deploy.yaml b/.github/workflows/docker_build_deploy.yaml new file mode 100644 index 0000000000..151fc4d509 --- /dev/null +++ b/.github/workflows/docker_build_deploy.yaml @@ -0,0 +1,152 @@ +name: "01 Maintain: Build and Deploy Site" +description: "Build and deploy the lesson site using the carpentries/workbench-docker container" +on: + push: + branches: + - 'main' + - 'l10n_main' + paths-ignore: + - '.github/workflows/**.yaml' + - '.github/workbench-docker-version.txt' + schedule: + - cron: '0 0 * * 2' + workflow_run: + workflows: ["03 Maintain: Apply Package Cache"] + types: + - completed + workflow_dispatch: + inputs: + name: + description: 'Who triggered this build?' + required: true + default: 'Maintainer (via GitHub)' + CACHE_VERSION: + description: 'Optional renv cache version override' + required: false + default: '' + reset: + description: 'Reset cached markdown files' + required: true + default: false + type: boolean + force-skip-manage-deps: + description: 'Skip build-time dependency management' + required: true + default: false + type: boolean + +# only one build/deploy at a time +concurrency: + group: docker-build-deploy + cancel-in-progress: true + +jobs: + preflight: + name: "Preflight: Schedule, Push, or PR?" + runs-on: ubuntu-latest + outputs: + do-build: ${{ steps.build-check.outputs.do-build }} + renv-needed: ${{ steps.build-check.outputs.renv-needed }} + renv-cache-hashsum: ${{ steps.build-check.outputs.renv-cache-hashsum }} + dependency-image-ref: ${{ steps.build-check.outputs.dependency-image-ref }} + workbench-container-file-exists: ${{ steps.wb-vers.outputs.workbench-container-file-exists }} + wb-vers: ${{ steps.wb-vers.outputs.container-version }} + last-wb-vers: ${{ steps.wb-vers.outputs.last-container-version }} + workbench-update: ${{ steps.wb-vers.outputs.workbench-update }} + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - name: "Should we run build and deploy?" + id: build-check + uses: carpentries/actions/build-preflight@v1 + + - name: "Checkout Lesson" + if: steps.build-check.outputs.do-build == 'true' + uses: actions/checkout@v6 + + - name: "Get container version info" + id: wb-vers + if: steps.build-check.outputs.do-build == 'true' + uses: carpentries/actions/container-version@v1 + with: + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG }} + renv-needed: ${{ steps.build-check.outputs.renv-needed }} + token: ${{ secrets.GITHUB_TOKEN }} + + full-build: + name: "Build Full Site" + runs-on: ubuntu-latest + needs: preflight + if: | + needs.preflight.outputs.do-build == 'true' && + needs.preflight.outputs.workbench-update != 'true' + env: + RENV_EXISTS: ${{ needs.preflight.outputs.renv-needed }} + RENV_HASH: ${{ needs.preflight.outputs.renv-cache-hashsum }} + permissions: + checks: write + contents: write + pages: write + container: + image: ${{ needs.preflight.outputs.dependency-image-ref }} + env: + WORKBENCH_PROFILE: "ci" + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: /home/rstudio/lesson/renv + RENV_PROFILE: "lesson-requirements" + RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library" + WORKBENCH_OVERLAY_CUSTOMIZATION: ${{ vars.WORKBENCH_OVERLAY_CUSTOMIZATION || '' }} + volumes: + - ${{ github.workspace }}:/home/rstudio/lesson + options: --cpus 1 + steps: + - uses: actions/checkout@v6 + + - name: "Debugging Info" + run: | + cd /home/rstudio/lesson + echo "Current Directory: $(pwd)" + echo "RENV_HASH is $RENV_HASH" + ls -lah /home/rstudio/.workbench + ls -lah $(pwd) + Rscript -e 'sessionInfo()' + shell: bash + + - name: "Mark Repository as Safe" + run: | + git config --global --add safe.directory $(pwd) + shell: bash + + - name: "Run Container and Build Site" + id: build-and-deploy + uses: carpentries/actions/build-and-deploy@v1 + with: + reset: ${{ vars.BUILD_RESET || github.event.inputs.reset || 'false' }} + skip-manage-deps: ${{ github.event.inputs.force-skip-manage-deps == 'true' || contains(needs.preflight.outputs.dependency-image-ref, '-deps:') }} + lang-code: ${{ vars.LANG_CODE || '' }} + + update-container-version: + name: "Update container version used" + runs-on: ubuntu-latest + needs: [preflight] + permissions: + actions: write + contents: write + pull-requests: write + id-token: write + if: | + needs.preflight.outputs.do-build == 'true' && + ( + needs.preflight.outputs.workbench-container-file-exists == 'false' || + needs.preflight.outputs.workbench-update == 'true' + ) + steps: + - name: "Record container version used" + uses: carpentries/actions/record-container-version@v1 + with: + CONTAINER_VER: ${{ needs.preflight.outputs.wb-vers }} + OMIT_PR: ${{ vars.OMIT_PR_CONTAINER_VERSION_UPDATE || 'false' }} + AUTO_MERGE: ${{ vars.AUTO_MERGE_CONTAINER_VERSION_UPDATE || 'false' }} + token: ${{ secrets.GITHUB_TOKEN }} + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} diff --git a/.github/workflows/docker_pr_receive.yaml b/.github/workflows/docker_pr_receive.yaml new file mode 100644 index 0000000000..9ed11c951e --- /dev/null +++ b/.github/workflows/docker_pr_receive.yaml @@ -0,0 +1,221 @@ +name: "Bot: Receive Pull Request" +description: "Receive a pull request and build the markdown source files" +on: + pull_request: + types: + [opened, synchronize, reopened] + workflow_dispatch: + inputs: + pr_number: + type: number + required: true + +concurrency: + group: ${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + pull-requests: write + +jobs: + preflight: + name: "Preflight: md-outputs exists?" + runs-on: ubuntu-latest + outputs: + branch-exists: ${{ steps.check.outputs.exists }} + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v6 + + - name: "Check if md-outputs branch exists" + id: check + run: | + # 💡 Checking for md-outputs branch # + if [[ -n $(git ls-remote --exit-code --heads origin md-outputs) ]]; then + echo "exists=true" >> $GITHUB_OUTPUT + else + echo "exists=false" >> $GITHUB_OUTPUT + echo "::error::md-outputs branch required but does not exist." + echo "::error::Please merge any open package update PRs to trigger the '03 Maintain: Apply Package Cache' and '01: Maintain: Build and Deploy Site' workflows." + + echo "## ❌ ERROR: md-outputs branch required" >> $GITHUB_STEP_SUMMARY + echo "Please merge any open package update PRs to trigger the '03 Maintain: Apply Package Cache' and '01: Maintain: Build and Deploy Site' workflows." >> $GITHUB_STEP_SUMMARY + + exit 1 + fi + shell: bash + + test-pr: + name: "Record PR number" + if: | + github.event.action != 'closed' && + needs.preflight.outputs.branch-exists == 'true' + runs-on: ubuntu-latest + needs: preflight + outputs: + is_valid: ${{ steps.check-pr.outputs.VALID }} + pr_number: ${{ env.NR }} + pr_branch: ${{ env.PR_BRANCH }} + steps: + - name: "Grab PR" + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + if [[ "${{ github.event_name }}" == "pull_request" ]] ; then + PR_NUMBER=${{ github.event.number }} + elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]] ; then + PR_NUMBER=${{ inputs.pr_number }} + fi + + echo $PR_NUMBER > ${{ github.workspace }}/NR + echo "NR=$PR_NUMBER" >> $GITHUB_ENV + echo "PR_BRANCH=$(gh -R ${{ github.repository }} pr view $PR_NUMBER --json headRefName --jq '.headRefName')" >> $GITHUB_ENV + shell: bash + + - name: "Upload PR number" + id: upload + if: always() + uses: actions/upload-artifact@v7 + with: + name: pr + path: ${{ github.workspace }}/NR + + - name: "Get Invalid Hashes File" + id: hash + run: | + echo "json<> $GITHUB_OUTPUT + shell: bash + + - name: "Debug Hashes Output" + run: | + echo "${{ steps.hash.outputs.json }}" + shell: bash + + - name: "Check PR" + id: check-pr + uses: carpentries/actions/check-valid-pr@v1 + with: + pr: ${{ env.NR }} + invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} + + check-renv: + name: "Check If We Need {renv}" + runs-on: ubuntu-latest + outputs: + renv-needed: ${{ steps.renv-check.outputs.renv-needed }} + renv-cache-hashsum: ${{ steps.renv-check.outputs.renv-cache-hashsum }} + dependency-image-ref: ${{ steps.renv-check.outputs.dependency-image-ref }} + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v6 + + - name: "Is renv required?" + id: renv-check + uses: carpentries/actions/renv-checks@v1 + with: + CACHE_VERSION: ${{ inputs.CACHE_VERSION || '' }} + skip-cache-check: true + + build-md-source: + name: "Build markdown source files if valid" + needs: + - test-pr + - check-renv + runs-on: ubuntu-latest + if: needs.test-pr.outputs.is_valid == 'true' + env: + CHIVE: ${{ github.workspace }}/site/chive + PR: ${{ github.workspace }}/site/pr + GHWMD: ${{ github.workspace }}/site/built + PR_BRANCH: ${{ needs.test-pr.outputs.pr_branch }} + PR_NUMBER: ${{ needs.test-pr.outputs.pr_number }} + permissions: + contents: read + container: + image: ${{ needs.check-renv.outputs.dependency-image-ref }} + env: + WORKBENCH_PROFILE: "ci" + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: /home/rstudio/lesson/renv + RENV_PROFILE: "lesson-requirements" + RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library" + volumes: + - ${{ github.workspace }}:/home/rstudio/lesson + options: --cpus 2 + outputs: + build-site: ${{ steps.build-site.outcome }} + steps: + - uses: actions/checkout@v6 + + - name: "Check Out Staging Branch" + uses: actions/checkout@v6 + with: + ref: md-outputs + path: ${{ env.GHWMD }} + + - name: Mark Repository as Safe + run: | + git config --global --add safe.directory $(pwd) + git config --global --add safe.directory /home/rstudio/lesson + shell: bash + + - name: "Ensure sandpaper is loadable" + run: | + .libPaths() + library(sandpaper) + shell: Rscript {0} + + - name: Setup Lesson Dependencies + run: | + Rscript /home/rstudio/.workbench/setup_lesson_deps.R + shell: bash + + - name: "Validate and Build Markdown" + id: build-site + run: | + sandpaper::package_cache_trigger(TRUE) + sandpaper::validate_lesson(path = '/home/rstudio/lesson') + sandpaper:::build_markdown(path = '/home/rstudio/lesson', quiet = FALSE) + shell: Rscript {0} + + - name: "Generate Artifacts" + id: generate-artifacts + run: | + sandpaper:::ci_bundle_pr_artifacts( + repo = '${{ github.repository }}', + pr_number = '${{ env.PR_NUMBER }}', + path_md = '/home/rstudio/lesson/site/built', + path_pr = '/home/rstudio/lesson/site/pr', + path_archive = '/home/rstudio/lesson/site/chive', + branch = 'md-outputs' + ) + shell: Rscript {0} + + - name: "Upload PR" + uses: actions/upload-artifact@v7 + with: + name: pr + path: ${{ env.PR }} + overwrite: true + + - name: "Upload Diff" + uses: actions/upload-artifact@v7 + with: + name: diff + path: ${{ env.CHIVE }} + retention-days: 1 + + - name: "Upload Build" + uses: actions/upload-artifact@v7 + with: + name: built + path: ${{ env.GHWMD }} + retention-days: 1 + + - name: "Teardown" + run: sandpaper::reset_site() + shell: Rscript {0} diff --git a/.github/workflows/pr-close-signal.yaml b/.github/workflows/pr-close-signal.yaml old mode 100755 new mode 100644 index b1303c2615..de1f254482 --- a/.github/workflows/pr-close-signal.yaml +++ b/.github/workflows/pr-close-signal.yaml @@ -16,7 +16,7 @@ jobs: mkdir -p ./pr printf ${{ github.event.number }} > ./pr/NUM - name: Upload Diff - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: pr path: ./pr diff --git a/.github/workflows/pr-comment.yaml b/.github/workflows/pr-comment.yaml old mode 100755 new mode 100644 index f80d9d0c03..c2928514ad --- a/.github/workflows/pr-comment.yaml +++ b/.github/workflows/pr-comment.yaml @@ -1,38 +1,28 @@ name: "Bot: Comment on the Pull Request" - -# read-write repo token -# access to secrets +description: "Comment on the pull request with the results of the markdown generation" on: workflow_run: - workflows: ["Receive Pull Request"] + workflows: ["Bot: Receive Pull Request"] types: - completed -concurrency: - group: pr-${{ github.event.workflow_run.pull_requests[0].number }} - cancel-in-progress: true - - jobs: # Pull requests are valid if: # - they match the sha of the workflow run head commit # - they are open - # - no .github files were committed + # - no .github files were committed, except for .github/workbench-docker-version.txt test-pr: name: "Test if pull request is valid" - runs-on: ubuntu-22.04 - if: > - github.event.workflow_run.event == 'pull_request' && - github.event.workflow_run.conclusion == 'success' + runs-on: ubuntu-latest outputs: is_valid: ${{ steps.check-pr.outputs.VALID }} payload: ${{ steps.check-pr.outputs.payload }} number: ${{ steps.get-pr.outputs.NUM }} msg: ${{ steps.check-pr.outputs.MSG }} steps: - - name: 'Download PR artifact' + - name: "Download PR artifact" id: dl - uses: carpentries/actions/download-workflow-artifact@main + uses: carpentries/actions/download-workflow-artifact@v1 with: run: ${{ github.event.workflow_run.id }} name: 'pr' @@ -50,16 +40,48 @@ jobs: run: | echo '::error::A pull request number was not recorded. The pull request that triggered this workflow is likely malicious.' exit 1 + + - name: "Checkout Lesson" + uses: actions/checkout@v6 + + - name: "Verify committed files" + id: changed-files + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + ## Get list of changed files in the PR ## + ONLY_VERSION=$(gh pr view ${{ steps.get-pr.outputs.NUM }} --json files --jq ' + .files | + length == 1 and + .[0].path == ".github/workbench-docker-version.txt" + ') + + if [[ "$ONLY_VERSION" == "true" ]]; then + echo "only_version_file=true" >> $GITHUB_OUTPUT + else + echo "only_version_file=false" >> $GITHUB_OUTPUT + fi + shell: bash + + - name: "Skip checks for Workbench version file updates" + if: steps.changed-files.outputs.only_version_file == 'true' + run: | + echo "# 🔧 Wait for Next Cache Update #" + echo "Only workbench-docker-version.txt changed." + exit 0 + shell: bash + - name: "Get Invalid Hashes File" id: hash run: | echo "json<> $GITHUB_OUTPUT + - name: "Check PR" id: check-pr if: ${{ steps.dl.outputs.success == 'true' }} - uses: carpentries/actions/check-valid-pr@main + uses: carpentries/actions/check-valid-pr@v1 with: pr: ${{ steps.get-pr.outputs.NUM }} sha: ${{ github.event.workflow_run.head_sha }} @@ -67,6 +89,14 @@ jobs: invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} fail_on_error: true + - name: "Comment result of validation" + id: comment-diff + if: always() + uses: carpentries/actions/comment-diff@v1 + with: + pr: ${{ steps.get-pr.outputs.NUM }} + body: ${{ steps.check-pr.outputs.MSG }} + # Create an orphan branch on this repository with two commits # - the current HEAD of the md-outputs branch # - the output from running the current HEAD of the pull request through @@ -74,32 +104,32 @@ jobs: create-branch: name: "Create Git Branch" needs: test-pr - runs-on: ubuntu-22.04 - if: ${{ needs.test-pr.outputs.is_valid == 'true' }} + runs-on: ubuntu-latest + if: needs.test-pr.outputs.is_valid == 'true' env: NR: ${{ needs.test-pr.outputs.number }} permissions: contents: write steps: - - name: 'Checkout md outputs' - uses: actions/checkout@v4 + - name: "Checkout md outputs" + uses: actions/checkout@v6 with: ref: md-outputs path: built fetch-depth: 1 - - name: 'Download built markdown' + - name: "Download built markdown" id: dl - uses: carpentries/actions/download-workflow-artifact@main + uses: carpentries/actions/download-workflow-artifact@v1 with: run: ${{ github.event.workflow_run.id }} name: 'built' - - if: ${{ steps.dl.outputs.success == 'true' }} + - if: steps.dl.outputs.success == 'true' run: unzip built.zip - name: "Create orphan and push" - if: ${{ steps.dl.outputs.success == 'true' }} + if: steps.dl.outputs.success == 'true' run: | cd built/ git config --local user.email "actions@github.com" @@ -120,27 +150,27 @@ jobs: comment-pr: name: "Comment on Pull Request" needs: [test-pr, create-branch] - runs-on: ubuntu-22.04 - if: ${{ needs.test-pr.outputs.is_valid == 'true' }} + runs-on: ubuntu-latest + if: needs.test-pr.outputs.is_valid == 'true' env: NR: ${{ needs.test-pr.outputs.number }} permissions: pull-requests: write steps: - - name: 'Download comment artifact' + - name: "Download comment artifact" id: dl - uses: carpentries/actions/download-workflow-artifact@main + uses: carpentries/actions/download-workflow-artifact@v1 with: run: ${{ github.event.workflow_run.id }} name: 'diff' - - if: ${{ steps.dl.outputs.success == 'true' }} + - if: steps.dl.outputs.success == 'true' run: unzip ${{ github.workspace }}/diff.zip - name: "Comment on PR" id: comment-diff - if: ${{ steps.dl.outputs.success == 'true' }} - uses: carpentries/actions/comment-diff@main + if: steps.dl.outputs.success == 'true' + uses: carpentries/actions/comment-diff@v1 with: pr: ${{ env.NR }} path: ${{ github.workspace }}/diff.md @@ -150,24 +180,26 @@ jobs: comment-changed-workflow: name: "Comment if workflow files have changed" needs: test-pr - runs-on: ubuntu-22.04 - if: ${{ always() && needs.test-pr.outputs.is_valid == 'false' }} + runs-on: ubuntu-latest + if: | + always() && + needs.test-pr.outputs.is_valid == 'false' env: - NR: ${{ github.event.workflow_run.pull_requests[0].number }} + NR: ${{ needs.test-pr.outputs.number }} body: ${{ needs.test-pr.outputs.msg }} permissions: pull-requests: write steps: - - name: 'Check for spoofing' + - name: "Check for spoofing" id: dl - uses: carpentries/actions/download-workflow-artifact@main + uses: carpentries/actions/download-workflow-artifact@v1 with: run: ${{ github.event.workflow_run.id }} name: 'built' - - name: 'Alert if spoofed' + - name: "Alert if spoofed" id: spoof - if: ${{ steps.dl.outputs.success == 'true' }} + if: steps.dl.outputs.success == 'true' run: | echo 'body<> $GITHUB_ENV echo '' >> $GITHUB_ENV @@ -178,7 +210,7 @@ jobs: - name: "Comment on PR" id: comment-diff - uses: carpentries/actions/comment-diff@main + uses: carpentries/actions/comment-diff@v1 with: pr: ${{ env.NR }} body: ${{ env.body }} diff --git a/.github/workflows/pr-post-remove-branch.yaml b/.github/workflows/pr-post-remove-branch.yaml old mode 100755 new mode 100644 index 9419e2be74..ebbe22dfe2 --- a/.github/workflows/pr-post-remove-branch.yaml +++ b/.github/workflows/pr-post-remove-branch.yaml @@ -17,7 +17,7 @@ jobs: contents: write steps: - name: 'Download artifact' - uses: carpentries/actions/download-workflow-artifact@main + uses: carpentries/actions/download-workflow-artifact@v1 with: run: ${{ github.event.workflow_run.id }} name: pr @@ -27,6 +27,6 @@ jobs: unzip pr.zip echo "NUM=$(<./NUM)" >> $GITHUB_OUTPUT - name: 'Remove branch' - uses: carpentries/actions/remove-branch@main + uses: carpentries/actions/remove-branch@v1 with: pr: ${{ steps.get-pr.outputs.NUM }} diff --git a/.github/workflows/pr-preflight.yaml b/.github/workflows/pr-preflight.yaml old mode 100755 new mode 100644 index 34ad7aed0b..d292d91f1e --- a/.github/workflows/pr-preflight.yaml +++ b/.github/workflows/pr-preflight.yaml @@ -11,7 +11,7 @@ jobs: test-pr: name: "Test if pull request is valid" if: ${{ github.event.action != 'closed' }} - runs-on: ubuntu-22.04 + runs-on: ubuntu-latest outputs: is_valid: ${{ steps.check-pr.outputs.VALID }} permissions: @@ -25,7 +25,7 @@ jobs: EOF" >> $GITHUB_OUTPUT - name: "Check PR" id: check-pr - uses: carpentries/actions/check-valid-pr@main + uses: carpentries/actions/check-valid-pr@v1 with: pr: ${{ github.event.number }} invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} @@ -33,7 +33,7 @@ jobs: - name: "Comment result of validation" id: comment-diff if: ${{ always() }} - uses: carpentries/actions/comment-diff@main + uses: carpentries/actions/comment-diff@v1 with: pr: ${{ github.event.number }} body: ${{ steps.check-pr.outputs.MSG }} diff --git a/.github/workflows/pr-receive.yaml b/.github/workflows/pr-receive.yaml deleted file mode 100755 index 7fbff6cdda..0000000000 --- a/.github/workflows/pr-receive.yaml +++ /dev/null @@ -1,132 +0,0 @@ -name: "Receive Pull Request" - -on: - pull_request: - types: - [opened, synchronize, reopened] - -concurrency: - group: ${{ github.ref }} - cancel-in-progress: true - -jobs: - test-pr: - name: "Record PR number" - if: ${{ github.event.action != 'closed' }} - runs-on: ubuntu-22.04 - outputs: - is_valid: ${{ steps.check-pr.outputs.VALID }} - steps: - - name: "Record PR number" - id: record - if: ${{ always() }} - run: | - echo ${{ github.event.number }} > ${{ github.workspace }}/NR # 2022-03-02: artifact name fixed to be NR - - name: "Upload PR number" - id: upload - if: ${{ always() }} - uses: actions/upload-artifact@v4 - with: - name: pr - path: ${{ github.workspace }}/NR - - name: "Get Invalid Hashes File" - id: hash - run: | - echo "json<> $GITHUB_OUTPUT - - name: "echo output" - run: | - echo "${{ steps.hash.outputs.json }}" - - name: "Check PR" - id: check-pr - uses: carpentries/actions/check-valid-pr@main - with: - pr: ${{ github.event.number }} - invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} - - build-md-source: - name: "Build markdown source files if valid" - needs: test-pr - runs-on: ubuntu-22.04 - if: ${{ needs.test-pr.outputs.is_valid == 'true' }} - env: - GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} - RENV_PATHS_ROOT: ~/.local/share/renv/ - CHIVE: ${{ github.workspace }}/site/chive - PR: ${{ github.workspace }}/site/pr - MD: ${{ github.workspace }}/site/built - steps: - - name: "Check Out Main Branch" - uses: actions/checkout@v4 - - - name: "Check Out Staging Branch" - uses: actions/checkout@v4 - with: - ref: md-outputs - path: ${{ env.MD }} - - - name: "Set up R" - uses: r-lib/actions/setup-r@v2 - with: - use-public-rspm: true - install-r: false - - - name: "Set up Pandoc" - uses: r-lib/actions/setup-pandoc@v2 - - - name: "Setup Lesson Engine" - uses: carpentries/actions/setup-sandpaper@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Setup Package Cache" - uses: carpentries/actions/setup-lesson-deps@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Validate and Build Markdown" - id: build-site - run: | - sandpaper::package_cache_trigger(TRUE) - sandpaper::validate_lesson(path = '${{ github.workspace }}') - sandpaper:::build_markdown(path = '${{ github.workspace }}', quiet = FALSE) - shell: Rscript {0} - - - name: "Generate Artifacts" - id: generate-artifacts - run: | - sandpaper:::ci_bundle_pr_artifacts( - repo = '${{ github.repository }}', - pr_number = '${{ github.event.number }}', - path_md = '${{ env.MD }}', - path_pr = '${{ env.PR }}', - path_archive = '${{ env.CHIVE }}', - branch = 'md-outputs' - ) - shell: Rscript {0} - - - name: "Upload PR" - uses: actions/upload-artifact@v4 - with: - name: pr - path: ${{ env.PR }} - overwrite: true - - - name: "Upload Diff" - uses: actions/upload-artifact@v4 - with: - name: diff - path: ${{ env.CHIVE }} - retention-days: 1 - - - name: "Upload Build" - uses: actions/upload-artifact@v4 - with: - name: built - path: ${{ env.MD }} - retention-days: 1 - - - name: "Teardown" - run: sandpaper::reset_site() - shell: Rscript {0} diff --git a/.github/workflows/sandpaper-main.yaml b/.github/workflows/sandpaper-main.yaml deleted file mode 100755 index b3d1de8c8c..0000000000 --- a/.github/workflows/sandpaper-main.yaml +++ /dev/null @@ -1,64 +0,0 @@ -name: "01 Build and Deploy Site" - -on: - push: - branches: - - main - - master - schedule: - - cron: '0 0 * * 2' - workflow_dispatch: - inputs: - name: - description: 'Who triggered this build?' - required: true - default: 'Maintainer (via GitHub)' - reset: - description: 'Reset cached markdown files' - required: false - default: false - type: boolean -jobs: - full-build: - name: "Build Full Site" - - # 2024-10-01: ubuntu-latest is now 24.04 and R is not installed by default in the runner image - # pin to 22.04 for now - runs-on: ubuntu-22.04 - permissions: - checks: write - contents: write - pages: write - env: - GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} - RENV_PATHS_ROOT: ~/.local/share/renv/ - steps: - - - name: "Checkout Lesson" - uses: actions/checkout@v4 - - - name: "Set up R" - uses: r-lib/actions/setup-r@v2 - with: - use-public-rspm: true - install-r: false - - - name: "Set up Pandoc" - uses: r-lib/actions/setup-pandoc@v2 - - - name: "Setup Lesson Engine" - uses: carpentries/actions/setup-sandpaper@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Setup Package Cache" - uses: carpentries/actions/setup-lesson-deps@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Deploy Site" - run: | - reset <- "${{ github.event.inputs.reset }}" == "true" - sandpaper::package_cache_trigger(TRUE) - sandpaper:::ci_deploy(reset = reset) - shell: Rscript {0} diff --git a/.github/workflows/sandpaper-version.txt b/.github/workflows/sandpaper-version.txt deleted file mode 100644 index ce62dc55bf..0000000000 --- a/.github/workflows/sandpaper-version.txt +++ /dev/null @@ -1 +0,0 @@ -0.16.9 diff --git a/.github/workflows/update-cache.yaml b/.github/workflows/update-cache.yaml old mode 100755 new mode 100644 index a011c0c069..b711f2d8e2 --- a/.github/workflows/update-cache.yaml +++ b/.github/workflows/update-cache.yaml @@ -1,26 +1,45 @@ -name: "03 Maintain: Update Package Cache" - +name: "02 Maintain: Check for Updated Packages" +description: "Check for updated R packages and create a pull request to update the lesson's renv lockfile and package cache" on: + schedule: + - cron: '0 0 * * 2' # every Tuesday at midnight UTC workflow_dispatch: inputs: name: - description: 'Who triggered this build (enter github username to tag yourself)?' + description: 'Who triggered this build?' required: true - default: 'monthly run' - schedule: - # Run every tuesday - - cron: '0 0 * * 2' + default: 'Maintainer (via GitHub)' + force-renv-init: + description: 'Force full lockfile update?' + required: false + default: false + type: boolean + update-packages: + description: 'Install any package updates?' + required: false + default: true + type: boolean + generate-cache: + description: 'Generate separate package cache?' + required: false + default: false + type: boolean + +env: + LOCKFILE_CACHE_GEN: ${{ vars.LOCKFILE_CACHE_GEN || github.event.inputs.generate-cache || 'false' }} + FORCE_RENV_INIT: ${{ vars.FORCE_RENV_INIT || github.event.inputs.force-renv-init || 'false' }} + UPDATE_PACKAGES: ${{ vars.UPDATE_PACKAGES || github.event.inputs.update-packages || 'true' }} jobs: preflight: - name: "Preflight Check" - runs-on: ubuntu-22.04 + name: "Preflight: Manual or Scheduled Trigger?" + runs-on: ubuntu-latest outputs: ok: ${{ steps.check.outputs.ok }} steps: - id: check run: | - if [[ ${{ github.event_name }} == 'workflow_dispatch' ]]; then + if [[ "${{ github.event_name }}" == 'workflow_dispatch' ]]; then echo "ok=true" >> $GITHUB_OUTPUT echo "Running on request" # using single brackets here to avoid 08 being interpreted as octal @@ -33,50 +52,44 @@ jobs: echo "ok=false" >> $GITHUB_OUTPUT echo "Not Running Today" fi + shell: bash - check_renv: - name: "Check if We Need {renv}" - runs-on: ubuntu-22.04 + check-renv: + name: "Check If We Need {renv}" + runs-on: ubuntu-latest needs: preflight - if: ${{ needs.preflight.outputs.ok == 'true'}} + if: ${{ needs.preflight.outputs.ok == 'true' }} outputs: - needed: ${{ steps.renv.outputs.exists }} + renv-needed: ${{ steps.renv-check.outputs.renv-needed }} steps: - name: "Checkout Lesson" - uses: actions/checkout@v4 - - id: renv - run: | - if [[ -d renv ]]; then - echo "exists=true" >> $GITHUB_OUTPUT - fi + uses: actions/checkout@v6 - check_token: - name: "Check SANDPAPER_WORKFLOW token" - runs-on: ubuntu-22.04 - needs: check_renv - if: ${{ needs.check_renv.outputs.needed == 'true' }} - outputs: - workflow: ${{ steps.validate.outputs.wf }} - repo: ${{ steps.validate.outputs.repo }} - steps: - - name: "validate token" - id: validate - uses: carpentries/actions/check-valid-credentials@main + - name: "Is renv required?" + id: renv-check + uses: carpentries/actions/renv-checks@v1 with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} + CACHE_VERSION: ${{ inputs.CACHE_VERSION || '' }} + skip-cache-check: true update_cache: - name: "Update Package Cache" - needs: check_token - if: ${{ needs.check_token.outputs.repo== 'true' }} + name: "Create Package Update Pull Request" runs-on: ubuntu-22.04 + needs: check-renv + permissions: + contents: write + pull-requests: write + actions: write + issues: write + id-token: write + if: needs.check-renv.outputs.renv-needed == 'true' env: GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} RENV_PATHS_ROOT: ~/.local/share/renv/ steps: - - name: "Checkout Lesson" - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: "Set up R" uses: r-lib/actions/setup-r@v2 @@ -86,16 +99,62 @@ jobs: - name: "Update {renv} deps and determine if a PR is needed" id: update - uses: carpentries/actions/update-lockfile@main + uses: carpentries/actions/update-lockfile@v1 with: + update: ${{ env.UPDATE_PACKAGES }} + force-renv-init: ${{ env.FORCE_RENV_INIT }} + generate-cache: ${{ env.LOCKFILE_CACHE_GEN }} cache-version: ${{ secrets.CACHE_VERSION }} - - name: Create Pull Request + - name: "Validate Current Org and Workflow" + id: validate-org-workflow + uses: carpentries/actions/validate-org-workflow@v1 + with: + repo: ${{ github.repository }} + workflow: ${{ github.workflow }} + + - name: "Configure AWS credentials via OIDC" + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.role-to-assume }} + aws-region: ${{ env.aws-region }} + + - name: "Set PAT from AWS Secrets Manager" + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + id: set-pat + run: | + SECRET=$(aws secretsmanager get-secret-value \ + --secret-id carpentries-bot/github-pat \ + --query SecretString --output text) + PAT=$(echo "$SECRET" | jq -r .[]) + echo "::add-mask::$PAT" + echo "pat=$PAT" >> "$GITHUB_OUTPUT" + shell: bash + + # Create the PR with the following roles in order of preference: + # - Carpentries Bot classic PAT fetched from AWS (will only work in official Carpentries repos) + # - repo-scoped SANDPAPER_WORKFLOW classic PAT (will work in all scenarios) + # - default GITHUB_TOKEN (will work suitably, but workflows need to be triggered) + - name: "Create Pull Request" id: cpr - if: ${{ steps.update.outputs.n > 0 }} + if: | + steps.update.outputs.n > 0 uses: carpentries/create-pull-request@main with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} + token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }} delete-branch: true branch: "update/packages" commit-message: "[actions] update ${{ steps.update.outputs.n }} packages" @@ -123,3 +182,9 @@ jobs: [1]: https://github.com/carpentries/create-pull-request/tree/main labels: "type: package cache" draft: false + + - name: "Skip PR creation" + if: steps.update.outputs.n == 0 + run: | + echo "No updates needed, skipping PR creation" + shell: bash diff --git a/.github/workflows/update-workflows.yaml b/.github/workflows/update-workflows.yaml old mode 100755 new mode 100644 index 6414cf2874..6d63a32cc1 --- a/.github/workflows/update-workflows.yaml +++ b/.github/workflows/update-workflows.yaml @@ -1,55 +1,106 @@ -name: "02 Maintain: Update Workflow Files" +name: "04 Maintain: Update Workflow Files" +description: "Update workflow files from the carpentries/sandpaper repository" on: + schedule: + - cron: '0 0 * * 2' workflow_dispatch: inputs: name: description: 'Who triggered this build (enter github username to tag yourself)?' required: true default: 'weekly run' + version: + description: 'Workflows version number (e.g. 0.0.1), branch name (e.g. main), or "latest"' + required: false + default: 'latest' clean: description: 'Workflow files/file extensions to clean (no wildcards, enter "" for none)' required: false default: '.yaml' - schedule: - # Run every Tuesday - - cron: '0 0 * * 2' jobs: - check_token: - name: "Check SANDPAPER_WORKFLOW token" - runs-on: ubuntu-22.04 - outputs: - workflow: ${{ steps.validate.outputs.wf }} - repo: ${{ steps.validate.outputs.repo }} - steps: - - name: "validate token" - id: validate - uses: carpentries/actions/check-valid-credentials@main - with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} - update_workflow: name: "Update Workflow" - runs-on: ubuntu-22.04 - needs: check_token - if: ${{ needs.check_token.outputs.workflow == 'true' }} + runs-on: ubuntu-latest + permissions: + contents: write + pull-requests: write + id-token: write steps: - name: "Checkout Repository" - uses: actions/checkout@v4 + uses: actions/checkout@v6 + + - name: "Validate Current Org and Workflow" + id: validate-org-workflow + uses: carpentries/actions/validate-org-workflow@v1 + with: + repo: ${{ github.repository }} + workflow: ${{ github.workflow }} + + - name: Configure AWS credentials via OIDC + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.role-to-assume }} + aws-region: ${{ env.aws-region }} + + - name: Set PAT from AWS Secrets Manager + id: set-pat + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + run: | + SECRET=$(aws secretsmanager get-secret-value \ + --secret-id carpentries-bot/github-pat \ + --query SecretString --output text) + PAT=$(echo "$SECRET" | jq -r .[]) + echo "::add-mask::$PAT" + echo "pat=$PAT" >> "$GITHUB_OUTPUT" + shell: bash + + - name: "Validate token" + id: validate-token + uses: carpentries/actions/check-valid-credentials@v1 + with: + token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }} + + - name: "No Token Found: Skipping Workflow Update" + if: ${{ steps.validate-token.outputs.wf == 'false' }} + run: | + echo "❗No valid SANDPAPER_WORKFLOW token or PAT from AWS found, cannot update workflows." + + echo "## ❌ Workflow Update Failed" >> $GITHUB_STEP_SUMMARY + echo "No valid SANDPAPER_WORKFLOW token or PAT from AWS found, cannot update workflows." >> $GITHUB_STEP_SUMMARY + shell: bash - name: Update Workflows id: update - uses: carpentries/actions/update-workflows@main + if: ${{ steps.validate-token.outputs.wf == 'true' }} + uses: carpentries/actions/update-workflows@v1 with: - clean: ${{ github.event.inputs.clean }} + version: ${{ github.event.inputs.version || 'latest' }} + clean: ${{ github.event.inputs.clean || '.yaml' }} + token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }} - name: Create Pull Request id: cpr - if: "${{ steps.update.outputs.new }}" + if: | + steps.update.outputs.new && + steps.validate-token.outputs.wf == 'true' uses: carpentries/create-pull-request@main with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} + token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }} delete-branch: true branch: "update/workflows" commit-message: "[actions] update sandpaper workflow to version ${{ steps.update.outputs.new }}" @@ -62,5 +113,5 @@ jobs: - Auto-generated by [create-pull-request][1] on ${{ steps.update.outputs.date }} [1]: https://github.com/carpentries/create-pull-request/tree/main - labels: "type: template and tools" + labels: "type: workflows" draft: false diff --git a/.github/workflows/workflows-version.txt b/.github/workflows/workflows-version.txt new file mode 100644 index 0000000000..e946d6bb71 --- /dev/null +++ b/.github/workflows/workflows-version.txt @@ -0,0 +1 @@ +v1.0.3 diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index f19b804953..4153dd42f8 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -9,5 +9,5 @@ Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by following our [reporting guidelines][coc-reporting]. -[coc-reporting]: https://docs.carpentries.org/topic_folders/policies/incident-reporting.html -[coc]: https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html +[coc-reporting]: https://docs.carpentries.org/policies/coc/incident-reporting.html +[coc]: https://docs.carpentries.org/policies/coc/ diff --git a/LICENSE.md b/LICENSE.md index 7632871ff5..fd13a1b636 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -69,11 +69,11 @@ SOFTWARE. ## Trademark "The Carpentries", "Software Carpentry", "Data Carpentry", and "Library -Carpentry" and their respective logos are registered trademarks of [Community -Initiatives][ci]. +Carpentry" and their respective logos are registered trademarks of +[The Carpentries, Inc.][carpentries]. [cc-by-human]: https://creativecommons.org/licenses/by/4.0/ [cc-by-legal]: https://creativecommons.org/licenses/by/4.0/legalcode [mit-license]: https://opensource.org/licenses/mit-license.html -[ci]: https://communityin.org/ +[carpentries]: https://carpentries.org [osi]: https://opensource.org diff --git a/episodes/01-run-quit.md b/episodes/01-run-quit.md index 54763a8d0e..258fb716c7 100644 --- a/episodes/01-run-quit.md +++ b/episodes/01-run-quit.md @@ -355,7 +355,7 @@ Table: Showing some markdown syntax and its rendered output. | ``` |

| | 1. Use numbers | 1. Use numbers | | 1. to create | 2. to create | -| 1. bullet lists. | 3. numbered lists. | +| 1. numbered lists. | 3. numbered lists. | | ``` | | +---------------------------------------+------------------------------------------------+ +---------------------------------------+------------------------------------------------+ diff --git a/episodes/04-built-in.md b/episodes/04-built-in.md index e11685be73..f8f7463aef 100644 --- a/episodes/04-built-in.md +++ b/episodes/04-built-in.md @@ -259,6 +259,64 @@ NameError: name 'aege' is not defined - Fix syntax errors by reading the source and runtime errors by tracing execution. +## Other ways to get help +There are several other ways that people often get help when they are stuck with their Python code. + +* Search the internet: + paste the last line of your error message or the word "python" and a short description of what you want to do into your favourite search engine + and you will usually find several examples where other people have encountered the same problem and came looking for help. + * [StackOverflow](https://stackoverflow.com/questions) can be particularly helpful for this: answers to questions are presented as a ranked thread ordered according to how useful other users found them to be. + * **Take care:** copying and pasting code written by somebody else is risky unless you understand exactly what it is doing! +* ask somebody "in the real world". + If you have a colleague or friend with more expertise in Python than you have, show them the problem you are having and ask them for help. +* Sometimes, the act of articulating your question can help you to identify what is going wrong. + This is known as ["rubber duck debugging"](https://en.wikipedia.org/wiki/Rubber_duck_debugging) among programmers. + +### Generative AI + +::::::::::::::::::::::::::::: instructor + +### Choose how to teach this section +The section on generative AI is intended to be concise but Instructors may choose to devote more time to the topic in a workshop. +Depending on your own level of experience and comfort with talking about and using these tools, you could choose to do any of the following: + +* Explain how large language models work and are trained, and/or the difference between generative AI, other forms of AI that currently exist, and the limits of what LLMs can do (e.g., they can't "reason"). +* Demonstrate how you recommend that learners use generative AI. +* Discuss the ethical concerns listed below, as well as others that you are aware of, to help learners make an informed choice about whether or not to use generative AI tools. + +This is a fast-moving technology. +If you are preparing to teach this section and you feel it has become outdated, please open an issue on the lesson repository to let the Maintainers know and/or a pull request to suggest updates and improvements. + +:::::::::::::::::::::::::::::::::::::::: + +It is increasingly common for people to use _generative AI_ chatbots such as ChatGPT to get help while coding. +You will probably receive some useful guidance by presenting your error message to the chatbot and asking it what went wrong. +However, the way this help is provided by the chatbot is different. +Answers on StackOverflow have (probably) been given by a human as a direct response to the question asked. +But generative AI chatbots, which are based on an advanced statistical model, respond by generating the _most likely_ sequence of text that would follow the prompt they are given. + +While responses from generative AI tools can often be helpful, they are not always reliable. +These tools sometimes generate plausible but incorrect or misleading information, so (just as with an answer found on the internet) it is essential to verify their accuracy. +You need the knowledge and skills to be able to understand these responses, to judge whether or not they are accurate, and to fix any errors in the code it offers you. + +In addition to asking for help, programmers can use generative AI tools to generate code from scratch; extend, improve and reorganise existing code; translate code between programming languages; figure out what terms to use in a search of the internet; and more. +However, there are drawbacks that you should be aware of. + +The models used by these tools have been "trained" on very large volumes of data, much of it taken from the internet, and the responses they produce reflect that training data, and may recapitulate its inaccuracies or biases. +The environmental costs (energy and water use) of LLMs are a lot higher than other technologies, both during development (known as training) and when an individual user uses one (also called inference). For more information see the [AI Environmental Impact Primer](https://huggingface.co/blog/sasha/ai-environment-primer) developed by researchers at HuggingFace, an AI hosting platform. +Concerns also exist about the way the data for this training was obtained, with questions raised about whether the people developing the LLMs had permission to use it. +Other ethical concerns have also been raised, such as reports that workers were exploited during the training process. + +**We recommend that you avoid getting help from generative AI during the workshop** for several reasons: + +1. For most problems you will encounter at this stage, help and answers can be found among the first results returned by searching the internet. +2. The foundational knowledge and skills you will learn in this lesson by writing and fixing your own programs are essential to be able to evaluate the correctness and safety of any code you receive from online help or a generative AI chatbot. + If you choose to use these tools in the future, the expertise you gain from learning and practising these fundamentals on your own will help you use them more effectively. +3. As you start out with programming, the mistakes you make will be the kinds that have also been made -- and overcome! -- by everybody else who learned to program before you. + Since these mistakes and the questions you are likely to have at this stage are common, they are also better represented than other, more specialised problems and tasks in the data that was used to train generative AI tools. + This means that a generative AI chatbot is _more likely to produce accurate responses_ to questions that novices ask, which could give you a false impression of how reliable they will be when you are ready to do things that are more advanced. + + ::::::::::::::::::::::::::::::::::::::: challenge ## What Happens When diff --git a/episodes/14-looping-data-sets.md b/episodes/14-looping-data-sets.md index 94d7ddbe4d..1a7af685c1 100644 --- a/episodes/14-looping-data-sets.md +++ b/episodes/14-looping-data-sets.md @@ -207,13 +207,21 @@ for filename in glob.glob('data/gapminder_gdp*.csv'): # convenient abstractions for working with filesystem paths and could solve this as well: # from pathlib import Path # region = Path(filename).stem.split('_')[-1] - region = filename.split('_')[-1][:-4] + region = filename.split('_')[-1][:-4] + # extract the years from the columns of the dataframe + headings = dataframe.columns[1:] + years = headings.str.split('_').str.get(1) # pandas raises errors when it encounters non-numeric columns in a dataframe computation # but we can tell pandas to ignore them with the `numeric_only` parameter dataframe.mean(numeric_only=True).plot(ax=ax, label=region) # NOTE: another way of doing this selects just the columns with gdp in their name using the filter method # dataframe.filter(like="gdp").mean().plot(ax=ax, label=region) - +# set the title and labels +ax.set_title('GDP Per Capita for Regions Over Time') +ax.set_xticks(range(len(years))) +ax.set_xticklabels(years) +ax.set_xlabel('Year') +plt.tight_layout() plt.legend() plt.show() ``` diff --git a/episodes/15-coffee.md b/episodes/15-coffee.md index 2cb1ddc11f..03646988a9 100644 --- a/episodes/15-coffee.md +++ b/episodes/15-coffee.md @@ -9,10 +9,12 @@ break: 15 Over break, reflect on and discuss the following: -- A common refrain in software engineering is "Don't Repeat Yourself". How do the techniques we've learned in the last - lessons help us avoid repeating ourselves? *Note that in practice there is some nuance to this and should be balanced - with doing the simplest thing that could possibly work.* -- What are the pros / cons of making a variable global or local to a function? -- When would you consider turning a block of code into a function definition? +- Think about existing data you are working with. Are the data split up into multiple files? How are the files named and organized? Would you be able to efficiently process these files using a for loop? +- If you are creating a new data set, what file naming and data organization practices would help you and your collaborators analyze and interpret your data now and in five or ten years? +Here are some guides for file naming and data collection that we've found. Let us know if you have additional resources you'd like to share! +- https://carpentries-lab.github.io/good-enough-practices/05-project_organization.html +- https://datamanagement.hms.harvard.edu/plan-design/file-naming-conventions +- https://guides.library.stanford.edu/data-best-practices +- https://book.the-turing-way.org/reproducible-research/rdm/rdm-storage/ diff --git a/episodes/16-writing-functions.md b/episodes/16-writing-functions.md index a2f599d0f5..a036f90316 100644 --- a/episodes/16-writing-functions.md +++ b/episodes/16-writing-functions.md @@ -58,6 +58,8 @@ print_greeting() ```output Hello! +The weather is nice today. +Right? ``` ## Arguments in a function call are matched to its defined parameters. @@ -667,6 +669,15 @@ column in the dataframe called `data` labeled `lifExp`. We use the value of this column for every row in the dataframe. +:::::::::::::::::::::::::::::::::::::::::::::::::: + +::::::::::::::::::::::::::::::::::::::::: callout + +## Reflections (optional) + +- How can functions help make code easier to reuse, understand, or change? +- When would you consider turning a block of code into a function definition? + :::::::::::::::::::::::::::::::::::::::::::::::::: :::::::::::::::::::::::::::::::::::::::: keypoints @@ -677,6 +688,7 @@ value of this column for every row in the dataframe. - Arguments in a function call are matched to its defined parameters. - Functions may return a result to their caller using `return`. + :::::::::::::::::::::::::::::::::::::::::::::::::: diff --git a/episodes/17-scope.md b/episodes/17-scope.md index afa533b985..c2b95d1e3c 100644 --- a/episodes/17-scope.md +++ b/episodes/17-scope.md @@ -130,13 +130,14 @@ KeyError: 'Friday' -::::::::::::::::::::::::: +::::::::::::::::::::::::::::::::::::::::: :::::::::::::::::::::::::::::::::::::::::::::::::: :::::::::::::::::::::::::::::::::::::::: keypoints - The scope of a variable is the part of a program that can 'see' that variable. +- What are the pros / cons of making a variable global or local to a function? :::::::::::::::::::::::::::::::::::::::::::::::::: diff --git a/links.md b/links.md index f8e1e4a057..b907142e13 100644 --- a/links.md +++ b/links.md @@ -1,8 +1,8 @@ [cc-by-human]: https://creativecommons.org/licenses/by/4.0/ [cc-by-legal]: https://creativecommons.org/licenses/by/4.0/legalcode [ci]: https://communityin.org/ -[coc-reporting]: https://docs.carpentries.org/topic_folders/policies/incident-reporting.html -[coc]: https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html +[coc-reporting]: https://docs.carpentries.org/policies/coc/incident-reporting.html +[coc]: https://docs.carpentries.org/policies/coc/ [concept-maps]: https://carpentries.github.io/instructor-training/05-memory/ [contrib-covenant]: https://contributor-covenant.org/ [cran-checkpoint]: https://cran.r-project.org/package=checkpoint diff --git a/profiles/learner-profiles.md b/profiles/learner-profiles.md index 434e335aae..a6cbe81b4b 100644 --- a/profiles/learner-profiles.md +++ b/profiles/learner-profiles.md @@ -1,5 +1,23 @@ --- -title: FIXME +title: Learner Profiles --- -This is a placeholder file. Please add content here. +## Carla Correlation + +Carla is a social scientist with a PhD in Sociology and 2 years of research experience. She primarily uses SPSS for data analysis and has extensive domain knowledge in her field. Her department is increasingly adopting Python for research, but she has never written code beyond SPSS syntax files. + +This course will teach Carla the Python fundamentals needed to analyze cross-country demographic data for her paper on education outcomes. She's particularly interested in the data importing, visualization, and statistical analysis episodes. After completing the workshop, Carla will be able to independently import CSV files, perform basic data cleaning, create simple visualizations of demographic trends, and produce summary statistics from tabular data without relying on colleagues for code support. + +This workshop serves as a first step in Carla's Python journey, providing her with a coherent mental model of programming and data visualization that will form the foundation of her future learning. She will be equipped to ask well informed questions about programming in the future, recognize what's possible with Python, and independently explore resources to expand her skills. + +## Jim JIT + +Jim teaches biology and environmental science to high school students with 12 years of teaching experience. While comfortable with educational technology, he has no programming experience. He wants to incorporate real-world data analysis into his curriculum to engage students with current global challenges and teach them valuable skills, but is concerned about simplifying complex concepts for teenage students and designing activities that can fit within 45-minute class periods. + +This course will provide Jim with the foundational Python knowledge needed to create data science activities for his classroom. He's most interested in the basic programming concepts and data visualization aspects that he can adapt for student use. After the workshop, Jim will be able to create simple, guided activities using the Gapminder dataset that connect scientific concepts to real-world data, demonstrate basic data visualization techniques to his students, and confidently answer questions about the code. He'll develop lesson materials that help prepare his students for college-level coursework while teaching both scientific concepts and analytical skills. + +## Peter Pandas + +Peter is a medical student conducting research during a gap year before residency. With an MD degree completed and some undergraduate coursework in statistics, they need to analyze global health data for a project on childhood vaccination rates. Despite being tech-savvy with other digital tools, Peter has never programmed before and has very limited time available between clinical responsibilities. They struggle with abstract concepts and prefer learning through concrete examples directly applicable to their work. + +This course will teach Peter the essential Python skills needed to analyze datasets for their research projects. They're particularly interested in the data filtering and visualization episodes that will allow them to extract meaningful patterns from data. After completing the workshop, Peter will be able to import their research data, filter it based on specific criteria like geographic region or time period and create visualizations illustrating relationships. For repeated steps, they will be able to write functions to not have to repeat themselves.