Automated package checks

As well as running the crosswalk by hand, you can make your package's own test suite verify that its metadata and formatting stay clean: that running ResearchSoftwareMetadata.crosswalk() on the package regenerates exactly the files that are committed (so the metadata is consistent and up to date), and that running JuliaFormatter leaves the source code unchanged (so the code is well formatted). If either check would change a file, the test fails until you re-run the crosswalk or formatter and commit the result. The crosswalk check also fails if the crosswalk logs a warning, so the package's metadata has to be complete, with a description, a category and build instructions, as well as consistent.

This is how ResearchSoftwareMetadata tests itself, and the test files are written so that they can be copied into any other package.

Files to copy

Copy these three files from ResearchSoftwareMetadata's test/ directory into your own package's test/ directory:

GitUtils.jl — provides is_repo_clean, which the other two files use to check whether the checks changed anything.
clean_ResearchSoftwareMetadata.jl — checks the metadata crosswalk is clean.
clean_JuliaFormatter.jl — checks the code formatting is clean.

You can use either or both of the clean_*.jl files; the test-suite code below discovers whichever ones are present.

None of the three files needs adapting apart from its first line. Each starts with # SPDX-License-Identifier: MIT, which is this package's licence, and the crosswalk stops with an error if it finds a file marked with a licence that your package has not declared. So either change that line in each file to your own package's licence, or keep the files under the MIT licence and say so in your Project.toml:

[rsmd]
additional_licenses = ["MIT"]

The two clean_*.jl files are:

# SPDX-License-Identifier: MIT

module CleanRSMD
using Test
using Git
using Logging
using ResearchSoftwareMetadata

include("GitUtils.jl")
using .GitUtils

# Does not currently work on Windows runners on GitHub due to file writing issues
if !haskey(ENV, "RUNNER_OS") || ENV["RUNNER_OS"] ≠ "Windows"
    @testset "RSMD" begin
        git_dir = readchomp(`$(Git.git()) rev-parse --show-toplevel`)
        @test isnothing(ResearchSoftwareMetadata.crosswalk())
        global_logger(SimpleLogger(stderr, Logging.Warn))
        @test_nowarn ResearchSoftwareMetadata.crosswalk()
        global_logger(SimpleLogger(stderr, Logging.Info))
        @test is_repo_clean(git_dir, strict = haskey(ENV, "RUNNER_OS"))
    end
else
    @test_broken !haskey(ENV, "RUNNER_OS") || ENV["RUNNER_OS"] ≠ "Windows"
end

end

# SPDX-License-Identifier: MIT

module CleanJuliaFormatter
using Test
using Git
using JuliaFormatter

include("GitUtils.jl")
using .GitUtils

# Does not currently work on Windows runners on GitHub due to file writing issues
if !haskey(ENV, "RUNNER_OS") || ENV["RUNNER_OS"] ≠ "Windows"
    @testset "JuliaFormatter" begin
        git_dir = readchomp(`$(Git.git()) rev-parse --show-toplevel`)
        @test_nowarn format(git_dir)
        @test is_repo_clean(git_dir, strict = haskey(ENV, "RUNNER_OS"))
    end
else
    @test_broken !haskey(ENV, "RUNNER_OS") || ENV["RUNNER_OS"] ≠ "Windows"
end

end

The formatting check assumes a .JuliaFormatter.toml file in your package root defining your formatting standard, as described at the end of the home page.

Running them from your test suite

Add this block to the end of your test/runtests.jl. It discovers any clean_*.jl files in your test/ directory and runs each one in its own testset:

rsmd = get(ENV, "RSMD_CROSSWALK", "FALSE")
if rsmd == "TRUE" || !haskey(ENV, "RUNNER_OS") # Crosswalk runner or local testing
    # Test RSMD crosswalk and other hygiene issues

    # Identify files that are checking package hygiene
    cleanbase = map(file -> replace(file, r"clean_(.*).jl$" => s"\1"),
                    filter(str -> occursin(r"^clean_.*\.jl$", str),
                           readdir(@__DIR__)))

    if length(cleanbase) > 0
        @info "Crosswalk and clean testing:"
        @testset begin
            for c in cleanbase
                println("    = $c")
            end
            println()

            @testset for c in cleanbase
                fn = "clean_$c.jl"
                println("    * Verifying $c.jl ...")
                include(fn)
            end
        end
    end
end

The checks always run locally (when RUNNER_OS is not set); on GitHub runners they only run when the RSMD_CROSSWALK environment variable is set to TRUE (see below).

Finally, add the test dependencies to your Project.toml — Git, JuliaFormatter, Logging, ResearchSoftwareMetadata and Test need to appear in [extras] (with their UUIDs) and in the test list in [targets].

Running them on CI

On GitHub Actions, set RSMD_CROSSWALK: TRUE on one canonical runner rather than the whole matrix — formatter output can differ between Julia and JuliaFormatter versions, so running the checks everywhere invites spurious failures. With a matrix like this package's, that looks like:

      - uses: julia-actions/julia-runtest@v1
        env:
          RSMD_CROSSWALK: ${{ (matrix.os == 'ubuntu-latest' && matrix.julia-version == '1') && 'TRUE' || 'FALSE' }}

Some things to be aware of:

The crosswalk queries orcid.org, ror.org, spdx.org, doi.org and the Julia General registry, and for a version that has not been released it asks your git remote which branch is its default, so the runner needs network access. It also interrogates the git history and tags, so the repository must be checked out in full (fetch-depth: 0 on actions/checkout).
Windows runners are skipped by both checks (marked as broken) due to file writing issues.
Until a version has a release tag, its dates in codemeta.json are provisional. On your default branch every crosswalk sets them to today's date (in UTC), so there the check fails on any later day until the crosswalk is run again; that is what keeps the dates right in the code you release. On any other branch, and so in a pull request, the dates are left as they are, and ResearchSoftwareMetadata.crosswalk(update = true) or one of the increase_ functions brings them up to date.
On CI the repository must be strictly clean — no staged, unstaged or untracked changes after the checks run. Locally the criterion is relaxed: only unstaged changes fail the tests, so work you have already staged doesn't stop you running the suite.