diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ba0fe56..2bfcbaa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -58,12 +58,12 @@ jobs: // extension matches (they're workflow definitions with real // behavioral weight, some running with pull_request_target // privileges). Everything else — including .claude/*.md prompt - // and command docs, which carry no execution weight themselves - // — counts. + // and command docs, and README.html (a generated rendering of + // README.asc, no execution weight of its own) — counts. // // This does NOT skip claude-review: that's a separate workflow // gated by its own `if:`, unaffected by this check's outputs. - const DOC_EXTENSIONS = /\.(md|asc|adoc|asciidoc)$/i; + const DOC_EXTENSIONS = /\.(md|asc|adoc|asciidoc|html)$/i; const changedFiles = await github.paginate(github.rest.pulls.listFiles, { owner: context.repo.owner, repo: context.repo.repo, diff --git a/HISTORY.asc b/HISTORY.asc index 8b002e6..32537fc 100644 --- a/HISTORY.asc +++ b/HISTORY.asc @@ -1,5 +1,13 @@ STABLE ------ +== Add `make pgxntool-version` to print the embedded pgxntool version +There was no way to tell which pgxntool version a project had embedded via +`git subtree` short of digging through git history. `make pgxntool-version` +(and `pgxntool/bin/version` directly) now prints it, read from the +first line of `pgxntool/HISTORY.asc` -- the same line the release process +stamps with the version number. The line is rejected unless it's `STABLE` +or a plain `X.Y.Z` version (digits only, no pre-release suffixes). + == Fix `make pgtle PGTLE_VERSION=X` generating all versions instead of one `base.mk`'s `pgtle` target never passed the `PGTLE_VERSION` make variable through to `pgtle.sh --pgtle-version`, so it silently generated all pg_tle diff --git a/README.asc b/README.asc index c592f2e..b059805 100644 --- a/README.asc +++ b/README.asc @@ -265,6 +265,11 @@ TIP: The actual work is done by `pgxntool/pgxntool-sync.sh`, so you can run it d TIP: There is also a `pgxntool-sync-%` rule if you need to do more advanced things. `make pgxntool-sync-` pulls from the ` ` defined by the `pgxntool-sync-` make variable. +=== pgxntool-version +`make pgxntool-version` prints the version of the embedded pgxntool copy, read from the first line of `pgxntool/HISTORY.asc`. That line is "STABLE" instead of a version number if this copy was synced from an unreleased commit rather than a tagged release. + +TIP: The actual work is done by `pgxntool/bin/version`, so you can run it directly (`pgxntool/bin/version`) if you'd rather not go through `make`. + === distclean `make distclean` removes generated configuration files (`META.json` — generated from `META.in.json` — plus `meta.mk` and `control.mk`) that survive a normal `make clean`. diff --git a/README.html b/README.html index 8926728..e1bf26f 100644 --- a/README.html +++ b/README.html @@ -457,20 +457,22 @@

PGXNtool

  • 4.7. tag
  • 4.8. dist
  • 4.9. pgxntool-sync
    • -
  • 4.10. distclean
    • -
  • 4.11. pgtle
    • -
  • 4.12. check-pgtle
    • -
  • 4.13. run-pgtle
    • +
  • 4.10. pgxntool-version
    • +
  • 4.11. distclean
    • +
  • 4.12. pgtle
    • +
  • 4.13. check-pgtle
    • +
  • 4.14. run-pgtle
  • 5. Version-Specific SQL Files
  • 6. Document Handling @@ -518,7 +520,7 @@

    PGXNtool

    PGXNtool is meant to make developing new Postgres extensions for PGXN easier.

    -

    Currently, it consists a base Makefile that you can include instead of writing your own, a template META.json, and some test framework. More features will be added over time.

    +

    Currently, it consists a base Makefile that you can include instead of writing your own, a template META.in.json (from which META.json is generated — you edit the former, never the latter directly), and some test framework. More features will be added over time.

    If you find any bugs or have ideas for improvements, please open an issue.

    @@ -802,6 +804,81 @@

    Key detail: Install files and regular tests run in a single pg_regress invocation. This means the database is NOT dropped between install and test phases — state created by install files persists into the main test suite.

    +
    +

    4.4.1. Update & Upgrade (U&U) Testing

    +
    +

    Beyond validating a plain install, it’s worth testing that your extension behaves correctly across the two transitions every extension with more than one release eventually goes through:

    +
    +
    +
      +
    • +

      Update: ALTER EXTENSION ext UPDATE, moving to a newer extension version on the same PostgreSQL version.

      +
      • +
    • +

      Upgrade: pg_upgrade, moving to a newer PostgreSQL major version while carrying the extension’s on-disk catalog state along with it.

      +
      • +
    +
    +
    +

    Both catch bugs that a simple "did the script run without error" check misses — wrong mappings, missing `REVOKE`s, objects that drifted from the base SQL, or catalog state that doesn’t survive a binary upgrade cleanly. This isn’t a niche concern that only applies if your extension does something unusual (adds enum types, custom operators, etc.) — it applies to any extension that will ever be updated or upgraded in place, which in practice is every extension.

    +
    +
    +

    test/install is the recommended place to build this, because its "load once, committed, before the suite" mechanic (described above) is exactly what update/upgrade testing needs.

    +
    +
    +

    The pattern: load the extension exactly once, committed, from a test/install/*.sql file, in one of several modes selected by a custom backend setting that your Makefile sets via PGOPTIONS (e.g. -c myext.test_load_mode=$(TEST_LOAD_SOURCE)) and that test/install reads with current_setting('myext.test_load_mode'):

    +
    +
    +
      +
    • +

      fresh: CREATE EXTENSION ext;

      +
      • +
    • +

      update: CREATE EXTENSION ext VERSION 'oldest_supported'; then ALTER EXTENSION ext UPDATE; (to the current default_version, or an explicit intermediate version)

      +
      • +
    • +

      existing: the extension is already installed — by a real pg_upgrade, or an update performed outside the suite — and test/install only asserts it’s present at the expected version, without dropping or touching it

      +
      • +
    +
    +
    +

    The same test/sql/ suite and the *same test/expected/ output then run unchanged against every mode. Identical expected output passing in *update mode is the update-equivalence assertion — if the updated database behaves any differently than a fresh install, some test will diff. Running that same suite in existing mode against a database that just went through a real pg_upgrade gives you upgrade testing using the same test files, no separate suite required.

    +
    +
    +

    If you adopt this pattern, test/deps.sql should no longer run CREATE EXTENSION itself for the per-test setup — the committed install performed by test/install persists into every (rolled-back) test file, exactly as described above.

    +
    +
    +

    Why the install step must be committed, not run per-test in test/deps.sql: every test/sql/ file runs inside BEGIN; …​ ROLLBACK; — a pgxntool convention (test/pgxntool/setup.sql opens the transaction; since it’s never committed, it rolls back when the session ends). Running the update inside that transaction means the suite never actually exercises a committed update — which doesn’t match production, where ALTER EXTENSION UPDATE commits before anything else touches the database, and it hides bugs that only show up once the update is durably committed. (Modifying an enum is one example of how this can turn into an outright failure rather than just a hidden bug — but it’s just an example; the reason to commit first holds regardless of what the update script does.)

    +
    +
    +

    As a bonus, sharing one committed install across every test file also removes the per-test re-install cost.

    +
    +
    + + + + + +
    +
    Note
    +    		 +This pattern isn’t (yet) native to pgxntool — you wire up the mode selection and PGOPTIONS plumbing yourself in test/install and your Makefile. First-class support (a standard mode-switching toggle, and scaffolding for real pg_upgrade testing) is planned; see issue #42. +
    +
    +
    +

    Working examples (specific files, not full PRs — both extensions' histories include plenty of unrelated changes):

    +
    +
    +
      +
    • +

      cat_tools test/install/load.sql and the TEST_LOAD_SOURCE block in its Makefile — the fresh/update/existing pattern above, including the PGOPTIONS wiring. This is a reasonably complete example, and somewhat more elaborate than the minimum needed to get started.

      +
      • +
    • +

      pg_count_nulls test/sql/extension_tests.sql additionally demonstrates testing an extension installed into a non-default schema. It’s a useful reference for that specific problem, but it also defines its assertions as plpgsql functions rather than plain pgTAP calls — a pattern that adds complexity of its own and isn’t recommended as a model for U&U testing itself.

      +
      • +
    +
    +

    4.5. testdeps

    @@ -906,7 +983,7 @@

    4.7. tag

    -

    make tag will create a git branch for the current version of your extension, as determined by the META.json file. The reason to do this is so you can always refer to the exact code that went into a released version.

    +

    make tag will create an annotated git tag for the current version of your extension, as determined by the META.json file (generated from META.in.json — see PGXN Distributions vs. Extensions), and push it to origin. The reason to do this is so you can always refer to the exact code that went into a released version.

    If there’s already a tag for the current version that probably means you forgot to update META.json, so you’ll get an error. If you’re certain you want to over-write the tag, you can do make forcetag, which removes the existing tag (via make rmtag) and creates a new one.

    @@ -927,7 +1004,7 @@

    4.

    4.8. dist

    -

    make dist will create a .zip file for your current version that you can upload to PGXN. The file is named after the PGXN name and version (the top-level "name" and "version" attributes in META.json). The .zip file is placed in the parent directory so as not to clutter up your git repo.

    +

    make dist will create a .zip file for your current version that you can upload to PGXN. It first runs tag (creating/pushing the version’s git tag as described above if needed), then uses git archive at that tag to build the .zip file, so the archive always matches the exact tagged commit. The file is named after the PGXN name and version (the top-level "name" and "version" attributes in META.json, generated from META.in.json). The .zip file is placed in the parent directory so as not to clutter up your git repo.

    @@ -985,9 +1062,27 @@

    -

    4.10. distclean

    +

    4.10. pgxntool-version

    +
    +

    make pgxntool-version prints the version of the embedded pgxntool copy, read from the first line of pgxntool/HISTORY.asc. That line is "STABLE" instead of a version number if this copy was synced from an unreleased commit rather than a tagged release.

    +
    +
    +
    + + + + +
    +
    Tip
    +    		 +The actual work is done by pgxntool/bin/version, so you can run it directly (pgxntool/bin/version) if you’d rather not go through make. +
    +
    +
    +
    +

    4.11. distclean

    -

    make distclean removes generated configuration files (META.json, meta.mk, control.mk) that survive a normal make clean.

    +

    make distclean removes generated configuration files (META.json — generated from META.in.json — plus meta.mk and control.mk) that survive a normal make clean.

    @@ -1011,7 +1106,7 @@

    -

    4.11. pgtle

    +

    4.12. pgtle

    Generates pg_tle (Trusted Language Extensions) registration SQL files for deploying extensions in managed environments like AWS RDS/Aurora. See [_pg_tle_Support] for complete documentation.

    @@ -1020,7 +1115,7 @@

    -

    4.12. check-pgtle

    +

    4.13. check-pgtle

    Checks if pg_tle is installed and reports the version. This target: - Reports the version from pg_extension if CREATE EXTENSION pg_tle has been run in the database @@ -1036,7 +1131,7 @@

    -

    4.13. run-pgtle

    +

    4.14. run-pgtle

    Registers all extensions with pg_tle by executing the generated pg_tle registration SQL files in a PostgreSQL database. This target: - Requires pg_tle extension to be installed (checked via check-pgtle) @@ -1081,14 +1176,36 @@

    -

    5.1. How Version Files Are Generated

    +

    5.1. PGXN Distributions vs. Extensions

    +
    +

    PGXN distinguishes two things it’s easy to conflate: a distribution and an extension. Per the PGXN meta spec, a distribution is "a collection of extensions, source code, utilities, tests, and/or documents that are distributed together" — the thing you release and upload to PGXN, identified by the top-level name/version in META.json (generated from META.in.json — you edit the latter, never META.json directly). An extension is the individual thing installed via CREATE EXTENSION, described by its own .control file. A single distribution routinely provides more than one extension — META.in.json’s `provides map lists each one; the pgTAP distribution, for example, provides both the pgtap and schematap extensions from one release.

    +
    +
    +

    This split matters for versioning, and it’s where pgxntool has a real gap: META.json’s top-level `version is the distribution’s release version (used by make tag and make dist — it’s the git tag name and the .zip filename). Each extension’s own version, though, ends up tracked in two separate places that pgxntool does not keep in sync for you:

    +
    +
    +
      +
    • +

      the extension’s .control file (default_version) — what PostgreSQL and pgxntool’s own build actually use; see What Controls the Version Number below

      +
      • +
    • +

      that extension’s entry under META.in.json’s `provides map (provides.{extension}.version) — which the PGXN meta spec defines as "a Version for the extension" in its own right, not the distribution’s version, and which is what PGXN’s public index reads

      +
      • +
    +
    +
    +

    Nothing generates one from the other or checks that they still agree, so it’s easy to bump one and forget the other; issue #47 tracks fixing that.

    +
    +
    +
    +

    5.2. How Version Files Are Generated

    When you run make (or make all), PGXNtool:

    1. -

      Reads your META.json file to determine the extension version from provides.{extension}.version

      +

      Reads each extension’s own .control file to determine its version from default_version (via control.mk.sh)

    2. Generates a Makefile rule that copies your base SQL file (sql/{extension}.sql) to the version-specific file (sql/{extension}--{version}.sql)

      @@ -1099,16 +1216,11 @@

      -

      For example, if your META.json contains:

      +

      For example, if your myext.control contains:

    -
    "provides": {
-  "myext": {
-    "version": "1.2.3",
-    ...
-  }
-}
    +
    default_version = '1.2.3'
    @@ -1116,21 +1228,22 @@

    -

    5.2. What Controls the Version Number

    +

    5.3. What Controls the Version Number

    -

    The version number comes from META.json → provides.{extension}.version, not from your .control file’s default_version field. The .control file’s default_version is used by PostgreSQL to determine which version to install by default, but the actual version-specific file that gets generated is determined by what’s in META.json.

    +

    The version number comes from each extension’s own .control file → default_version, not from META.json. PostgreSQL itself uses default_version to pick which versioned SQL file to load, so control.mk.sh parses the .control file(s) directly to keep the generated filename in sync with what PostgreSQL will actually use. META.in.json has its own, separate provides.{extension}.version for that same extension (see PGXN Distributions vs. Extensions above) — pgxntool never reads it when generating SQL files, and nothing keeps it in sync with the .control file’s default_version.

    -

    To change the version of your extension: -1. Update provides.{extension}.version in META.json +

    To change the version of one of your extensions: +1. Update default_version in that extension’s .control file 2. Run make to regenerate the version-specific file -3. Update default_version in your .control file to match (if needed)

    +3. Update that extension’s provides.{extension}.version in META.in.json to match by hand, and regenerate META.json (make META.json) — pgxntool won’t do this for you +4. Update META.in.json’s top-level `version too if you’re also cutting a new distribution release

    -

    5.3. Committing Version Files

    +

    5.4. Committing Version Files

    -

    Version-specific SQL files are now treated as permanent files that should be committed to your repository. This makes it much easier to test updates to extensions, as you can see exactly what SQL was included in each version.

    +

    Version-specific SQL files are treated as permanent files that should be committed to your repository by default. This makes it much easier to test updates to extensions, as you can see exactly what SQL was included in each version.

    @@ -1144,9 +1257,82 @@

    +

    5.4.1. Why Commit Them: Update Testing

    +
    +

    The primary value of committing a version’s install script is update testing: with the file in place, you can install that exact version (CREATE EXTENSION ext VERSION 'x.y.z'), run ALTER EXTENSION ext UPDATE, and verify the update path actually works. In principle you would only ever need the very first committed version’s install script — every later version is reachable by chaining upgrade scripts from it. See test/install for how to wire this into your test suite.

    +
    +
    +

    In practice, you should keep most old versions tracked rather than relying purely on that chain, because you cannot predict when a new major PostgreSQL version will break the ability to install an older extension version — a system catalog column changes type, a SELECT * over a catalog starts failing, and so on. Committing the old version’s install script lets CI catch that regression directly, by attempting to install that exact historical version against the new PostgreSQL release. For any non-trivial extension, most old versions should stay tracked for this reason alone.

    +
    + +
    +

    5.4.2. When It’s OK to Skip a Version

    +
    +

    For a minor version change that doesn’t meaningfully alter the extension (e.g. a small bug fix), it’s unlikely to straddle a PostgreSQL supported-version boundary, so there’s much less test-coverage value in committing that specific version’s generated install script. For large extensions, deliberately not committing some of these individual version files is worth doing to keep repository size down — the base sql/{extension}.sql still fully captures the change in git history, and the install script itself is trivially regenerated from that base file at build time.

    +
    +
    +
    +

    5.4.3. Don’t .gitignore a Skipped Version — rm It Once

    +
    +

    When you decide not to track a particular version’s generated install script, do not add a .gitignore entry for it.

    +
    +
    +

    make only ever generates the current version’s file, stamped from the base source according to that extension’s .control file default_version. The moment you bump the version, make starts generating the new current version’s file and stops touching the old one — the old file left behind in your working tree is a stale, one-off artifact, not something make keeps recreating on every build.

    +
    +
    +

    Because it’s only ever transiently present, the correct cleanup is a single rm sql/{extension}--{old-version}.sql right after the version bump — not a permanent .gitignore rule. A per-version .gitignore line would be perpetual clutter added on every release, for a file that’s only ever present for one build cycle. A broad glob (e.g. sql/--.sql) is wrong for the same reason it’s wrong below in Alternative: Ignoring All Version Files: it would also hide the prior-version install scripts and upgrade scripts (sql/{extension}--{a}--{b}.sql) that you do want tracked and visible in git status.

    +
    +
    +
    +

    5.4.4. Never Hand-Edit a Version File That’s No Longer Current

    +
    +

    control.mk.sh generates a Make rule along these lines for the current version’s file:

    +
    +
    +
    +
    $(EXTENSION_ext_VERSION_FILE): sql/ext.sql extension.control
+	@echo '/* DO NOT EDIT - AUTO-GENERATED FILE */' > $(EXTENSION_ext_VERSION_FILE)
+	@cat sql/ext.sql >> $(EXTENSION_ext_VERSION_FILE)
    +
    +
    +
    +

    That rule only ever targets the file matching the extension’s current default_version, so editing the base sql/{extension}.sql and running make is the correct, safe way to change that one file.

    +
    +
    +

    Every other versioned file — any sql/{extension}--{version}.sql where {version} is no longer current — is a frozen historical record, not something make will ever regenerate or overwrite again. Its entire purpose is to preserve exactly what shipped in that version (see Why Commit Them: Update Testing above), so it can keep being used to test update paths and PostgreSQL-version compatibility against exactly what your users actually installed.

    +
    +
    +

    Hand-editing an old versioned file directly is never correct — it silently corrupts that historical record. If you need to change behavior after a version has already shipped, bump the version and add a proper sql/{extension}--{old}--{new}.sql upgrade script instead. Never edit sql/{extension}--{old}.sql in place.

    +
    +
    +

    + + + + +
    +
    Caution
    +    		 +This is especially relevant for AI coding agents, which won’t know this convention exists unless it’s spelled out. The generic DO NOT EDIT - AUTO-GENERATED FILE header on every version file doesn’t distinguish "regenerated on every build" (the current version) from "was generated once, now frozen" (every other version). An agent without this context may reasonably — but incorrectly — treat any auto-generated file as fair game to patch directly. Old versioned SQL files must instead be treated as append-only history: the fix is always a new version plus an upgrade script, never an in-place edit. +
    +
    +

    -

    5.4. Alternative: Ignoring Version Files

    +

    5.5. Alternative: Ignoring All Version Files

    +
    + + + + + +
    +
    Note
    +    		 +This is a different, all-or-nothing choice from When It’s OK to Skip a Version above, where the recommendation is to keep tracking most versions and selectively skip only a few low-value minor ones. This section instead covers not tracking any version-specific files at all. +
    +

    If you prefer not to commit version-specific SQL files, you must add them to your .gitignore to prevent make dist from failing due to untracked files. Add the following to your .gitignore:

    @@ -1174,13 +1360,13 @@

    -

    5.5. Distribution Inclusion

    +

    5.6. Distribution Inclusion

    Version-specific files are included in distributions created by make dist only if they are committed to git. Since make dist uses git archive, only tracked files are included in the distribution archive.

    -

    5.6. Multiple Versions

    +

    5.7. Multiple Versions

    If you need to support multiple versions of your extension:

    @@ -1193,7 +1379,7 @@

    Create upgrade scripts for version transitions (e.g., sql/myext—​1.0.0—​1.1.0.sql)

  • -

    Update META.json to reflect the current version you’re working on

    +

    Update default_version in the extension’s .control file to reflect the current version you’re working on

  • Commit all version files and upgrade scripts to your repository

    @@ -1201,7 +1387,7 @@

    -

    The version file for the current version (specified in META.json) will be automatically regenerated when you run make, but other version files you create manually will be preserved.

    +

    The version file for the current version (specified in the .control file’s default_version) will be automatically regenerated when you run make, but other version files you create manually will be preserved.

    @@ -1647,7 +1833,7 @@

    diff --git a/base.mk b/base.mk index 1a419d5..0f9d9c3 100644 --- a/base.mk +++ b/base.mk @@ -601,6 +601,12 @@ pgxntool-sync: pgxntool-sync-%: @pgxntool/pgxntool-sync.sh $($@) +# `make pgxntool-version` prints the version of the embedded pgxntool copy. +# Delegates to bin/version so it can be run without make too. +.PHONY: pgxntool-version +pgxntool-version: + @$(PGXNTOOL_DIR)/bin/version + # DANGER! Use these with caution. They may add extra crap to your history and # could make resolving merges difficult! # `pgxntool-sync` (no suffix) already pulls the canonical release; these are the diff --git a/bin/version b/bin/version new file mode 100755 index 0000000..c18af60 --- /dev/null +++ b/bin/version @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# +# bin/version - Print the pgxntool version embedded in this project +# +# The release process (see .claude/skills/release) stamps the first line of +# HISTORY.asc with the released version number, so that line is the source +# of truth for "what version of pgxntool is this?". If this copy was synced +# from an unreleased commit rather than a tagged release, the first line +# will be "STABLE" instead of a version number -- that's printed as-is, since +# it's an accurate answer too. Anything else (a corrupt or hand-edited file) +# is rejected rather than printed as if it were a real answer. +# +# Usage: bin/version [] +# +# history-file Path to HISTORY.asc to read. Defaults to the copy +# one level up from this script (i.e. pgxntool/HISTORY.asc +# in a normal checkout). + +set -o errexit -o errtrace -o pipefail +trap 'echo "Error on line ${LINENO}"' ERR + +PGXNTOOL_DIR="$(dirname "${BASH_SOURCE[0]}")/.." +source "$PGXNTOOL_DIR/lib.sh" + +history_file="${1:-$PGXNTOOL_DIR/HISTORY.asc}" + +[[ -f "$history_file" ]] || die 1 "$history_file not found" + +version=$(head -n1 "$history_file") + +[[ -n "$version" ]] || die 1 "$history_file is empty" + +# Matches the X.Y.Z format the release process validates and stamps. +[[ "$version" == "STABLE" || "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || \ + die 1 "$history_file: first line '$version' is neither STABLE nor a X.Y.Z version" + +echo "$version"