Migrate to spago v1

[pete-murphy]

Description of the change

Resolves #315

This migrates the client and staging Spago projects to use latest (1.0.4), and updates compiler version and package set to latest while I was at it. Note that 0.15.15 is latest version available on Hackage at time of writing.

For client updates, I just followed these instructions in Spago repo: https://github.com/purescript/spago#migrate-from-spagodhall-to-spagoyaml.

For staging/Haskell updates I followed the instructions in RELEASE.md that I also updated in these changes. I did not change stack.yaml resolver because 0.15.15 is still on lts-20.9.

This is a large diff, but most of that is in generated files. Ignoring staging/* (effectively generated from following the RELEASE.md instructions) and client/spago.lock, the diff is much more reasonable:

$ git diff origin/master --stat ':(exclude)staging/*' ':(exclude)client/spago.lock'

 .github/workflows/ci.yml              |  4 +--
 CHANGELOG.md                          |  3 ++
 README.md                             |  9 +++---
 RELEASE.md                            | 72 ++++++++++++++++++++++++--------------------
 client/config/dev/Try.Config.purs     | 15 ---------
 client/config/prod/Try.Config.purs    | 15 ---------
 client/package.json                   | 19 ++++++------
 client/packages.dhall                 |  5 ---
 client/spago.dhall                    | 45 ---------------------------
 client/spago.yaml                     | 48 +++++++++++++++++++++++++++++
 client/src/Try/Config.js              | 19 ++++++++++++
 client/src/Try/Config.purs            |  5 +++
 client/src/Try/SharedConfig.purs      |  7 +++--
 client/updateSharedConfigVersions.mjs | 20 +++++++++---
 stack.yaml                            |  2 +-
 stack.yaml.lock                       |  8 ++---
 16 files changed, 154 insertions(+), 142 deletions(-)

That said, I am open splitting this up into separate MRs.

---

Checklist:

• Added the change to the changelog's "Unreleased" section with a reference to this PR (e.g. "- Made a change (#0 by @)")
• Linked any existing issues or proposals that this pull request should close
• Updated or added relevant documentation
• Added a test for the contribution (if applicable)

[pete-murphy]

Oh no, I tried running locally with same memory settings as production build (set -o noglob && stack exec trypurescript -- +RTS -N2 -A128m -M3G -s -RTS 8081 $(spago sources)) and ran out of memory.

❯ (set -o noglob && stack exec trypurescript -- +RTS -N2 -A128m -M3G -s -RTS 8081 $(spago sources))
Reading Spago workspace configuration...

✓ Selecting package to build: try-purescript-server


Warning: nix (Nix) 2.34.7+1 is on the PATH (at /run/current-system/sw/bin/nix)
         but Stack's Nix integration is disabled. To mute this message in
         future, set notify-if-nix-on-path: false in Stack's configuration.
         
trypurescript: Heap exhausted;
trypurescript: Current maximum heap size is 3221225472 bytes (3072 MB).
trypurescript: Use `+RTS -M<size>' to increase it.
  36,907,900,144 bytes allocated in the heap
   7,635,067,328 bytes copied during GC
   3,479,980,704 bytes maximum residency (28 sample(s))
      21,874,656 bytes maximum slop
            3739 MiB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0       108 colls,    42 par    2.344s   3.292s     0.0305s    0.1313s
  Gen  1        28 colls,     5 par   39.524s  111.686s     3.9888s    7.0771s

  Parallel GC work balance: 86.50% (serial 0%, perfect 100%)

  TASKS: 9 (1 bound, 8 peak workers (8 total), using -N2)

  SPARKS: 7427 (7381 converted, 0 overflowed, 0 dud, 0 GC'd, 46 fizzled)

  INIT    time    0.001s  (  0.080s elapsed)
  MUT     time   42.664s  (  9.678s elapsed)
  GC      time   41.868s  (114.978s elapsed)
  EXIT    time    0.002s  (  0.005s elapsed)
  Total   time   84.534s  (124.742s elapsed)

  Alloc rate    865,091,047 bytes per MUT second

  Productivity  50.5% of total user, 7.8% of total elapsed

Running with -M4G succeeds.

[pete-murphy]

Dumping the result of investigating this with Opus: https://gist.github.com/pete-murphy/bd776708d72a00a80ba1b7e7845f2982. Option 3 seems the most attractive if it could work.

[thomashoneyman]

Option 3 is indeed attractive and I’m perfectly happy experimenting with reworking try PureScript, it’s definitely not received a lot of love recently.

I wouldn’t dismiss option 2 offhand — Try PureScript is browser based so certainly node packages, etc. can safely be excluded

[pete-murphy]

I wouldn’t dismiss option 2 offhand — Try PureScript is browser based so certainly node packages, etc. can safely be excluded

I did end up looking into Option 2 after all since it seems much easier and I am less likely to get to the harder Option 3 in near term. If I trim out Node packages & some large packages that have no dependents, I am able to run the server with (set -o noglob && stack exec trypurescript -- +RTS -N2 -A128m -M3G -s -RTS 8081 $(spago sources)) without exceeding that max heap size. The result was this commit: 0fd79d9.

The non-Node packages (largest packages with no dependents) that were omitted:

• framer-motion
• motion
• next-purs-rsc
• react-basic-dom-beta
• react-icons
• css-frameworks
• lumi-components
• yoga-sqlite
• rito

#4269d0 are packages that were kept
#e0820e is omitted (because Node)
#c33d3d is omitted (because big package with no dependents)

Here's a notebook for exploring what packages to omit if it's useful: https://new.observablehq.com/@pete-murphy/try-pure-script-package-set-trim-explorer (mostly made by Opus 4.8)

The "no dependents" criterion is arbitrary, open to ideas if you can think of something that would be more fair? (Age of package, some other popularity metric...)

[thomashoneyman]

Quick followup on your questions in the gist:

1. The droplet has 3.8 GiB RAM (and no swap, though we could add it), so bumping to -M5G is off the table for now; we already overpay for trypurescript IMO so I don't want to resize upward unless absolutely necessary. So I think we should do the trimmed package set path.

2. I looked, and production is actually already occasionally being OOM-killed with the current package set. journalctl shows the service dying with status=9/KILL four times on April 28 and again on May 24, with Restart=always quietly recovering each time. So the existing -M3G cap plus other overhead is already (occasionally) insufficient!

3. I also checked the staging/output directory, and this doesn't exist on the box

I think the option 2 is the way to go, but before we merge, could you measure the trimmed set's peak (maxLive from +RTS -s) on a cold boot (no output)? If it's much above ~2.5 GB, it's probably worth trimming a little further.

Longer term, option 3 looks even better, but we don't need to do it now.

[thomashoneyman]

Got a little access to Fable, so ran it over this change — here are results, lightly proofread:

<---- fable ---->
Following up on the trim-criteria question with some data. I scanned every package in the untrimmed set (638 packages) for bare JS import specifiers in their foreign modules — i.e. npm/Node imports that Try PureScript cannot resolve at runtime, since the playground ships no node_modules. One important nuance: client/public/frame.html already has a JSPM import map that shims exactly seven specifiers (react, react-dom, react-dom/client, react-dom/server, big-integer, decimal.js, uuid), so the disqualifying condition is a bare import not covered by the import map.

Results, combined with the dependency graph from spago.lock:

• 118 packages directly contain unshimmed bare imports. Including their transitive dependents, the "cannot run in the playground" closure is 148 packages / 3,658 modules — nearly half the full set's 7,426 modules.
• The current trim in 0fd79d9 cuts 79 packages / 2,945 modules, but 75 of the packages it keeps are in the cannot-run closure — they typecheck here but throw the moment you run them.
• Conversely, only 6 cut packages are actually runnable: spec, spec-mocha, test-unit, pmock, react-basic-dom-beta (all its imports are shimmed), and css-frameworks (runs, though its class names do nothing without the frameworks' stylesheets in the frame).

So my suggestion: trim by "cannot execute in the playground" instead of size/dependents. It cuts ~24% more modules than the current trim (more memory headroom), it's fully principled — nothing is removed for being big or unpopular, only for being unable to run — and it lets spec, test-unit, and react-basic-dom-beta come back.

Two caveats:

1. Package-level is an upper bound. Some flagged packages have a single broken module (e.g. which touching child_process); a user who never imports that module is unaffected. Fine to keep borderline popular ones.
2. "Cannot run" is fixable, not a death sentence. Browser-friendly packages (chartjs, xterm, idb, nanoid, markdown-it-js, react-markdown, …) could be rescued with one import-map entry each via the JSPM generator. That also gives package authors a concrete path to inclusion, which answers the fairness question better than any popularity metric.

75 packages kept by the current trim that cannot run

Package	Unshimmed imports (or broken dependency)
ace	ace-builds, ace/anchor, ace/background_tokenizer, ace/document, ace/edit_session …
address-rfc2821	address-rfc2821
chartjs	chart.js/auto
chartjs-halogen	depends on chartjs
choku	chalk
debounce	debouncing
deno	https://deno.land/std@0.144.0/crypto/mod.ts, https://deno.land/std@0.144.0/dotenv/mod.ts, https://deno.land/std@0.144.0/fs/mod.ts, https://deno.land/std@0.144.0/http/server.ts, https://deno.land/std@0.145.0/datetime/mod.ts …
droplet	pg
echarts-simple	echarts
elmish	react-dom/server.js
elmish-enzyme	enzyme
elmish-hooks	stacktrace-parser
elmish-html	depends on elmish
elmish-testing-library	@happy-dom/global-registrator, react-dom/test-utils.js
elmish-time-machine	depends on elmish, elmish-hooks, elmish-html
fahrtwind	depends on react-basic-emotion
faker-ffi	@faker-js/faker
fakerjs	@faker-js/faker
gojs	gojs
halogen-echarts-simple	depends on echarts-simple
halogen-xterm	depends on xterm
hylograph-d3-kernel	d3-drag, d3-force, d3-selection
hylograph-simulation	d3-drag, d3-force, d3-selection
hylograph-simulation-halogen	depends on hylograph-d3-kernel, hylograph-simulation
i18next	i18next, i18next-browser-languagedetector
idb	idb
jsdom	jsdom
leveldb	level
logging-journald	depends on systemd-journald
markdown-it-js	markdown-it, markdown-it-collapsible
milkdown	@milkdown/crepe, @milkdown/kit/core, @milkdown/kit/utils
milkis	node-fetch
mysql	mysql
n3	n3
nanoid	nanoid
nextjs	next/document, next/head, next/image.js, next/link, next/navigation …
nextui	@nextui-org/react, next-themes
node-sqlite3	sqlite3
oak	virtual-dom/create-element, virtual-dom/diff, virtual-dom/h, virtual-dom/patch
oak-debug	depends on oak
react-aria	@react-aria/button, @react-aria/combobox, @react-aria/focus, @react-aria/interactions, @react-aria/listbox …
react-basic-dnd	react-dnd, react-dnd-html5-backend, react-dnd-test-backend, react-dnd-touch-backend
react-basic-emotion	@emotion/react
react-basic-storybook	@storybook/addon-actions, @storybook/addon-docs, @storybook/testing-library
react-dnd-kit	@dnd-kit/abstract/modifiers, @dnd-kit/collision, @dnd-kit/dom, @dnd-kit/dom/modifiers, @dnd-kit/dom/utilities …
react-markdown	react-markdown, remark-breaks, remark-gfm
react-virtuoso	react-virtuoso
reactix	@testing-library/react, react-dom/test-utils
recharts	recharts
rough-notation	rough-notation
systemd-journald	systemd-journald
tanstack-query	@tanstack/react-query
toestand	depends on reactix
ulid	ulid
url-regex-safe	url-regex-safe
visx	@visx/annotation, @visx/axis, @visx/brush, @visx/clip-path, @visx/curve …
vitest	vitest
webextension-polyfill	webextension-polyfill
which	which
xterm	xterm, xterm-addon-fit, xterm-addon-web-links, xterm-addon-webgl
yaml-next	js-yaml
yoga-better-auth	better-auth, better-auth/client, better-auth/db, better-auth/plugins, pg
yoga-dynamodb	@aws-sdk/client-dynamodb, @aws-sdk/lib-dynamodb
yoga-elasticsearch	@elastic/elasticsearch
yoga-fastify	@fastify/cors, @fastify/helmet, @fastify/rate-limit, @fastify/websocket, argon2 …
yoga-fetch	node-fetch
yoga-heroui	@heroui/react
yoga-jaeger	jaeger-client
yoga-next-fastify	next
yoga-pino	@opentelemetry/api-logs, pino
yoga-postgres	pg
yoga-react-native	react-native, react-native-fs, react-native-macos, twrnc
yoga-redis	ioredis
yoga-shadcn	cmdk, embla-carousel-react, input-otp, radix-ui, react-day-picker …
z3	z3-solver

Full data: package → unshimmed bare imports, cannot-run closure, current cut list (JSON)

{
 "unshimmed": {
  "affjax-node": [
   "url",
   "xhr2"
  ],
  "yoga-pino": [
   "@opentelemetry/api-logs",
   "pino"
  ],
  "motion": [
   "motion/react",
   "motion/react-m"
  ],
  "yoga-om-strom": [
   "node:fs"
  ],
  "hylograph-d3-kernel": [
   "d3-drag",
   "d3-force",
   "d3-selection"
  ],
  "react-testing-library": [
   "@testing-library/react/pure.js",
   "@testing-library/user-event"
  ],
  "yoga-opentelemetry": [
   "@opentelemetry/api",
   "@opentelemetry/api-logs",
   "@opentelemetry/exporter-logs-otlp-http",
   "@opentelemetry/exporter-trace-otlp-http",
   "@opentelemetry/instrumentation-http",
   "@opentelemetry/instrumentation-pino",
   "@opentelemetry/resources",
   "@opentelemetry/sdk-logs",
   "@opentelemetry/sdk-node",
   "@opentelemetry/sdk-trace-base",
   "@opentelemetry/sdk-trace-node",
   "@opentelemetry/semantic-conventions"
  ],
  "node-fs": [
   "node:fs",
   "util"
  ],
  "org-doc": [
   "instaparse",
   "node:fs"
  ],
  "nanoid": [
   "nanoid"
  ],
  "react-virtuoso": [
   "react-virtuoso"
  ],
  "yoga-redis": [
   "ioredis"
  ],
  "chartjs": [
   "chart.js/auto"
  ],
  "yoga-fastify": [
   "@fastify/cors",
   "@fastify/helmet",
   "@fastify/rate-limit",
   "@fastify/websocket",
   "argon2",
   "fastify",
   "jose"
  ],
  "yoga-test-docker": [
   "child_process"
  ],
  "node-net": [
   "net",
   "node:net"
  ],
  "react-markdown": [
   "react-markdown",
   "remark-breaks",
   "remark-gfm"
  ],
  "graphql-client": [
   "@apollo/client/core/index.js",
   "@apollo/client/link/context/index.js",
   "@apollo/client/link/subscriptions/index.js",
   "@apollo/client/utilities/index.js",
   "@urql/core",
   "graphql-ws",
   "wonka"
  ],
  "yoga-fetch": [
   "node-fetch"
  ],
  "visx": [
   "@visx/annotation",
   "@visx/axis",
   "@visx/brush",
   "@visx/clip-path",
   "@visx/curve",
   "@visx/event",
   "@visx/geo",
   "@visx/gradient",
   "@visx/grid",
   "@visx/group",
   "@visx/heatmap",
   "@visx/hierarchy",
   "@visx/legend",
   "@visx/mock-data",
   "@visx/network",
   "@visx/pattern",
   "@visx/responsive",
   "@visx/scale",
   "@visx/shape",
   "@visx/stats",
   "@visx/threshold",
   "@visx/tooltip",
   "@visx/zoom",
   "d3-format",
   "d3-time-format",
   "topojson-client"
  ],
  "postgresql": [
   "buffer",
   "pg",
   "pg-copy-streams",
   "postgres-interval",
   "postgres-range",
   "stream"
  ],
  "node-child-process": [
   "node:child_process"
  ],
  "react-aria": [
   "@react-aria/button",
   "@react-aria/combobox",
   "@react-aria/focus",
   "@react-aria/interactions",
   "@react-aria/listbox",
   "@react-aria/overlays",
   "@react-aria/utils"
  ],
  "gojs": [
   "gojs"
  ],
  "echarts-simple": [
   "echarts"
  ],
  "fakerjs": [
   "@faker-js/faker"
  ],
  "yoga-heroui": [
   "@heroui/react"
  ],
  "xterm": [
   "xterm",
   "xterm-addon-fit",
   "xterm-addon-web-links",
   "xterm-addon-webgl"
  ],
  "recharts": [
   "recharts"
  ],
  "yaml-next": [
   "js-yaml"
  ],
  "react-basic-storybook": [
   "@storybook/addon-actions",
   "@storybook/addon-docs",
   "@storybook/testing-library"
  ],
  "nextui": [
   "@nextui-org/react",
   "next-themes"
  ],
  "url-regex-safe": [
   "url-regex-safe"
  ],
  "node-os": [
   "node:os",
   "node:util"
  ],
  "spec-discovery": [
   "fs",
   "path",
   "url"
  ],
  "which": [
   "which"
  ],
  "yoga-dynamodb": [
   "@aws-sdk/client-dynamodb",
   "@aws-sdk/lib-dynamodb"
  ],
  "droplet": [
   "pg"
  ],
  "oak": [
   "virtual-dom/create-element",
   "virtual-dom/diff",
   "virtual-dom/h",
   "virtual-dom/patch"
  ],
  "yoga-jaeger": [
   "jaeger-client"
  ],
  "simple-ulid": [
   "crypto"
  ],
  "yoga-postgres": [
   "pg"
  ],
  "node-zlib": [
   "node:zlib"
  ],
  "elmish-hooks": [
   "stacktrace-parser"
  ],
  "debounce": [
   "debouncing"
  ],
  "yoga-react-native": [
   "react-native",
   "react-native-fs",
   "react-native-macos",
   "twrnc"
  ],
  "rough-notation": [
   "rough-notation"
  ],
  "reactix": [
   "@testing-library/react",
   "react-dom/test-utils"
  ],
  "framer-motion": [
   "motion/react"
  ],
  "choku": [
   "chalk"
  ],
  "hylograph-simulation": [
   "d3-drag",
   "d3-force",
   "d3-selection"
  ],
  "yoga-next-fastify": [
   "next"
  ],
  "yoga-shadcn": [
   "cmdk",
   "embla-carousel-react",
   "input-otp",
   "radix-ui",
   "react-day-picker",
   "react-resizable-panels",
   "sonner",
   "vaul"
  ],
  "node-readline": [
   "node:readline"
  ],
  "node-human-signals": [
   "os"
  ],
  "next-purs-rsc": [
   "next/cache",
   "next/dynamic",
   "next/font/google",
   "next/font/local",
   "next/headers",
   "next/image",
   "next/link",
   "next/navigation",
   "next/og",
   "next/script",
   "next/server",
   "next/web-vitals"
  ],
  "systemd-journald": [
   "systemd-journald"
  ],
  "node-sqlite3": [
   "sqlite3"
  ],
  "ink": [
   "ink"
  ],
  "address-rfc2821": [
   "address-rfc2821"
  ],
  "node-http": [
   "node:http",
   "node:https"
  ],
  "deno": [
   "https://deno.land/std@0.144.0/crypto/mod.ts",
   "https://deno.land/std@0.144.0/dotenv/mod.ts",
   "https://deno.land/std@0.144.0/fs/mod.ts",
   "https://deno.land/std@0.144.0/http/server.ts",
   "https://deno.land/std@0.145.0/datetime/mod.ts",
   "https://deno.land/std@0.145.0/log/mod.ts",
   "https://deno.land/std@0.145.0/uuid/mod.ts"
  ],
  "elmish-enzyme": [
   "enzyme"
  ],
  "yoga-config": [
   "smol-toml",
   "yaml"
  ],
  "cbor-stream": [
   "cbor-x"
  ],
  "blessed": [
   "@shamansir/everblessed",
   "blessed",
   "fs",
   "path",
   "reblessed",
   "terminal-size",
   "url",
   "util"
  ],
  "node-tls": [
   "node:tls"
  ],
  "elmish-testing-library": [
   "@happy-dom/global-registrator",
   "react-dom/test-utils.js"
  ],
  "node-process": [
   "process"
  ],
  "node-buffer": [
   "node:buffer",
   "util"
  ],
  "react-dnd-kit": [
   "@dnd-kit/abstract/modifiers",
   "@dnd-kit/collision",
   "@dnd-kit/dom",
   "@dnd-kit/dom/modifiers",
   "@dnd-kit/dom/utilities",
   "@dnd-kit/geometry",
   "@dnd-kit/helpers",
   "@dnd-kit/react",
   "@dnd-kit/react/sortable"
  ],
  "rito": [
   "three/examples/jsm/loaders/GLTFLoader.js",
   "three/examples/jsm/postprocessing/BloomPass.js",
   "three/examples/jsm/postprocessing/EffectComposer.js",
   "three/examples/jsm/postprocessing/GlitchPass.js",
   "three/examples/jsm/postprocessing/RenderPass.js",
   "three/examples/jsm/postprocessing/ShaderPass.js",
   "three/examples/jsm/postprocessing/UnrealBloomPass.js",
   "three/examples/jsm/renderers/CSS2DRenderer.js",
   "three/examples/jsm/renderers/CSS3DRenderer.js",
   "three/src/cameras/PerspectiveCamera.js",
   "three/src/core/BufferAttribute.js",
   "three/src/core/BufferGeometry.js",
   "three/src/core/InstancedBufferAttribute.js",
   "three/src/core/Raycaster.js",
   "three/src/geometries/BoxGeometry.js",
   "three/src/geometries/CapsuleGeometry.js",
   "three/src/geometries/CylinderGeometry.js",
   "three/src/geometries/PlaneGeometry.js",
   "three/src/geometries/SphereGeometry.js",
   "three/src/lights/AmbientLight.js",
   "three/src/lights/DirectionalLight.js",
   "three/src/lights/PointLight.js",
   "three/src/loaders/CubeTextureLoader.js",
   "three/src/loaders/TextureLoader.js",
   "three/src/materials/MeshBasicMaterial.js",
   "three/src/materials/MeshLambertMaterial.js",
   "three/src/materials/MeshPhongMaterial.js",
   "three/src/materials/MeshStandardMaterial.js",
   "three/src/materials/RawShaderMaterial.js",
   "three/src/materials/ShaderMaterial.js",
   "three/src/math/Box3.js",
   "three/src/math/Color.js",
   "three/src/math/Euler.js",
   "three/src/math/Matrix4.js",
   "three/src/math/Quaternion.js",
   "three/src/math/Sphere.js",
   "three/src/math/Vector2.js",
   "three/src/math/Vector3.js",
   "three/src/objects/Group.js",
   "three/src/objects/InstancedMesh.js",
   "three/src/objects/Mesh.js",
   "three/src/objects/Points.js",
   "three/src/renderers/WebGLRenderer.js",
   "three/src/scenes/FogExp2.js",
   "three/src/scenes/Scene.js"
  ],
  "milkis": [
   "node-fetch"
  ],
  "node-http2": [
   "node:http2"
  ],
  "webextension-polyfill": [
   "webextension-polyfill"
  ],
  "i18next": [
   "i18next",
   "i18next-browser-languagedetector"
  ],
  "supabase": [
   "@supabase/ssr",
   "@supabase/supabase-js"
  ],
  "lumi-components": [
   "jss",
   "jss-preset-default",
   "qrcode.react",
   "react-media-hook",
   "react-password-strength-bar"
  ],
  "marked": [
   "marked"
  ],
  "elmish": [
   "react-dom/server.js"
  ],
  "prospero": [
   "@fastify/multipart",
   "graphql",
   "node:crypto"
  ],
  "axon": [
   "bun",
   "node:http",
   "node:net",
   "node:stream",
   "stream"
  ],
  "markdown-it-js": [
   "markdown-it",
   "markdown-it-collapsible"
  ],
  "react-basic-dnd": [
   "react-dnd",
   "react-dnd-html5-backend",
   "react-dnd-test-backend",
   "react-dnd-touch-backend"
  ],
  "optparse": [
   "child_process"
  ],
  "node-path": [
   "node:path"
  ],
  "node-url": [
   "node:url"
  ],
  "z3": [
   "z3-solver"
  ],
  "ulid": [
   "ulid"
  ],
  "node-streams": [
   "node:stream"
  ],
  "leveldb": [
   "level"
  ],
  "whine-core": [
   "glob",
   "micromatch",
   "vscode",
   "vscode-languageclient/node.js",
   "vscode-languageserver-textdocument",
   "vscode-languageserver/node",
   "yaml"
  ],
  "ace": [
   "ace-builds",
   "ace/anchor",
   "ace/background_tokenizer",
   "ace/document",
   "ace/edit_session",
   "ace/editor",
   "ace/ext/language_tools",
   "ace/range",
   "ace/scrollbar",
   "ace/search",
   "ace/selection",
   "ace/token_iterator",
   "ace/tokenizer",
   "ace/undomanager",
   "ace/virtual_renderer"
  ],
  "node-stream-pipes": [
   "stream"
  ],
  "node-event-emitter": [
   "node:events"
  ],
  "golem-fetch": [
   "node:url"
  ],
  "express": [
   "cookie-parser",
   "express",
   "http",
   "https"
  ],
  "idb": [
   "idb"
  ],
  "yoga-elasticsearch": [
   "@elastic/elasticsearch"
  ],
  "mysql": [
   "mysql"
  ],
  "yoga-sqlite": [
   "@libsql/client"
  ],
  "meowclient": [
   "keys-converter",
   "meowclient"
  ],
  "n3": [
   "n3"
  ],
  "milkdown": [
   "@milkdown/crepe",
   "@milkdown/kit/core",
   "@milkdown/kit/utils"
  ],
  "yoga-acp-om": [
   "@anthropic-ai/claude-agent-sdk"
  ],
  "vitest": [
   "vitest"
  ],
  "ezfetch": [
   "buffer",
   "stream"
  ],
  "tanstack-query": [
   "@tanstack/react-query"
  ],
  "yoga-better-auth": [
   "better-auth",
   "better-auth/client",
   "better-auth/db",
   "better-auth/plugins",
   "pg"
  ],
  "node-execa": [
   "process"
  ],
  "webb-commandline": [
   "child_process"
  ],
  "csv-stream": [
   "csv-parse",
   "csv-stringify"
  ],
  "react-icons": [
   "react-icons/ai",
   "react-icons/bi",
   "react-icons/bs",
   "react-icons/cg",
   "react-icons/ci",
   "react-icons/di",
   "react-icons/fa",
   "react-icons/fa6",
   "react-icons/fc",
   "react-icons/fi",
   "react-icons/gi",
   "react-icons/go",
   "react-icons/gr",
   "react-icons/hi",
   "react-icons/hi2",
   "react-icons/im",
   "react-icons/io",
   "react-icons/io5",
   "react-icons/lia",
   "react-icons/lu",
   "react-icons/md",
   "react-icons/pi",
   "react-icons/ri",
   "react-icons/rx",
   "react-icons/si",
   "react-icons/sl",
   "react-icons/tb",
   "react-icons/tfi",
   "react-icons/ti",
   "react-icons/vsc",
   "react-icons/wi"
  ],
  "react-basic-emotion": [
   "@emotion/react"
  ],
  "faker-ffi": [
   "@faker-js/faker"
  ],
  "nextjs": [
   "next/document",
   "next/head",
   "next/image.js",
   "next/link",
   "next/navigation",
   "next/script",
   "swr"
  ],
  "node-workerbees": [
   "worker_threads"
  ],
  "jsdom": [
   "jsdom"
  ]
 },
 "broken_closure": [
  "ace",
  "address-rfc2821",
  "affjax-node",
  "axon",
  "benchlib",
  "blessed",
  "cbor-stream",
  "chartjs",
  "chartjs-halogen",
  "choku",
  "compile-fail",
  "css-class-name-extractor",
  "csv-stream",
  "debounce",
  "deno",
  "dotenv",
  "droplet",
  "echarts-simple",
  "elmish",
  "elmish-enzyme",
  "elmish-hooks",
  "elmish-html",
  "elmish-testing-library",
  "elmish-time-machine",
  "environment",
  "express",
  "ezfetch",
  "fahrtwind",
  "faker-ffi",
  "fakerjs",
  "framer-motion",
  "gojs",
  "golden-test",
  "golem-fetch",
  "graphql-client",
  "halogen-echarts-simple",
  "halogen-xterm",
  "httpurple",
  "hylograph-d3-kernel",
  "hylograph-simulation",
  "hylograph-simulation-halogen",
  "i18next",
  "idb",
  "ink",
  "jsdom",
  "leveldb",
  "logging-journald",
  "lumi-components",
  "markdown-it-js",
  "marked",
  "meowclient",
  "milkdown",
  "milkis",
  "motion",
  "mysql",
  "n3",
  "nanoid",
  "next-purs-rsc",
  "nextjs",
  "nextui",
  "node-buffer",
  "node-child-process",
  "node-event-emitter",
  "node-execa",
  "node-fs",
  "node-glob-basic",
  "node-http",
  "node-http2",
  "node-human-signals",
  "node-net",
  "node-os",
  "node-path",
  "node-process",
  "node-readline",
  "node-sqlite3",
  "node-stream-pipes",
  "node-streams",
  "node-tls",
  "node-url",
  "node-workerbees",
  "node-zlib",
  "oak",
  "oak-debug",
  "open-mkdirp-aff",
  "optparse",
  "org-doc",
  "postgresql",
  "prospero",
  "psa-utils",
  "react-aria",
  "react-basic-dnd",
  "react-basic-emotion",
  "react-basic-storybook",
  "react-dnd-kit",
  "react-icons",
  "react-markdown",
  "react-testing-library",
  "react-virtuoso",
  "reactix",
  "recharts",
  "rito",
  "rough-notation",
  "simple-ulid",
  "spec-discovery",
  "spec-node",
  "spec-reporter-xunit",
  "supabase",
  "systemd-journald",
  "tanstack-query",
  "toestand",
  "ts-bridge",
  "ulid",
  "url-regex-safe",
  "visx",
  "vitest",
  "webb-commandline",
  "webb-directory",
  "webb-file",
  "webb-test",
  "webextension-polyfill",
  "which",
  "whine-core",
  "xterm",
  "yaml-next",
  "yoga-acp-om",
  "yoga-better-auth",
  "yoga-config",
  "yoga-docker-compose",
  "yoga-dynamodb",
  "yoga-elasticsearch",
  "yoga-fastify",
  "yoga-fastify-om",
  "yoga-fetch",
  "yoga-fetch-om",
  "yoga-heroui",
  "yoga-jaeger",
  "yoga-next-fastify",
  "yoga-om-strom",
  "yoga-om-workerbees",
  "yoga-opentelemetry",
  "yoga-pino",
  "yoga-postgres",
  "yoga-react-native",
  "yoga-redis",
  "yoga-shadcn",
  "yoga-sqlite",
  "yoga-test-docker",
  "z3"
 ],
 "cut": [
  "affjax-node",
  "axon",
  "benchlib",
  "blessed",
  "cbor-stream",
  "compile-fail",
  "css-class-name-extractor",
  "css-frameworks",
  "csv-stream",
  "dotenv",
  "environment",
  "express",
  "ezfetch",
  "framer-motion",
  "golden-test",
  "golem-fetch",
  "graphql-client",
  "httpurple",
  "ink",
  "lumi-components",
  "marked",
  "meowclient",
  "motion",
  "next-purs-rsc",
  "node-buffer",
  "node-child-process",
  "node-event-emitter",
  "node-execa",
  "node-fs",
  "node-glob-basic",
  "node-http",
  "node-http2",
  "node-human-signals",
  "node-net",
  "node-os",
  "node-path",
  "node-process",
  "node-readline",
  "node-stream-pipes",
  "node-streams",
  "node-tls",
  "node-url",
  "node-workerbees",
  "node-zlib",
  "open-mkdirp-aff",
  "optparse",
  "org-doc",
  "pmock",
  "postgresql",
  "prospero",
  "psa-utils",
  "react-basic-dom-beta",
  "react-icons",
  "react-testing-library",
  "rito",
  "simple-ulid",
  "spec",
  "spec-discovery",
  "spec-mocha",
  "spec-node",
  "spec-reporter-xunit",
  "supabase",
  "test-unit",
  "ts-bridge",
  "webb-commandline",
  "webb-directory",
  "webb-file",
  "webb-test",
  "whine-core",
  "yoga-acp-om",
  "yoga-config",
  "yoga-docker-compose",
  "yoga-fastify-om",
  "yoga-fetch-om",
  "yoga-om-strom",
  "yoga-om-workerbees",
  "yoga-opentelemetry",
  "yoga-sqlite",
  "yoga-test-docker"
 ]
}

Scan script (run from staging/ after spago install)

import os, re, json, collections

SHIMMED = {"big-integer", "decimal.js", "react", "react-dom",
           "react-dom/client", "react-dom/server", "uuid"}
PAT = re.compile(
    r'''(?:\bfrom\s*|(?<!\w)import\s*\(\s*|(?<!\w)require\s*\(\s*)['"]([^'"\n]+)['"]''')
SKIP = {'node_modules', 'test', 'tests', '.git', 'examples', 'example', 'bench'}

bare = collections.defaultdict(set)
for pkgdir in sorted(os.listdir('.spago/p')):
    root = os.path.join('.spago/p', pkgdir)
    if not os.path.isdir(root): continue
    name = re.sub(r'-\d+\.\d+\.\d+.*$', '', pkgdir)
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames if d not in SKIP]
        for f in filenames:
            if not f.endswith(('.js', '.mjs', '.cjs')): continue
            try: text = open(os.path.join(dirpath, f), encoding='utf-8', errors='replace').read()
            except OSError: continue
            for m in PAT.finditer(text):
                spec = m.group(1)
                if spec.startswith(('.', '/')) or ' ' in spec or spec.startswith('['): continue
                bare[name].add(spec)

unshimmed = {p: sorted(s - SHIMMED) for p, s in bare.items() if s - SHIMMED}

lock = json.load(open('spago.lock'))
deps = {p: set(v.get('dependencies', [])) for p, v in lock['packages'].items()}
rev = collections.defaultdict(set)
for p, ds in deps.items():
    for d in ds: rev[d].add(p)

broken = set(unshimmed) & set(deps)
frontier = list(broken)
while frontier:
    p = frontier.pop()
    for r in rev[p]:
        if r not in broken:
            broken.add(r); frontier.append(r)

print(json.dumps({"unshimmed": unshimmed, "cannot_run_closure": sorted(broken)}, indent=2))

[pete-murphy]

before we merge, could you measure the trimmed set's peak (maxLive from +RTS -s) on a cold boot (no output)? If it's much above ~2.5 GB, it's probably worth trimming a little further.

maxLive for the current (on this branch) trimmed package set is 2,064 MiB (2,164 MB). If I apply the suggestion from Fable (trim by "cannot execute in the playground" instead of size/dependents), maxLive is down to 1,899 MiB (1,991 MB). I had Fable run some experiments on a branch in my fork. The HTML report of findings: report.html, and a tl;dr chart:

The recent findings / this chart show the warm case (a populated .psci_modules cache) is actually worse memory-wise than cold boot, for reasons that I am not yet fully understanding (Fable says that cold start is able to do sharing that warm start is not able to do.) If that's true, I think that refutes the reasoning behind Option 3, which was based on a bad assumption that wasn't tested. I ran the server manually a couple times, with and without a .psci_modules cache, and was able to reproduce the results that Fable reported, but haven't had a chance to look into it.

I also didn't realize, til looking at these findings, that the full package set is just barely over the limit 🧐 .

production is actually already occasionally being OOM-killed with the current package set

Hmm, that's not great: even the trimmed package set here is bigger than what it currently is. Do you think it would be best to pare it down even further (trim the "cannot run" packages Fable found above)? That would get it to a lower module-count than the currently-deployed package set: 4,482 modules on this branch (trim-current in chart above), 3,769 if trim "cannot run" (trim-cannot-run), and 3,946 in what's currently deployed according to what I measured here.

It seems like good news, that the warm cache is the worst case for max-live heap usage and could seemingly be optimized to behave more like the cold cache. But then, it's a new mystery to me why the server would OOM with the current package set size, I'd expect its memory usage to be somewhere between trim-current and trim-cannot-run (just based on module count as a rough approximation of size).

[thomashoneyman]

OK, I had some time today to run this branch locally and things look good. Walking through my findings so I can remember this as part of merging:

#! /usr/bin/env bash

set -ex
set -o noglob
export XDG_CACHE_HOME="$PWD/.spago-cache"
spago install
# Note: the heap cap below must fit within the droplet's RAM; the package set
# as of psc-0.15.15-20260701 (641 packages) needs more than the 3G that
# sufficed for psc-0.15.13-20231219 (459 packages).
exec trypurescript +RTS -N2 -A128m -M6G -RTS 8081 $(spago sources)

• The droplet itself has legacy spago 0.20.3 and can't read spago.yaml; it has no purs binary, and the new spago won't run anything without PureScript. So one thing we need to do on merge here is update both of those on the machine.
• The droplet has node 18.19.1, but the spago npm package declares engines: { node: '>=22.5.0' }, so I'll also need to update that version which is now very old
• spago install compiles now, but the server doesn't use that output, so we get a useless purs compile on every deploy; we should replace that with just fetching (i.e. spago fetch --pure)

So for this pull request, could you update the deploy script to use spago fetch instead?

export XDG_CACHE_HOME="$PWD/.spago-cache"
spago fetch --pure
exec trypurescript +RTS -N2 -A128m -M3G -RTS 8081 $(spago sources)

...plus I will fix the droplet to upgrade node to ≥22.5, install new spago/purescript, and remove the legacy spago. Will also delete the stale legacy staging/.spago contents on the server.

Other stuff

This branch trimmed thingss down, but we don't have record of the exclusions — the next time someone runs spago install $(spago ls packages --json ...) they'd reinstall the full set right? Should we introduce an exclusion list as an artifact, like staging/excluded-packages.txt with a one-line rationale per package or per-category, and have the RELEASE.md install step filter through it?

$ spago install $(spago ls packages --json --quiet | jq -r 'to_entries[] | select(.value.type == "registry") | .key' | grep -vxF -f excluded-packages.txt)

It would also be useful to add your maxLive measurement method (+RTS -s on a cold boot, the way you measured 2,064 MiB in this thread) to the existing "check the memory" paragraph in RELEASE.md; right now it says that a reviewer should check memory but not how, and this would be useful to retain.

[thomashoneyman]

This note isn't true anymore since the new spago doesn't rely on the metadata package. We can delete this line.

[thomashoneyman]

We've advanced to 77.10.1 if you want to update this

[thomashoneyman]

Everything else in here is pinned to 0.15.15; this works, but should use the same version for consistency?

[thomashoneyman]

Regarding your research -- I see; while I don't know why the numbers diverge necessarily, at least we know that production is basically always in the warm case so that's the number for us to look at.

So yeah:

1. Let's trim the packages that can't run, the one that has the lowest warm-start memory use in your charts
2. Agree on "warm could be optimized to behave like cold", but we could do it in a followup issue
3. Let's make the deploy/start changes from my previous comment

...and otherwise we can merge this, and I'll make the droplet changes when it's deploy time.

[pete-murphy]

So for this pull request, could you update the deploy script to use spago fetch instead?

Addressed here: ff61412

Should we introduce an exclusion list as an artifact, like staging/excluded-packages.txt with a one-line rationale per package or per-category, and have the RELEASE.md install step filter through it?

1. Let's trim the packages that can't run, the one that has the lowest warm-start memory use in your charts

We've advanced to 77.10.1 if you want to update this

Addressed all of these in this commit: bd91097

Note that I just added a staging/excluded-packages.txt containing the list of packages, but no script for reproducing that list. I'm open to including a script, but I'm also hoping that this is a stopgap solution, and the memory management can be improved such that this exclusion list is no longer necessary. But, maybe we should plan on it being more permanent?

Everything else in here is pinned to 0.15.15; this works, but should use the same version for consistency?

37fbf95