From 370731f182b783f3384d6fd706aeedc493b3cf4d Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 05:28:52 -0400 Subject: [PATCH 01/17] docs: spec the package commands and run-scoped views [REPORT-146] Requirements and implementation spec for cc-data's package commands and the two dataset views that replace the spike's _lib queries. The views are run_answers (every answer with each run whose membership holds it) and learner_endpoints (one row per learner per assignment of an answers report, counted by user_id). They reproduce _lib's answer and learner counts on staging data, register empty on any dataset, and materialize. Log freshness needs no view: logs.event_time already exists. package init writes a manifest and a stdlib-only run.py that creates or reuses its Student Answers run and counts through the new views. package run stages exactly what build ships and gives it the runner's scope.json, environment, time bound, display.md cap and result reading. build zips reproducibly and keeps each file's execute bit, which unzip restores on the VM. publish posts the zip to report-server with the cc-data token and checks the recorded checksum. The package rules and applicability are report-server's alone: build and run ask a validate route and an applies route that a companion report-service story adds, with one matcher and the existing profile deriver behind them. Until those routes exist, a 404 is a warning. VM parity moves to REPORT-147. The plan was built and tested in a scratch copy of main before it was written down; the mutations each step's tests catch are listed with it. --- .../implementation.md | 1914 +++++++++++++++++ .../requirements.md | 347 +++ 2 files changed, 2261 insertions(+) create mode 100644 specs/REPORT-146-package-commands-and-dataset-views/implementation.md create mode 100644 specs/REPORT-146-package-commands-and-dataset-views/requirements.md diff --git a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md new file mode 100644 index 0000000..d6800a0 --- /dev/null +++ b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md @@ -0,0 +1,1914 @@ +# Implementation Plan: cc-data-cli: the package commands and the dataset views that replace `_lib` + +**Jira**: https://concord-consortium.atlassian.net/browse/REPORT-146 +**Requirements Spec**: [requirements.md](requirements.md) +**Status**: **In Development** + +## Implementation Plan + +The plan comes in six commits. Each one builds and passes `go test ./...` on its own, in this order. All of the code below was compiled and tested in a scratch copy of `main` (`58c9172`) while this plan was written: the full suite passes, `go vet` is clean for Linux, and `GOOS=windows go vet` is clean for `internal/packages` and `internal/api`. A table under each step names mutations its tests catch; every one listed was applied and made a test fail. The scratch copy has been deleted. + +The package rules and applicability are report-server's (requirements, Dependencies). cc-data calls `POST /api/v1/packages/validate` and `POST /api/v1/packages/applies`, which the companion report-service story adds. A 404 from either means an older report-server, and is a warning, not a failure. So this plan ships and works before that story lands, and gains its checks when it does. + +### The run-scoped views: `run_answers` and `learner_endpoints` + +**Summary**: R1 to R5. Adds two static views to `internal/duck/views.go`, documents them where the drift guards check, and tests the counts on a dataset where each wrong way of counting gives a different number. Static views are derived everywhere else, so `cmd/query.go`'s help and the MCP `query` description pick both up with no edit. + +**Files affected**: +- `internal/duck/views.go`: two builders and a helper, registered after `runMembershipView`. +- `internal/duck/run_views_test.go`: new. +- `internal/guidance/src/core.md`: two Views entries and the freshness query. +- `docs/researcher-guide.md`: two table rows. + +**Estimated diff size**: ~230 lines + +```diff +--- /tmp/o 2026-10-08 05:22:45.429149300 -0400 ++++ internal/duck/views.go 2026-10-07 17:23:24.424211223 -0400 +@@ -8,6 +8,7 @@ + "io" + "os" + "path/filepath" ++ "regexp" + "sort" + "strings" + "time" +@@ -71,6 +72,8 @@ + stmts = append(stmts, vs.storeView(store.TypeAnswers)) + stmts = append(stmts, vs.storeView(store.TypeHistory)) + stmts = append(stmts, vs.runMembershipView()) ++ stmts = append(stmts, vs.runAnswersView()) ++ stmts = append(stmts, vs.learnerEndpointsView()) + stmts = append(stmts, vs.downloadsView()) + stmts = append(stmts, vs.attachmentFilesView()) + stmts = append(stmts, vs.attachmentStatesView()) +@@ -360,6 +363,88 @@ + return viewStmt{name: name, primary: primary, fallback: fallback, files: files} + } + ++// runAnswersView is every answer once per Student Answers run whose membership holds it: the ++// type-qualified membership join, registered on every dataset so a run with no answers counts ++// zero rather than failing to bind the way answers_ does. It must follow both views it ++// reads, and it declares no files, so materializing reads it through theirs. ++func (vs viewSet) runAnswersView() viewStmt { ++ name := vs.prefix + `"run_answers"` ++ answers := vs.prefix + sqlIdent(store.TypeAnswers) ++ membership := vs.prefix + `"run_membership"` ++ primary := fmt.Sprintf("CREATE VIEW %s AS SELECT m.run_id, s.* FROM %s s JOIN %s m USING (source_key, remote_endpoint, question_id) WHERE m.type = %s", ++ name, answers, membership, sqlStr(store.TypeAnswers)) ++ fallback := fmt.Sprintf("CREATE VIEW %s AS SELECT CAST(NULL AS BIGINT) AS run_id, s.* FROM %s s WHERE false", name, answers) ++ return viewStmt{name: name, primary: primary, fallback: fallback} ++} ++ ++// resEndpointColumn is an answers report's per-assignment endpoint column; N is the ++// assignment's position in that run's list. ++var resEndpointColumn = regexp.MustCompile(`^res_([0-9]+)_remote_endpoint$`) ++ ++// learnerEndpointsView is one row per learner per assignment of every answers-type report CSV: ++// the Portal user, the assignment's offering and the endpoint that joins them to run_answers. ++// A student has one endpoint per assignment, which is why learners are counted by user_id. ++// Each CSV is scanned once, its assignment columns unnested as structs, because a real class ++// carries eight to fourteen of them. ++func (vs viewSet) learnerEndpointsView() viewStmt { ++ name := vs.prefix + `"learner_endpoints"` ++ standIn := fmt.Sprintf("CREATE VIEW %s AS SELECT CAST(NULL AS BIGINT) AS run_id, CAST(NULL AS BIGINT) AS user_id, CAST(NULL AS BIGINT) AS offering_id, CAST(NULL AS VARCHAR) AS remote_endpoint WHERE false", name) ++ var members, files []string ++ for _, dl := range vs.m.Downloads { ++ if dl.Type != "report" || dl.ReportType != dataset.ReportTypeAnswers || len(dl.Files) == 0 || dl.Columns == nil { ++ continue ++ } ++ structs := learnerStructs(dl) ++ if _, ok := dl.Columns["user_id"]; !ok || len(structs) == 0 { ++ continue ++ } ++ if fileMissing(filepath.Join(vs.canonDir, dl.Files[0])) { ++ vs.warnf("report CSV %s is missing on disk; contributing zero rows to learner_endpoints (run cc-data dataset reindex)", dl.Files[0]) ++ continue ++ } ++ members = append(members, fmt.Sprintf( ++ "SELECT CAST(run_id AS BIGINT) AS run_id, TRY_CAST(user_id AS BIGINT) AS user_id, unnest([%s]) AS a FROM (%s)", ++ strings.Join(structs, ", "), vs.csvScan(dl, true, nil))) ++ files = append(files, dl.Files...) ++ } ++ if len(members) == 0 { ++ return viewStmt{name: name, primary: standIn, fallback: standIn} ++ } ++ // A bare endpoint ending in "/" is a learner with no secure key, shared by every such ++ // learner, so it is withheld as in the dimension views rather than joined. ++ primary := fmt.Sprintf("CREATE VIEW %s AS SELECT run_id, user_id, a.offering_id AS offering_id, "+ ++ "CASE WHEN a.remote_endpoint LIKE %s THEN NULL ELSE a.remote_endpoint END AS remote_endpoint FROM (\n%s\n) WHERE a.remote_endpoint IS NOT NULL", ++ name, sqlStr("%/"), strings.Join(members, "\nUNION ALL\n")) ++ return viewStmt{name: name, primary: primary, fallback: standIn, files: files} ++} ++ ++// learnerStructs is one {offering_id, remote_endpoint} struct per assignment the CSV records, ++// in assignment order. A CSV without an assignment's offering_id gives that struct a NULL one. ++func learnerStructs(dl dataset.Download) []string { ++ type res struct { ++ n int ++ col string ++ } ++ var found []res ++ for col := range dl.Columns { ++ if m := resEndpointColumn.FindStringSubmatch(col); m != nil { ++ var n int ++ fmt.Sscanf(m[1], "%d", &n) ++ found = append(found, res{n, col}) ++ } ++ } ++ sort.Slice(found, func(i, j int) bool { return found[i].n < found[j].n }) ++ structs := make([]string, 0, len(found)) ++ for _, r := range found { ++ offering := "CAST(NULL AS BIGINT)" ++ if col := fmt.Sprintf("res_%d_offering_id", r.n); dl.Columns[col] != "" { ++ offering = fmt.Sprintf("TRY_CAST(%s AS BIGINT)", sqlIdent(col)) ++ } ++ structs = append(structs, fmt.Sprintf("{'offering_id': %s, 'remote_endpoint': CAST(%s AS VARCHAR)}", offering, sqlIdent(r.col))) ++ } ++ return structs ++} ++ + // downloadsView is a VALUES dimension table from the manifest. + // + // hide_names separates the two meanings of a name column: where a run hid names, student_name +``` + +`learner_endpoints` scans each CSV once and unnests one `{offering_id, remote_endpoint}` struct per assignment. Scanning once per assignment column would read a real class's CSV eight to fourteen times. The view declares its CSVs as files, so `dataset materialize` writes it. `run_answers` declares none, so materializing reads it through the materialized `answers` and `run_membership`. Both behaviors were run against a copy of `staging-smoketest`. + +```go +package duck + +import ( + "encoding/json" + "fmt" + "testing" + "time" + + "github.com/concord-consortium/cc-data-cli/internal/store" +) + +// Each wrong way to count gives a different number here: counting endpoints instead of users +// (learner 1 answered in two assignments), joining the bare-endpoint learner (who would +// borrow learner 4's answer), counting an answer once instead of once per run (e3/q1 is in +// runs 584 and 585), leaking class 2's run into class 1's, and joining membership untyped +// (run 584's history holds e1/q1 too). +func TestRunAnswersAndLearnerEndpointsCountLikeLib(t *testing.T) { + d := newDS(t, "ds") + buildStore(t, d, 584, [][]byte{ + answerRec("s", "https://p/d/e1", "q1", "a"), answerRec("s", "https://p/d/e1b", "q1", "b"), + answerRec("s", "https://p/d/e3", "q1", "c"), answerRec("s", "https://p/d/", "q1", "bare"), + }) + hist, _ := json.Marshal(map[string]any{"source_key": "s", "remote_endpoint": "https://p/d/e1", "question_id": "q1", "history_id": "h1"}) + seg := store.OpenSegment(d.Dir, store.TypeHistory, 584) + if err := seg.AppendPage([][]byte{hist}, time.Unix(584, 0).UTC(), 584); err != nil { + t.Fatal(err) + } + seg.WriteCursor(&store.Cursor{Items: 1}) + if _, err := d.MergeCompact(store.TypeHistory, 584, seg); err != nil { + t.Fatal(err) + } + buildStore(t, d, 585, [][]byte{answerRec("s", "https://p/d/e3", "q1", "c"), answerRec("s", "https://p/d/e9", "q1", "z")}) + addReportCSV(t, d, 584, "answers", "student_id,user_id,class_id,res_1_offering_id,res_1_remote_endpoint,res_2_offering_id,res_2_remote_endpoint\n"+ + "Prompt,,,,,,\nCorrect answer,,,,,,\n"+ + "10,1,1,7,https://p/d/e1,8,https://p/d/e1b\n"+ + "30,3,1,7,https://p/d/e3,8,\n"+ + "40,4,1,7,https://p/d/,8,\n") + addReportCSV(t, d, 585, "answers", "student_id,user_id,class_id,res_1_offering_id,res_1_remote_endpoint\n"+ + "Prompt,,,,\nCorrect answer,,,,\n30,3,2,9,https://p/d/e3\n90,9,2,9,https://p/d/e9\n") + e := openEngine(t, []DatasetSpec{{DS: d}}, nil) + + if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 584"); n != 4 { + t.Errorf("run 584 answers = %d, want 4", n) + } + if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 585"); n != 2 { + t.Errorf("run 585 answers = %d, want 2", n) + } + learners := "SELECT count(DISTINCT e.user_id) FROM learner_endpoints e JOIN run_answers a USING (run_id, remote_endpoint) WHERE run_id = %d" + if n := queryInt(t, e, fmt.Sprintf(learners, 584)); n != 2 { + t.Errorf("run 584 learners = %d, want 2 (users 1 and 3; user 4's endpoint is bare)", n) + } + if n := queryInt(t, e, fmt.Sprintf(learners, 585)); n != 2 { + t.Errorf("run 585 learners = %d, want 2", n) + } + if n := queryInt(t, e, "SELECT count(*) FROM learner_endpoints WHERE remote_endpoint IS NULL AND user_id = 4"); n != 1 { + t.Errorf("the bare endpoint was not nulled") + } + if n := queryInt(t, e, "SELECT count(*) FROM learner_endpoints WHERE run_id = 584 AND offering_id = 8"); n != 1 { + t.Errorf("assignment 2's offering_id did not follow its endpoint") + } + var types string + rows, _ := e.Query(t.Context(), "SELECT string_agg(column_type, ',' ORDER BY column_name) FROM (DESCRIBE learner_endpoints)") + rows.Next() + rows.Scan(&types) + rows.Close() + if types != "BIGINT,VARCHAR,BIGINT,BIGINT" { + t.Errorf("learner_endpoints types (offering_id, remote_endpoint, run_id, user_id) = %s", types) + } +} + +func TestRunViewsAreEmptyNotMissingOnAFreshDataset(t *testing.T) { + e := openEngine(t, []DatasetSpec{{DS: newDS(t, "ds")}}, nil) + for _, v := range []string{"run_answers", "learner_endpoints"} { + if n := queryInt(t, e, "SELECT count(*) FROM "+v); n != 0 { + t.Errorf("%s has %d rows", v, n) + } + } + if n := queryInt(t, e, "SELECT count(*) FROM (DESCRIBE learner_endpoints)"); n != 4 { + t.Errorf("learner_endpoints stand-in has %d columns, want 4", n) + } +} +``` + +| Mutation | Test that fails | +|---|---| +| bare endpoints not nulled | learners for run 584 = 3 | +| `m.type = 'answers'` dropped | run 584 answers = 5 (the history membership joins) | +| offering taken from assignment 1 | "assignment 2's offering_id did not follow its endpoint" | +| `run_id` left uncast | the DESCRIBE type check (INTEGER) | + +Documentation (the two guards in `internal/guidance/guard_test.go` fail until both files carry the names; the dashes in `core.md` match that list's existing entries): + +```diff +--- /tmp/o 2026-10-08 05:22:45.441149321 -0400 ++++ internal/guidance/src/core.md 2026-10-07 17:21:28.050911252 -0400 +@@ -122,6 +122,23 @@ + **type-qualified**: `answers a JOIN run_membership m USING + (source_key, remote_endpoint, question_id) WHERE m.run_id = 584 AND m.type = + 'answers'`. History joins add `history_id` to the USING list. ++- `run_answers` — every `answers` row with `run_id`, once for each Student Answers ++ run whose membership holds it, so one run's answers are `SELECT count(*) FROM ++ run_answers WHERE run_id = 584`. It is the type-qualified membership join above, ++ always present, so a run with no answers counts zero rather than failing to bind ++ as `answers_` does. Its `run_id` is membership; the store's own `_run_id` is ++ only the run that last fetched the record, so filtering `answers` by `_run_id` ++ undercounts a run whose answers a later run re-fetched. ++- `learner_endpoints` — one row per learner per assignment in each answers-type ++ report CSV: `run_id`, `user_id` (the Portal user), `offering_id` and ++ `remote_endpoint`, built from every `res__remote_endpoint` column. A bare ++ endpoint ending in `/` (a learner with no secure key) is NULL, as in ++ `student_id_mapping`. Count a run's learners with an answer by `user_id`, never by ++ endpoint, since a student has one endpoint per assignment: `SELECT count(DISTINCT ++ e.user_id) FROM learner_endpoints e JOIN run_answers a USING (run_id, ++ remote_endpoint) WHERE run_id = 584`. ++- A run's log freshness is `SELECT count(*) AS logs, max(event_time) AS ++ log_freshness_at FROM logs WHERE run_id = `. + - Reports-to-stores join: `reports.res__remote_endpoint = + answers.remote_endpoint`, with `res___*` pairing to + `answers.question_id`. +--- /tmp/o 2026-10-08 05:22:45.444149326 -0400 ++++ docs/researcher-guide.md 2026-10-07 17:21:28.050652765 -0400 +@@ -346,6 +346,8 @@ + | `attachment_content` | The text/JSON content of every saved CODAP/SageModeler snapshot, queryable and diffable. | + | `student_id_mapping` | One row per learner from Student ID Mapping runs, with the ids that join them to their answers and history. | + | `student_metadata` | One row per learner from Student Metadata runs: name, username, class, school, teachers, permission forms. | ++| `run_answers` | Every answer with the `run_id` of each run that holds it: `SELECT count(*) FROM run_answers WHERE run_id = 584` counts one run's answers. | ++| `learner_endpoints` | One row per learner per assignment in each Student Answers report: `user_id`, `offering_id` and the `remote_endpoint` that joins them to `run_answers`. Count a run's learners by `user_id`, since one student has an endpoint per assignment. | + | `run_membership`, `downloads` | Provenance: which run's fetch covered which records, and what each download was, including whether its run hid names. | + + ### Speeding up a large dataset +``` + +--- + +### A package's files: the manifest fields, collecting and zipping, the result + +**Summary**: R6 (cc-data's part), R8, R14, R17, R18. Adds `internal/packages`: +- **Reading the manifest.** It reads only the fields the runner acts on. +- **Collecting the package.** It gathers the file set `build` ships and `run` stages, each file with the mode it ships with, so an executable entrypoint stays executable after `unzip`. +- **Zipping.** It writes a reproducible zip. +- **Reading the result.** It reads `display.md`, `summary.txt` and `counts.json` as the runner does. + +Links are checked with `Lstat` then `os.SameFile` rather than `O_NOFOLLOW`, which Windows (CI only) lacks. The package has no callers yet and is independently reviewable. + +**Files affected**: +- `internal/packages/manifest.go`: new. +- `internal/packages/archive.go`: new. +- `internal/packages/result.go`: new. +- `internal/packages/files_test.go`: new; holds the shared helpers `goodManifest` and `writeTree`. + +**Estimated diff size**: ~430 lines + +```go +// Package packages is the local side of a Researcher Dashboard package: reading the manifest +// fields the runner acts on, collecting and zipping the package, staging and running it the +// way the runner does, and reading the result files. The package rules themselves, and +// applicability, are report-server's, which cc-data asks rather than copies. +package packages + +import ( + "bytes" + "encoding/json" + "fmt" + "regexp" + "strings" + "unicode" +) + +var nameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,62}$`) + +// ValidName reports whether name fits the catalog's name grammar, which init needs before +// report-server has seen anything. +func ValidName(name string) bool { return nameRe.MatchString(name) } + +// URLs is a manifest's applicability patterns, passed to report-server as written. +type URLs struct { + All []string `json:"all"` + Any []string `json:"any"` + None []string `json:"none"` +} + +// DeclaresPatterns reports whether any sense holds a pattern; a package that declares none +// applies to every scope. +func (u URLs) DeclaresPatterns() bool { + return len(u.All)+len(u.Any)+len(u.None) > 0 +} + +// Manifest is the part of manifest.json the runner acts on. Everything else, and every limit, +// is report-server's to check (cmd's validate call). +type Manifest struct { + Name string `json:"name"` + Title string `json:"title"` + Version string `json:"version"` + Description string `json:"description,omitempty"` + URLs URLs `json:"urls"` + CluePrepull bool `json:"clue_prepull"` + Entrypoint string `json:"entrypoint"` + ExpectedDurationSeconds int `json:"expected_duration_seconds"` +} + +// ReadManifest reads the fields package run needs to stage and start a package: the +// entrypoint, which must be one of files, the duration that sets the time bound, the patterns +// and clue_prepull. A wrongly typed field is refused here because the run cannot proceed +// without it; the full rules are report-server's. +func ReadManifest(data []byte, files map[string]bool) (Manifest, error) { + var m Manifest + if err := json.Unmarshal(data, &m); err != nil { + return m, fmt.Errorf("manifest.json: %v", err) + } + var raw map[string]json.RawMessage + json.Unmarshal(data, &raw) + if !files[m.Entrypoint] || UnsafePath(m.Entrypoint) { + return m, fmt.Errorf("manifest.json: entrypoint %q is not a file in the package", m.Entrypoint) + } + if v := raw["expected_duration_seconds"]; len(v) == 0 || v[0] == '"' || bytes.ContainsAny(v, ".eE") || m.ExpectedDurationSeconds < 1 { + return m, fmt.Errorf("manifest.json: expected_duration_seconds must be a positive integer") + } + return m, nil +} + +// UnsafePath is report-server's Archive.unsafe_path?: absolute, a drive letter, or a ".." segment. +func UnsafePath(name string) bool { + if strings.HasPrefix(name, "/") || strings.HasPrefix(name, `\`) { + return true + } + if len(name) >= 2 && name[1] == ':' && unicode.IsLetter(rune(name[0])) { + return true + } + for _, seg := range strings.FieldsFunc(name, func(r rune) bool { return r == '/' || r == '\\' }) { + if seg == ".." { + return true + } + } + return false +} +``` +```go +package packages + +import ( + "archive/zip" + "bytes" + "crypto/sha256" + "encoding/hex" + "fmt" + "io" + "io/fs" + "os" + "path" + "path/filepath" + "runtime" + "sort" + "strings" + "time" +) + +// buildTime is every entry's modification time, so the same tree always zips to the same +// bytes and the same checksum. +var buildTime = time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC) + +// Files is a package's contents: slash-separated paths relative to its directory, sorted, +// each with the mode it ships with. +type Files struct { + Dir string + Paths []string + Modes map[string]os.FileMode + Bytes int64 +} + +// shipMode is the one mode a file keeps through build and unzip: 0755 when any execute bit is +// set, else 0644. unzip restores it on the VM, where a non-.py entrypoint is executed directly. +func shipMode(m fs.FileMode) os.FileMode { + if m.Perm()&0o111 != 0 { + return 0o755 + } + return 0o644 +} + +// Set is the paths as a lookup, the shape ReadManifest checks the entrypoint against. +func (f Files) Set() map[string]bool { + set := make(map[string]bool, len(f.Paths)) + for _, p := range f.Paths { + set[p] = true + } + return set +} + +// excluded is what a build leaves behind: anything under a segment starting with ".", +// which covers this tool's own .cc-data-run and .cc-data-build as well as .git, and the +// three things cc-data-studies ignores. +func excluded(rel string, isDir bool) bool { + base := path.Base(rel) + if strings.HasPrefix(base, ".") || base == "__pycache__" || (isDir && base == "local-data") { + return true + } + return !isDir && strings.HasSuffix(base, ".pyc") +} + +// Collect walks dir for the files a package is made of. A symbolic link or any other +// non-regular file is refused rather than skipped, since the runner refuses an archive +// holding one and a link silently dropped would be a package that differs from its tree. +func Collect(dir string) (Files, error) { + files := Files{Dir: dir, Modes: map[string]os.FileMode{}} + err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if p == dir { + return nil + } + rel, err := filepath.Rel(dir, p) + if err != nil { + return err + } + rel = filepath.ToSlash(rel) + if excluded(rel, d.IsDir()) { + if d.IsDir() { + return filepath.SkipDir + } + return nil + } + switch { + case d.Type()&fs.ModeSymlink != 0: + return fmt.Errorf("%s is a symbolic link; a package may hold only regular files", rel) + case d.IsDir(): + return nil + case !d.Type().IsRegular(): + return fmt.Errorf("%s is not a regular file", rel) + } + info, err := d.Info() + if err != nil { + return err + } + files.Paths = append(files.Paths, rel) + files.Modes[rel] = shipMode(info.Mode()) + files.Bytes += info.Size() + return nil + }) + sort.Strings(files.Paths) + return files, err +} + +// Zip writes the files as a reproducible archive: sorted entries, Deflate, a fixed time, each +// file's ship mode, and no extra fields. The size limits are report-server's to apply. +func Zip(files Files) ([]byte, error) { + var buf bytes.Buffer + w := zip.NewWriter(&buf) + for _, rel := range files.Paths { + h := &zip.FileHeader{Name: rel, Method: zip.Deflate, Modified: buildTime} + h.SetMode(files.Modes[rel]) + dst, err := w.CreateHeader(h) + if err != nil { + return nil, err + } + src, err := os.Open(filepath.Join(files.Dir, filepath.FromSlash(rel))) + if err != nil { + return nil, err + } + _, err = io.Copy(dst, src) + src.Close() + if err != nil { + return nil, err + } + } + if err := w.Close(); err != nil { + return nil, err + } + return buf.Bytes(), nil +} + +// CheckEntrypointMode refuses a non-.py entrypoint that would not be executable once unzipped, +// since the runner executes such an entrypoint directly. Windows records no execute bit, so +// there the check cannot be made. +func (f Files) CheckEntrypointMode(entrypoint string) error { + if runtime.GOOS == "windows" || strings.HasSuffix(entrypoint, ".py") || f.Modes[entrypoint]&0o111 != 0 { + return nil + } + return fmt.Errorf("the entrypoint %s is not executable; the runner executes a non-.py entrypoint directly (chmod +x it)", entrypoint) +} + +// Checksum is the catalog's spelling of an archive's digest. +func Checksum(b []byte) string { + sum := sha256.Sum256(b) + return "sha256:" + hex.EncodeToString(sum[:]) +} +``` +```go +package packages + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "io/fs" + "os" + "path/filepath" + "strings" +) + +// DisplayLimitBytes is the cap on display.md, counted in UTF-8 bytes. final-design.md +// section 10 is the only place this number is decided; the runner and this constant both +// cite it, because a local cap looser than the runner's is a check that passes here and a +// VM that then refuses. +const DisplayLimitBytes = 64 * 1024 + +// countKeys are the counts.json keys the runner reads; anything else a package writes there +// is ignored, as it is on the VM. +var countKeys = []string{"answers", "logs", "log_freshness_at"} + +// Result is what the runner keeps of a run: the display, the summary line and the counts. +type Result struct { + DisplayPath string `json:"display"` + DisplayBytes int64 `json:"display_bytes"` + Summary *string `json:"summary"` + Counts map[string]any `json:"counts"` +} + +// OutputRefused is a result the runner would refuse: no display.md, one that is a link or +// not a file, or one over the cap. +type OutputRefused struct{ Reason string } + +func (e *OutputRefused) Error() string { return e.Reason } + +// ReadResult reads the output directory as the runner does, following no link: the +// directory must be itself, and each file must be a regular file. +func ReadResult(outDir string) (Result, error) { + var res Result + if err := RealDir(outDir); err != nil { + return res, &OutputRefused{Reason: "the package replaced its output directory with a link"} + } + _, size, err := readOwnFile(outDir, "display.md", DisplayLimitBytes) + switch { + case errors.Is(err, fs.ErrNotExist): + return res, &OutputRefused{Reason: "the package wrote no display.md"} + case err != nil: + return res, err + case size > DisplayLimitBytes: + return res, &OutputRefused{Reason: fmt.Sprintf("the package's display.md is %d bytes, over the %d-byte limit", size, DisplayLimitBytes)} + } + res.DisplayPath = filepath.Join(outDir, "display.md") + res.DisplayBytes = size + if text, _, err := readOwnFile(outDir, "summary.txt", DisplayLimitBytes); err == nil { + line := strings.TrimSpace(strings.SplitN(strings.ReplaceAll(text, "\r\n", "\n"), "\n", 2)[0]) + if line != "" { + res.Summary = &line + } + } else if !errors.Is(err, fs.ErrNotExist) { + return res, err + } + res.Counts = map[string]any{} + if text, _, err := readOwnFile(outDir, "counts.json", DisplayLimitBytes); err == nil { + var parsed map[string]any + if json.Unmarshal([]byte(text), &parsed) == nil { + for _, k := range countKeys { + if v, ok := parsed[k]; ok { + res.Counts[k] = v + } + } + } + } + return res, nil +} + +// readOwnFile returns up to limit+1 bytes of a regular, unlinked file and its full size. The +// Lstat-then-SameFile pair stands in for O_NOFOLLOW, which Windows lacks. +func readOwnFile(dir, name string, limit int64) (string, int64, error) { + p := filepath.Join(dir, name) + before, err := os.Lstat(p) + if err != nil { + return "", 0, err + } + if before.Mode()&fs.ModeSymlink != 0 { + return "", 0, &OutputRefused{Reason: fmt.Sprintf("the package's %s is a link, not a file", name)} + } + if !before.Mode().IsRegular() { + return "", 0, &OutputRefused{Reason: fmt.Sprintf("the package's %s is not a file", name)} + } + f, err := os.Open(p) + if err != nil { + return "", 0, err + } + defer f.Close() + after, err := f.Stat() + if err != nil || !os.SameFile(before, after) { + return "", 0, &OutputRefused{Reason: fmt.Sprintf("the package's %s changed while it was read", name)} + } + b, err := io.ReadAll(io.LimitReader(f, limit+1)) + return string(b), after.Size(), err +} + +// RealDir reports an error unless p is a directory reached without following a link. +func RealDir(p string) error { + info, err := os.Lstat(p) + if err != nil { + return err + } + if info.Mode()&fs.ModeSymlink != 0 || !info.IsDir() { + return fmt.Errorf("%s is a link or not a directory", p) + } + return nil +} +``` + +Tests (the link and mode tests skip on Windows, where links need privileges and files have no execute bit): +- **`TestReadManifestNeedsWhatTheRunActsOn`**: a missing entrypoint is refused, and so are `"300"`, `1.5` and `0` for the duration. +- **`TestCollectLeavesOutWhatBuildMustNotShip`**: the tree holds `.git/`, `.cc-data-run/`, `.cc-data-build/`, `__pycache__/cache.txt`, `lib/y.pyc`, `local-data/` and `.DS_Store`. Exactly `lib/q.sql,manifest.json,run.py` is collected. `__pycache__` holds a non-`.pyc` file so the directory rule is tested on its own. +- **`TestCollectRefusesALink`**. +- **`TestZipIsReproducible`**: a touched mtime leaves the checksum unchanged. +- **`TestModesSurviveBuildAndStaging`**: `run.py` (0755 on disk) zips as 0755 and a 0600 `manifest.json` as 0644. A 0644 `run.sh` entrypoint fails `CheckEntrypointMode`, and a 0644 `run.py` passes it. +- **`TestDisplayCapEdges`**: 65,536 bytes pass and 65,537 are refused, and the constant equals 65,536. +- **`TestResultReadsOnlyTheRunnersKeys`**: the summary is the first line, trimmed, and `learners` in `counts.json` is dropped. +- **`TestResultRefusesALinkedDisplay`**. + +| Mutation | Test that fails | +|---|---| +| `size >= DisplayLimitBytes` | `TestDisplayCapEdges` | +| `__pycache__` not excluded | `TestCollectLeavesOutWhatBuildMustNotShip` | +| every entry zipped 0644 | `TestModesSurviveBuildAndStaging` | +| a quoted duration accepted | `TestReadManifestNeedsWhatTheRunActsOn` | + +--- + +### `package run`'s engine: scope, layout, environment, execution + +**Summary**: R10 to R16. +- **Scope.** Parses the author's scope file. +- **Layout.** Rebuilds `.cc-data-run/` (`pkg/`, `in/`, `out/`, `data/` and a `.gitignore` of `*`) and writes the runner's `scope.json`. +- **Applicability.** Asks an injected `Applies` function, which the commands step wires to report-server. +- **Environment and interpreter.** Builds the runner's environment plus the passthrough list, and picks the interpreter. +- **Execution.** Runs the entrypoint from the staged copy in its own process group, and kills the group at the bound. +- **Result.** Reads it with `ReadResult`. + +**Files affected**: +- `internal/packages/scope.go`: new. +- `internal/packages/run.go`: new. +- `internal/packages/proc_unix.go` and `proc_windows.go`: new. +- `internal/packages/run_test.go`: new; holds `scopeJSON`, `runOpts` and `shellPackage`. + +**Estimated diff size**: ~520 lines + +```go +package packages + +import ( + "bytes" + "encoding/json" + "fmt" + "regexp" +) + +var classHashRe = regexp.MustCompile(`^[0-9a-f]{48}$`) + +// Class and Assignment are scope.json's entries (final-design.md section 10). +type Class struct { + ClassHash string `json:"class_hash"` + ClassID int64 `json:"class_id"` +} + +type Assignment struct { + OfferingID int64 `json:"offering_id"` + RunnableID int64 `json:"runnable_id"` + Name *string `json:"name"` + URL string `json:"url"` +} + +// LocalScope is the four keys rigse supplies on the VM, which an author writes for a laptop. +type LocalScope struct { + Kind string `json:"kind"` + ID string `json:"id"` + Classes []Class `json:"classes"` + Assignments []Assignment `json:"assignments"` +} + +// ScopeFile is scope.json as the runner writes it. ClueSource is always "firebase" today, the +// runner's constant; Dataset is the bare dataset name, as the runner writes pkg-, while +// RD_DATASET carries the full ref. +type ScopeFile struct { + LocalScope + ClueSource string `json:"clue_source"` + Dataset string `json:"dataset"` + OutputDir string `json:"output_dir"` +} + +// MarshalJSON keeps the runner's key order, so a diff between a local and a VM scope.json is +// only the values. +func (s ScopeFile) MarshalJSON() ([]byte, error) { + type ordered struct { + Kind string `json:"kind"` + ID string `json:"id"` + Classes []Class `json:"classes"` + Assignments []Assignment `json:"assignments"` + ClueSource string `json:"clue_source"` + Dataset string `json:"dataset"` + OutputDir string `json:"output_dir"` + } + return json.Marshal(ordered{s.Kind, s.ID, s.Classes, s.Assignments, s.ClueSource, s.Dataset, s.OutputDir}) +} + +// ParseLocalScope reads an author's scope file, refusing any key the four do not name (the +// other three are the runner's to set) and any shape the runner would never send. +func ParseLocalScope(data []byte) (LocalScope, error) { + var s LocalScope + dec := json.NewDecoder(bytes.NewReader(data)) + dec.DisallowUnknownFields() + if err := dec.Decode(&s); err != nil { + return s, fmt.Errorf("scope file: %v (it holds exactly kind, id, classes and assignments)", err) + } + switch { + case s.Kind == "": + return s, fmt.Errorf("scope file: kind is required") + case s.ID == "": + return s, fmt.Errorf("scope file: id is required") + case len(s.Classes) == 0: + return s, fmt.Errorf("scope file: classes must name at least one class") + case s.Assignments == nil: + return s, fmt.Errorf("scope file: assignments is required (an empty list is allowed)") + } + for i, c := range s.Classes { + if !classHashRe.MatchString(c.ClassHash) || c.ClassID < 1 { + return s, fmt.Errorf("scope file: classes[%d] needs a 48-hex class_hash and a positive class_id", i) + } + } + for i, a := range s.Assignments { + if a.OfferingID < 1 || a.RunnableID < 1 { + return s, fmt.Errorf("scope file: assignments[%d] needs a positive offering_id and runnable_id", i) + } + } + return s, nil +} + +// AssignmentURLs is the scope's half of the URLs a package's patterns are matched against. +func (s LocalScope) AssignmentURLs() []string { + var urls []string + for _, a := range s.Assignments { + if a.URL != "" { + urls = append(urls, a.URL) + } + } + return urls +} +``` +```go +package packages + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "sort" + "strings" + "time" +) + +// TimeoutMargin is what the runner adds to expected_duration_seconds before it kills a run. +const TimeoutMargin = 10 * time.Minute + +// RunDirName is the working tree package run keeps inside the package directory. It starts +// with "." so Collect, and so build, never ships it. +const RunDirName = ".cc-data-run" + +// passthrough is what a laptop adds to the runner's environment so the package's own cc-data +// calls find the researcher's stored credential (keychain or credentials file) and run at +// all. The runner sets nothing beyond its own variables, so neither does this. +var passthrough = []string{ + "HOME", "PATH", "USER", "LOGNAME", "LANG", "TZ", "TMPDIR", + "DBUS_SESSION_BUS_ADDRESS", "XDG_RUNTIME_DIR", + // Windows has no release, but CI builds and tests there. + "SYSTEMROOT", "USERPROFILE", "APPDATA", "LOCALAPPDATA", "PATHEXT", "COMSPEC", +} + +// Verdict is report-server's answer to whether a package's patterns match a scope. Unconfirmed +// means the report-server answering has no applicability route yet, so nothing was checked. +type Verdict struct { + Applies bool + Reason string + Unconfirmed bool +} + +// Refused is a package the runner would not start: it does not apply, or it needs the CLUE +// pre-pull, which cc-data cannot do. +type Refused struct{ Reason string } + +func (e *Refused) Error() string { return e.Reason } + +// Failed is a package that started and did not finish cleanly. +type Failed struct{ Reason string } + +func (e *Failed) Error() string { return e.Reason } + +// RunOptions is everything package run decides before the entrypoint starts. +type RunOptions struct { + Dir string // the package directory + Manifest Manifest // validated against Files + Files Files // what build would ship + Scope LocalScope // the author's scope file + Applies func(context.Context, URLs) (Verdict, error) // report-server's applicability answer + Dataset string // the full ref, / + Name string // the ref's bare name + Portal string // the ref's portal host + DataRoot string // CC_DATA_ROOT as package run resolved it + ReportSrv string // the stored credential's server, or "" + Stderr io.Writer // where the package's own output goes + Environ func() []string + LookPath func(string) (string, error) + Timeout time.Duration // zero means expected_duration_seconds + TimeoutMargin +} + +// Run prepares the runner's layout, runs the entrypoint and reads its result. +func Run(ctx context.Context, o RunOptions) (Result, error) { + if o.Manifest.CluePrepull { + return Result{}, &Refused{Reason: "the package sets clue_prepull, and cc-data cannot fetch CLUE data yet (final-design.md section 13), so it runs only on the VM"} + } + if o.Manifest.URLs.DeclaresPatterns() { + v, err := o.Applies(ctx, o.Manifest.URLs) + switch { + case err != nil: + return Result{}, err + case v.Unconfirmed: + fmt.Fprintln(o.Stderr, "warning: report-server cannot check applicability yet; the package runs, and the VM's runner is the check") + case !v.Applies: + return Result{}, &Refused{Reason: "the package does not apply to this class: " + v.Reason} + } + } + run := filepath.Join(o.Dir, RunDirName) + paths, err := prepare(run, o.Files) + if err != nil { + return Result{}, err + } + scope := ScopeFile{LocalScope: o.Scope, ClueSource: "firebase", Dataset: o.Name, OutputDir: paths.out} + b, err := json.MarshalIndent(scope, "", " ") + if err != nil { + return Result{}, err + } + scopePath := filepath.Join(paths.in, "scope.json") + if err := os.WriteFile(scopePath, b, 0o644); err != nil { + return Result{}, err + } + + env := map[string]string{ + "CC_DATA_ROOT": o.DataRoot, + "CC_DATA_PORTAL": o.Portal, + "CC_DATA_LOCAL": paths.data, + "RD_DATASET": o.Dataset, + "RD_SCOPE_FILE": scopePath, + "RD_DATA_DIR": paths.data, + "RD_OUTPUT_DIR": paths.out, + } + if o.ReportSrv != "" { + env["RD_REPORT_SERVER_URL"] = o.ReportSrv + } + command, args, note, err := entrypointCommand(paths.pkg, o.Manifest.Entrypoint, o.LookPath) + if err != nil { + return Result{}, err + } + if note != "" { + fmt.Fprintln(o.Stderr, note) + } + + timeout := o.Timeout + if timeout == 0 { + timeout = time.Duration(o.Manifest.ExpectedDurationSeconds)*time.Second + TimeoutMargin + } + runCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + cmd := exec.CommandContext(runCtx, command, args...) + cmd.Dir = paths.pkg + cmd.Env = environment(o.Environ(), env) + cmd.Stdout = o.Stderr + cmd.Stderr = o.Stderr + killGroup(cmd) + if err := cmd.Run(); err != nil { + if errors.Is(runCtx.Err(), context.DeadlineExceeded) { + return Result{}, &Failed{Reason: fmt.Sprintf("the package ran past its %ds bound", int(timeout.Seconds()))} + } + return Result{}, &Failed{Reason: fmt.Sprintf("the package exited unsuccessfully: %v", err)} + } + return ReadResult(paths.out) +} + +type runPaths struct{ pkg, in, out, data string } + +// prepare empties and rebuilds the run tree. Every directory it removes or creates under the +// package must be a real directory, so a link left there can never aim the removal outside. +func prepare(run string, files Files) (runPaths, error) { + p := runPaths{ + pkg: filepath.Join(run, "pkg"), + in: filepath.Join(run, "in"), + out: filepath.Join(run, "out"), + data: filepath.Join(run, "data"), + } + if err := ensureRealDir(run); err != nil { + return p, err + } + if err := os.WriteFile(filepath.Join(run, ".gitignore"), []byte("*\n"), 0o644); err != nil { + return p, err + } + for _, d := range []string{p.pkg, p.in, p.out} { + if _, err := os.Lstat(d); err == nil { + if err := RealDir(d); err != nil { + return p, fmt.Errorf("refusing to empty %s: it is a link or not a directory", d) + } + if err := os.RemoveAll(d); err != nil { + return p, err + } + } + if err := os.Mkdir(d, 0o755); err != nil { + return p, err + } + } + if err := ensureRealDir(p.data); err != nil { + return p, err + } + abs, err := filepath.Abs(run) + if err != nil { + return p, err + } + p = runPaths{filepath.Join(abs, "pkg"), filepath.Join(abs, "in"), filepath.Join(abs, "out"), filepath.Join(abs, "data")} + for _, rel := range files.Paths { + if err := copyFile(filepath.Join(files.Dir, filepath.FromSlash(rel)), filepath.Join(p.pkg, filepath.FromSlash(rel)), files.Modes[rel]); err != nil { + return p, err + } + } + return p, nil +} + +func ensureRealDir(d string) error { + if _, err := os.Lstat(d); errors.Is(err, os.ErrNotExist) { + return os.Mkdir(d, 0o755) + } + if err := RealDir(d); err != nil { + return fmt.Errorf("refusing to use %s: it is a link or not a directory", d) + } + return nil +} + +// copyFile stages one file with the mode build gives it, so the staged package is the one +// unzip produces on the VM. +func copyFile(src, dst string, mode os.FileMode) error { + if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { + return err + } + in, err := os.Open(src) + if err != nil { + return err + } + defer in.Close() + out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_EXCL, mode) + if err != nil { + return err + } + if _, err := io.Copy(out, in); err != nil { + out.Close() + return err + } + return out.Close() +} + +// entrypointCommand is the runner's choice of interpreter: python3.11 for a .py entrypoint, +// otherwise the entrypoint itself. A laptop without python3.11 falls back to python3, saying so. +func entrypointCommand(pkgDir, entrypoint string, lookPath func(string) (string, error)) (string, []string, string, error) { + path := filepath.Join(pkgDir, filepath.FromSlash(entrypoint)) + if !strings.HasSuffix(entrypoint, ".py") { + return path, nil, "", nil + } + if p, err := lookPath("python3.11"); err == nil { + return p, []string{path}, "", nil + } + if p, err := lookPath("python3"); err == nil { + return p, []string{path}, "note: python3.11, the VM's interpreter, is not on PATH; running with python3", nil + } + return "", nil, "", fmt.Errorf("neither python3.11 nor python3 is on PATH") +} + +// environment is the passthrough taken from the caller's environment (LC_* included), then +// the runner's variables over it; nothing else is inherited. +func environment(from []string, set map[string]string) []string { + keep := map[string]string{} + for _, kv := range from { + k, v, ok := strings.Cut(kv, "=") + if !ok { + continue + } + if strings.HasPrefix(k, "LC_") || contains(passthrough, strings.ToUpper(k)) { + keep[k] = v + } + } + for k, v := range set { + keep[k] = v + } + out := make([]string, 0, len(keep)) + for k, v := range keep { + out = append(out, k+"="+v) + } + sort.Strings(out) + return out +} + +func contains(list []string, s string) bool { + for _, x := range list { + if x == s { + return true + } + } + return false +} +``` +```go +//go:build !windows + +package packages + +import ( + "os/exec" + "syscall" +) + +// killGroup runs the package in its own process group and kills the whole group on timeout, +// so a child the entrypoint started (cc-data, a shell) does not outlive the bound. +func killGroup(cmd *exec.Cmd) { + cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) } +} +``` +```go +//go:build windows + +package packages + +import "os/exec" + +// killGroup kills only the entrypoint on Windows, which has no release. +func killGroup(cmd *exec.Cmd) {} +``` + +Tests (the entrypoint is a `/bin/sh` script, so these skip on Windows): +- **`TestLocalScopeRefusesTheRunnersKeys`**: a scope file setting `dataset` is refused. +- **`TestRunGivesThePackageTheRunnersLayout`**: + - The package dumps its environment, copies `scope.json` and lists its working directory. + - The environment carries `RD_DATASET` (the full ref), `CC_DATA_PORTAL`, `RD_SCOPE_FILE` and a passed-through `LC_ALL`, and not an injected `AWS_SECRET_ACCESS_KEY`. + - `scope.json` has the bare `dataset`, `clue_source: "firebase"` and the absolute `output_dir`. + - The working directory has no `local-data/`, and `.gitignore` is `*`. +- **`TestRunRefusesBeforeStarting`**: when `Applies` says no, the error is the runner's prefix plus the reason, and the entrypoint never starts. A `clue_prepull` package is refused. +- **`TestRunWarnsAndRunsWhenApplicabilityIsUnconfirmed`**: an unconfirmed verdict prints the warning and runs. +- **`TestRunKillsAtTheBound`**: `sleep 30 & wait` under a 300 ms bound fails as "past its bound" within 5 seconds. +- **`TestRunRefusesALinkedRunDirectory`**: `.cc-data-run` linked to another directory is refused, and a file in that directory survives. + +| Mutation | Test that fails | +|---|---| +| no `Setpgid` | `TestRunKillsAtTheBound` (the orphaned `sleep` holds the output pipe for 30 s) | +| link check on the run directory removed | `TestRunRefusesALinkedRunDirectory` | +| the whole environment inherited | `TestRunGivesThePackageTheRunnersLayout` | +| run from the source directory, not the staged copy | `TestRunGivesThePackageTheRunnersLayout` | +| a "does not apply" verdict ignored | `TestRunRefusesBeforeStarting` | +| an unconfirmed verdict treated as a refusal | `TestRunWarnsAndRunsWhenApplicabilityIsUnconfirmed` | + +--- + +### The API client: publish, validate and applies + +**Summary**: R12, R19, R21 to R23, client side. +- **`send`.** `Client.do` keeps its signature and delegates to a new `send` that names the body's media type; `attempt` takes it as a parameter. +- **`PublishPackage` and `ValidatePackage`** post the raw zip. A POST is already never retried (`isIdempotent`), and `net/http` sets `Content-Length` from the `*bytes.Reader`, which report-server requires. +- **`PackageApplies`** posts JSON. +- **`RouteMissing`** recognizes the 404 an older report-server gives. + +**Files affected**: +- `internal/api/client.go`: `send`, plus a content-type parameter on `attempt`. +- `internal/api/packages.go`: new. + +**Estimated diff size**: ~140 lines; the requests' shapes are tested at the command level below + +```diff +--- /tmp/o 2026-10-08 05:22:45.439149318 -0400 ++++ internal/api/client.go 2026-10-07 17:17:44.313590304 -0400 +@@ -95,6 +95,11 @@ + // do runs a request under the retry policy and returns the 2xx body, or a typed + // *APIError (contract) / *TransientError (budget exhausted). + func (c *Client) do(ctx context.Context, method, path string, query url.Values, body []byte) ([]byte, error) { ++ return c.send(ctx, method, path, query, body, "application/json") ++} ++ ++// send is do with the body's media type named, for the one route that takes raw bytes. ++func (c *Client) send(ctx context.Context, method, path string, query url.Values, body []byte, contentType string) ([]byte, error) { + u := c.BaseURL + path + if len(query) > 0 { + u += "?" + query.Encode() +@@ -110,7 +115,7 @@ + return nil, err + } + +- data, status, err := c.attempt(ctx, method, u, body) ++ data, status, err := c.attempt(ctx, method, u, body, contentType) + if err != nil { + last = err + // A non-idempotent request (POST) may have reached the server, so +@@ -139,7 +144,7 @@ + + // attempt performs one HTTP request under a per-attempt deadline and returns the + // body and status; a transport error (including deadline) is returned as err. +-func (c *Client) attempt(ctx context.Context, method, u string, body []byte) ([]byte, int, error) { ++func (c *Client) attempt(ctx context.Context, method, u string, body []byte, contentType string) ([]byte, int, error) { + reqCtx := ctx + var cancel context.CancelFunc + if c.RequestTimeout > 0 { +@@ -160,7 +165,7 @@ + } + req.Header.Set("Accept", "application/json") + if body != nil { +- req.Header.Set("Content-Type", "application/json") ++ req.Header.Set("Content-Type", contentType) + } + + resp, err := c.HTTP.Do(req) +``` +```go +package api + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/url" +) + +// PublishedPackage is report-server's answer to a publish (REPORT-142). +type PublishedPackage struct { + CatalogID int64 `json:"catalog_id"` + Identity string `json:"identity"` + Version string `json:"version"` + Checksum string `json:"checksum"` + Visibility string `json:"visibility"` + Official bool `json:"official"` + CurrentVersion string `json:"current_version"` +} + +// PublishPackage posts a package archive as the raw request body. origin is "" for the +// caller's own users/, or projects/; official asks the server to mark it official, +// which only a publisher may. Like every POST it is never retried. +func (c *Client) PublishPackage(ctx context.Context, archive []byte, origin string, official bool) (*PublishedPackage, error) { + q := url.Values{} + if origin != "" { + q.Set("origin", origin) + } + if official { + q.Set("official", "true") + } + data, err := c.send(ctx, http.MethodPost, "/api/v1/packages", q, archive, "application/zip") + if err != nil { + return nil, err + } + var out PublishedPackage + if err := json.Unmarshal(data, &out); err != nil { + return nil, err + } + return &out, nil +} + +// ValidatedPackage is what a publish of the archive would record, as report-server's validate +// route answers it without recording anything. +type ValidatedPackage struct { + Identity string `json:"identity"` + Version string `json:"version"` + Checksum string `json:"checksum"` + Visibility string `json:"visibility"` +} + +// ValidatePackage asks report-server to apply every publish check to the archive and stop +// before storing it. A refusal is the same coded error a publish would answer. +func (c *Client) ValidatePackage(ctx context.Context, archive []byte, origin string) (*ValidatedPackage, error) { + q := url.Values{} + if origin != "" { + q.Set("origin", origin) + } + data, err := c.send(ctx, http.MethodPost, "/api/v1/packages/validate", q, archive, "application/zip") + if err != nil { + return nil, err + } + var out ValidatedPackage + if err := json.Unmarshal(data, &out); err != nil { + return nil, err + } + return &out, nil +} + +// AppliesRequest names the patterns and the scope's URLs. AssignmentURLs are followed into +// their activities by report-service's deriver, so the URLs matched are the runner's. +type AppliesRequest struct { + URLs map[string][]string `json:"urls"` + AssignmentURLs []string `json:"assignment_urls,omitempty"` + ScopeURLs []string `json:"scope_urls,omitempty"` +} + +// AppliesAnswer is report-server's verdict, with what the deriver read and could not. +type AppliesAnswer struct { + Applies bool `json:"applies"` + Reason string `json:"reason"` + InteractiveURLs []string `json:"interactive_urls"` + Unread []struct { + URL string `json:"url"` + Reason string `json:"reason"` + } `json:"unread"` + Truncated bool `json:"truncated"` +} + +// PackageApplies asks report-server's one matcher whether the patterns match the scope. +func (c *Client) PackageApplies(ctx context.Context, req AppliesRequest) (*AppliesAnswer, error) { + var out AppliesAnswer + if err := c.postJSON(ctx, "/api/v1/packages/applies", req, &out); err != nil { + return nil, err + } + return &out, nil +} + +// RouteMissing reports a 404 from a route this cc-data expects and the server does not have +// yet: a report-server older than the validate and applies routes. +func RouteMissing(err error) bool { + var apiErr *APIError + return errors.As(err, &apiErr) && apiErr.Status == http.StatusNotFound +} +``` + +--- + +### The `cc-data package` commands and the `init` stub + +**Summary**: R9, R12, R16, R19 to R23. +- **Wiring.** Adds the four subcommands to the root, and `Init` with the embedded stub. +- **The report-server calls.** `validate` is shared by `build` and `run`. `appliesFor` gives `run` its `Applies` function, with a 5-minute timeout for the deriver. +- **Testability.** `build` and `publish` carry a `run(ctx, client, ...)` method, as `reportCreateFlags` does, so both are tested against `httptest` with no stored credential. + +**Files affected**: +- `cmd/package.go`: new. +- `cmd/root.go`: `root.AddCommand(newPackageCmd())` after `newQueryCmd()`. +- `internal/packages/init.go`: new. +- `internal/packages/template/run.py`: new, embedded. +- `cmd/package_test.go`: new. + +**Estimated diff size**: ~600 lines + +```go +package cmd + +import ( + "context" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "time" + + "github.com/concord-consortium/cc-data-cli/internal/api" + "github.com/concord-consortium/cc-data-cli/internal/output" + "github.com/concord-consortium/cc-data-cli/internal/packages" + "github.com/spf13/cobra" +) + +const packageLong = `Develop, test and publish a Researcher Dashboard package. + +A package is a directory holding manifest.json and an entrypoint. init writes both; run +executes the package against your own dataset under the runner's rules; build zips it; +publish registers the zip in the catalog with your cc-data token. + +What run cannot reproduce from the VM: the package runs as you, with no network sandbox, +and with HOME, PATH and the locale passed through so its own cc-data calls find your +login. A .py entrypoint runs with python3.11 when it is on PATH, as on the VM, else +python3.` + +func newPackageCmd() *cobra.Command { + cmd := &cobra.Command{ + Use: "package", + Short: "Develop, test and publish a Researcher Dashboard package", + Long: packageLong, + } + cmd.AddCommand(newPackageInitCmd(), newPackageRunCmd(), newPackageBuildCmd(), newPackagePublishCmd()) + return cmd +} + +// packageDir is the optional [dir] argument, defaulting to the current directory. +func packageDir(args []string) string { + if len(args) == 1 { + return args[0] + } + return "." +} + +// loadPackage collects a package directory and validates its manifest against what it holds. +func loadPackage(dir string) (packages.Manifest, packages.Files, error) { + files, err := packages.Collect(dir) + if err != nil { + return packages.Manifest{}, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_PACKAGE", Message: err.Error()} + } + data, err := os.ReadFile(filepath.Join(dir, "manifest.json")) + if err != nil { + return packages.Manifest{}, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_MANIFEST", Message: fmt.Sprintf("no manifest.json in %s", dir)} + } + m, err := packages.ReadManifest(data, files.Set()) + if err != nil { + return m, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_MANIFEST", Message: err.Error()} + } + if err := files.CheckEntrypointMode(m.Entrypoint); err != nil { + return m, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_PACKAGE", Message: err.Error()} + } + return m, files, nil +} + +func newPackageInitCmd() *cobra.Command { + var name string + cmd := &cobra.Command{ + Use: "init [dir]", + Short: "Write a manifest.json skeleton and a run.py stub", + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + dir := packageDir(args) + written, err := packages.Init(dir, name) + if err != nil { + return output.Usagef("%v", err) + } + return output.ResultLine(map[string]any{"dir": dir, "written": written}) + }, + } + cmd.Flags().StringVar(&name, "name", "", "the package name (default: the directory's name, when it fits ^[a-z0-9][a-z0-9-]{0,62}$)") + return cmd +} + +func newPackageRunCmd() *cobra.Command { + var datasetRef, scopePath string + cmd := &cobra.Command{ + Use: "run [dir] --dataset --scope ", + Short: "Run a package locally exactly as the dashboard's runner will", + Long: "Run a package against your own dataset under the runner's rules: the same scope.json,\n" + + "environment variables, output files, display.md cap and applicability check.\n\n" + + "--scope names a file holding the scope's kind, id, classes and assignments.\n" + + "Results are written under /.cc-data-run/, which build never ships.", + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + if datasetRef == "" || scopePath == "" { + return output.Usagef("--dataset and --scope are required") + } + dir := packageDir(args) + m, files, err := loadPackage(dir) + if err != nil { + return err + } + raw, err := os.ReadFile(scopePath) + if err != nil { + return output.Usagef("reading the scope file: %v", err) + } + scope, err := packages.ParseLocalScope(raw) + if err != nil { + return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: err.Error()} + } + cfg, dataRoot, err := loadRuntime() + if err != nil { + return err + } + ref, err := resolveRef(cfg, datasetRef) + if err != nil { + return err + } + client, err := api.ForPortal(ref.Portal) + if err != nil { + return err + } + ctx := context.Background() + archive, err := packages.Zip(files) + if err != nil { + return output.Internalf("%v", err) + } + if err := validate(ctx, client, archive, ""); err != nil { + return err + } + start := time.Now() + res, err := packages.Run(ctx, packages.RunOptions{ + Dir: dir, Manifest: m, Files: files, Scope: scope, Applies: appliesFor(client, scope), + Dataset: ref.String(), Name: ref.Name, Portal: ref.Portal.Host(), DataRoot: dataRoot, + ReportSrv: client.BaseURL, Stderr: output.Stderr(), Environ: os.Environ, LookPath: exec.LookPath, + }) + if err != nil { + return packageRunError(err) + } + return output.ResultLine(map[string]any{ + "display": res.DisplayPath, "display_bytes": res.DisplayBytes, "summary": res.Summary, + "counts": res.Counts, "elapsed_seconds": int(time.Since(start).Seconds()), + }) + }, + } + cmd.Flags().StringVar(&datasetRef, "dataset", "", "the dataset the package pulls into and reads: /") + cmd.Flags().StringVar(&scopePath, "scope", "", "a JSON file holding the scope's kind, id, classes and assignments") + return cmd +} + +// derivationTimeout covers report-service's deriver, which may spend up to 240 seconds +// fetching a scope's activities; the client's 60-second default would cut it off. +const derivationTimeout = 5 * time.Minute + +// validate asks report-server whether a publish of the archive would be accepted. A +// report-server without the route is not a refusal: the archive is unchecked, and saying so +// is the whole answer. +func validate(ctx context.Context, client *api.Client, archive []byte, origin string) error { + _, err := client.ValidatePackage(ctx, archive, origin) + switch { + case api.RouteMissing(err): + output.Warnf("report-server has no validate route yet, so this package is not checked against the catalog's rules until publish") + return nil + case err != nil: + return api.AsCLIError(err) + } + return nil +} + +// appliesFor is package run's applicability check: report-server's one matcher, against the +// scope's assignment URLs and the interactive URLs report-service's deriver finds in them. +func appliesFor(client *api.Client, scope packages.LocalScope) func(context.Context, packages.URLs) (packages.Verdict, error) { + return func(ctx context.Context, urls packages.URLs) (packages.Verdict, error) { + long := *client + long.RequestTimeout = derivationTimeout + ans, err := long.PackageApplies(ctx, api.AppliesRequest{ + URLs: map[string][]string{"all": urls.All, "any": urls.Any, "none": urls.None}, + AssignmentURLs: scope.AssignmentURLs(), + }) + switch { + case api.RouteMissing(err): + return packages.Verdict{Unconfirmed: true}, nil + case err != nil: + return packages.Verdict{}, api.AsCLIError(err) + } + for _, u := range ans.Unread { + output.Warnf("the class profile could not read %s (%s), so its interactives are not matched", u.URL, u.Reason) + } + if ans.Truncated { + output.Warnf("the class profile was truncated, so not every interactive URL was matched") + } + return packages.Verdict{Applies: ans.Applies, Reason: ans.Reason}, nil + } +} + +func packageRunError(err error) error { + var refused *packages.Refused + var failed *packages.Failed + var out *packages.OutputRefused + switch { + case errors.As(err, &refused): + return &output.CLIError{ExitCode: output.ExitInternal, Code: "PACKAGE_REFUSED", Message: err.Error()} + case errors.As(err, &failed): + return &output.CLIError{ExitCode: output.ExitInternal, Code: "PACKAGE_FAILED", Message: err.Error()} + case errors.As(err, &out): + return &output.CLIError{ExitCode: output.ExitInternal, Code: "PACKAGE_OUTPUT_REFUSED", Message: err.Error()} + } + return output.Internalf("%v", err) +} + +type packageBuildFlags struct { + out, portal, origin string +} + +// run builds, validates and writes; it takes the client so it can be exercised against a test +// server without a stored credential. +func (f packageBuildFlags) run(ctx context.Context, client *api.Client, dir string) error { + m, files, err := loadPackage(dir) + if err != nil { + return err + } + out := f.out + if out == "" { + out = filepath.Join(dir, packages.BuildDirName, fmt.Sprintf("%s-%s.zip", m.Name, m.Version)) + } else if err := packages.CheckOutsidePackage(dir, out); err != nil { + return output.Usagef("%v", err) + } + archive, err := packages.Zip(files) + if err != nil { + return output.Internalf("%v", err) + } + if err := validate(ctx, client, archive, f.origin); err != nil { + return err + } + if err := packages.WriteBuild(out, archive); err != nil { + return output.Internalf("%v", err) + } + return output.ResultLine(map[string]any{ + "path": out, "checksum": packages.Checksum(archive), "bytes": len(archive), "files": len(files.Paths), + }) +} + +func newPackageBuildCmd() *cobra.Command { + var f packageBuildFlags + cmd := &cobra.Command{ + Use: "build [dir]", + Short: "Zip a package reproducibly, check it against the catalog's rules and print its checksum", + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + client, err := reportsClient(f.portal) + if err != nil { + return err + } + return f.run(context.Background(), client, packageDir(args)) + }, + } + cmd.Flags().StringVar(&f.out, "out", "", "where to write the zip (default: /.cc-data-build/-.zip)") + cmd.Flags().StringVar(&f.portal, "portal", "", "portal whose catalog rules to check against: an environment alias or a hostname") + cmd.Flags().StringVar(&f.origin, "origin", "", "projects/, when the package will be published as a project's") + return cmd +} + +type packagePublishFlags struct { + portal, origin string + official, asJSON bool +} + +// run publishes an archive already read and checked; it takes the client so it can be +// exercised against a test server without a stored credential. +func (f packagePublishFlags) run(ctx context.Context, client *api.Client, archive []byte) error { + got, err := client.PublishPackage(ctx, archive, f.origin, f.official) + if err != nil { + return api.AsWriteCLIError(err, "Publishing again is safe: ALREADY_EXISTS then means this attempt landed") + } + if want := packages.Checksum(archive); got.Checksum != want { + return output.Internalf("report-server recorded checksum %s, but the archive sent is %s", got.Checksum, want) + } + if f.asJSON { + return output.ResultLine(got) + } + fmt.Fprintf(output.Stdout(), "identity %s\nversion %s\nvisibility %s\nofficial %t\ncurrent version %s\nchecksum %s\n", + got.Identity, got.Version, got.Visibility, got.Official, got.CurrentVersion, got.Checksum) + return nil +} + +func newPackagePublishCmd() *cobra.Command { + var f packagePublishFlags + cmd := &cobra.Command{ + Use: "publish [--portal ]", + Short: "Publish a built package to the catalog with your cc-data token", + Long: "POST the zip to report-server's catalog, which creates the package private on its first\n" + + "publish and records the version. --origin projects/ publishes a project package;\n" + + "--official needs the publisher role. The request is never retried.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + if f.origin != "" && !strings.HasPrefix(f.origin, "projects/") { + return output.Usagef("--origin must be projects/; your own packages need no origin") + } + archive, err := os.ReadFile(args[0]) + if err != nil { + return output.Usagef("reading %s: %v", args[0], err) + } + client, err := reportsClient(f.portal) + if err != nil { + return err + } + return f.run(context.Background(), client, archive) + }, + } + cmd.Flags().StringVar(&f.portal, "portal", "", "portal whose catalog to publish to: an environment alias or a hostname") + cmd.Flags().StringVar(&f.origin, "origin", "", "projects/ to publish a project's package") + cmd.Flags().BoolVar(&f.official, "official", false, "mark the package official (publisher role only)") + cmd.Flags().BoolVar(&f.asJSON, "json", false, "emit the catalog's answer as JSON") + return cmd +} +``` +```go +package packages + +import ( + _ "embed" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "strings" +) + +// BuildDirName is where build writes by default: inside the package, and skipped by +// Collect like every dot path, so one build's zip never reaches the next. +const BuildDirName = ".cc-data-build" + +//go:embed template/run.py +var runStub []byte + +// Init writes manifest.json and run.py into dir, creating it if needed. It reads nothing in +// dir except to refuse when either file already exists, in which case it writes neither. +func Init(dir, name string) ([]string, error) { + if name == "" { + abs, err := filepath.Abs(dir) + if err != nil { + return nil, err + } + name = filepath.Base(abs) + if !ValidName(name) { + return nil, fmt.Errorf("the directory name %q is not a package name; pass --name (^[a-z0-9][a-z0-9-]{0,62}$)", name) + } + } else if !ValidName(name) { + return nil, fmt.Errorf("--name %q must match ^[a-z0-9][a-z0-9-]{0,62}$", name) + } + for _, f := range []string{"manifest.json", "run.py"} { + if _, err := os.Lstat(filepath.Join(dir, f)); !errors.Is(err, os.ErrNotExist) { + return nil, fmt.Errorf("%s already exists in %s; init writes only into a package that has neither file", f, dir) + } + } + manifest := Manifest{ + Name: name, Title: name, Version: "0.1.0", + Description: "One line saying what this package shows.", + URLs: URLs{All: []string{}, Any: []string{}, None: []string{}}, + Entrypoint: "run.py", ExpectedDurationSeconds: 300, + } + b, err := json.MarshalIndent(manifest, "", " ") + if err != nil { + return nil, err + } + if err := os.MkdirAll(dir, 0o755); err != nil { + return nil, err + } + if err := os.WriteFile(filepath.Join(dir, "manifest.json"), append(b, '\n'), 0o644); err != nil { + return nil, err + } + if err := os.WriteFile(filepath.Join(dir, "run.py"), runStub, 0o755); err != nil { + return nil, err + } + return []string{"manifest.json", "run.py"}, nil +} + +// CheckOutsidePackage refuses an explicit build output that a later build would collect. +func CheckOutsidePackage(dir, out string) error { + absDir, err := filepath.Abs(dir) + if err != nil { + return err + } + absOut, err := filepath.Abs(out) + if err != nil { + return err + } + rel, err := filepath.Rel(absDir, absOut) + if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { + return nil + } + if excluded(filepath.ToSlash(rel), false) || excludedAncestor(filepath.ToSlash(rel)) { + return nil + } + return fmt.Errorf("--out %s is inside the package, where the next build would include it", out) +} + +func excludedAncestor(rel string) bool { + parts := strings.Split(rel, "/") + for i := 1; i < len(parts); i++ { + if excluded(strings.Join(parts[:i], "/"), true) { + return true + } + } + return false +} + +// WriteBuild writes the archive, creating its directory, with the .gitignore every working +// directory this tool makes inside a package carries. +func WriteBuild(out string, archive []byte) error { + dir := filepath.Dir(out) + if err := os.MkdirAll(dir, 0o755); err != nil { + return err + } + if filepath.Base(dir) == BuildDirName { + if err := os.WriteFile(filepath.Join(dir, ".gitignore"), []byte("*\n"), 0o644); err != nil { + return err + } + } + return os.WriteFile(out, archive, 0o644) +} +``` +```python +#!/usr/bin/env python3 +"""A Researcher Dashboard package: counts a class's answers and the learners who wrote them. + +It runs the same way under `cc-data package run` on a laptop and on the dashboard's VM. It +reads the scope from RD_SCOPE_FILE, pulls its own data into RD_DATASET with cc-data, counts +through cc-data's views, and writes display.md, summary.txt and counts.json to the scope's +output_dir. Stdlib only: the VM installs no Python packages. + +Two limits to know before building on it: +- It creates a Student Answers run for the class the first time it meets one, on the server + the dataset's portal names. Later runs reuse that run and re-pull it. +- It filters the whole class in one run. Athena fails a whole-class Student Answers run above + about four assignments; chunking by scope["assignments"] is the fix. +""" +import json +import os +import subprocess +import sys + + +def cc_data(*args): + result = subprocess.run(["cc-data", *args], capture_output=True, text=True) + if result.returncode != 0: + detail = " ".join(p for p in (result.stderr.strip(), result.stdout.strip()) if p) + raise RuntimeError(f"cc-data {' '.join(args[:2])} exited {result.returncode}: {detail or '(no output)'}") + return result.stdout + + +def query(dataset, sql): + return json.loads(cc_data("query", "--dataset", dataset, sql, "--format", "json")) + + +def answers_run(dataset, class_id): + """The class's Student Answers run: one already in the dataset, else a new one.""" + # downloads first: until a report is pulled, the reports view has no class_id to filter on. + pulled = query(dataset, "SELECT run_id FROM downloads WHERE type = 'report' AND slug = 'student-answers'") + if pulled: + ids = ", ".join(str(int(row["run_id"])) for row in pulled) + found = query(dataset, f"SELECT max(run_id) AS run_id FROM reports WHERE run_id IN ({ids}) AND class_id = {int(class_id)}") + if found and found[0]["run_id"] is not None: + return int(found[0]["run_id"]) + created = json.loads(cc_data( + "reports", "create", "--portal", os.environ["CC_DATA_PORTAL"], + "--report-slug", "student-answers", + "--report-filter", json.dumps({"class": [int(class_id)]}), "--json")) + return int(created["run"]["run_id"]) + + +def main(): + with open(os.environ["RD_SCOPE_FILE"], encoding="utf-8") as handle: + scope = json.load(handle) + # RD_DATASET is the full / ref; scope["dataset"] is the bare name. + dataset = os.environ["RD_DATASET"] + class_id = scope["classes"][0]["class_id"] + + try: + cc_data("dataset", "create", dataset) + except RuntimeError as err: + if "exists" not in str(err).lower(): + raise + run = answers_run(dataset, class_id) + cc_data("get", "report", str(run), "--dataset", dataset, "--refresh") + cc_data("get", "answers", str(run), "--dataset", dataset) + + rows = query(dataset, f"SELECT count(*) AS n FROM report_{run}")[0]["n"] + if rows == 0: + raise RuntimeError("no report-service access to this class, or the class has no data") + answers = query(dataset, f"SELECT count(*) AS n FROM run_answers WHERE run_id = {run}")[0]["n"] + learners = query(dataset, f""" + SELECT count(DISTINCT e.user_id) AS n FROM learner_endpoints e + JOIN run_answers a USING (run_id, remote_endpoint) WHERE run_id = {run}""")[0]["n"] + + out = scope["output_dir"] + with open(os.path.join(out, "display.md"), "w", encoding="utf-8") as handle: + handle.write(f"# Class {class_id}\n\n- {answers} answers\n- {learners} learners with at least one answer\n") + with open(os.path.join(out, "summary.txt"), "w", encoding="utf-8") as handle: + handle.write(f"{answers} answers from {learners} learners\n") + with open(os.path.join(out, "counts.json"), "w", encoding="utf-8") as handle: + json.dump({"answers": answers}, handle) + + +if __name__ == "__main__": + try: + main() + except Exception as err: + print(str(err), file=sys.stderr) + sys.exit(1) +``` + +The stub was run through `Run` against a copy of `staging-smoketest`, using a wrapper `cc-data` that forwards `dataset` and `query` to the scratch build and turns `get` into a no-op. +- **The result.** It wrote "5 answers / 2 learners with at least one answer" and `counts.json` `{"answers": 5}`. +- **On an empty dataset** it called `reports create --report-slug student-answers --report-filter {"class": [6]}`. +- **Why it reads `downloads` first.** An earlier draft queried `reports.class_id` before any report existed, and the empty `reports` stand-in has no such column. + +Tests (against an `httptest` server that answers 404 for any route it was not given): +- **`TestPackageInitRefusesASecondInit`**. +- **`TestPackageBuildValidatesThenWrites`**: + - Exactly one validate request is made, carrying the bytes that were written, as `application/zip`. + - The zip is `wildfire-responses-0.1.0.zip` with two files, and the printed checksum is the archive's own. + - A second build from inside the package gives the same checksum. +- **`TestPackageBuildWritesNothingTheServerRefuses`**: a 422 `UNPROCESSABLE` keeps its code, exits 5, and no zip is written. +- **`TestPackageBuildOnAServerWithoutTheRouteStillBuilds`**: a 404 is a warning. +- **`TestPackageBuildRefusesAnOutputItWouldCollect`**. +- **`TestPackageRunAsksTheServersMatcher`**: the request carries the patterns and the assignment URLs, and a "does not apply" answer comes back as the verdict. +- **`TestPackageRunOnAServerWithoutTheRouteIsUnconfirmed`**. +- **`TestPackagePublishSendsTheZipAsTheBody`**: checks POST, `application/zip`, `Content-Length`, `origin=projects/20` present, `official` absent, and the bearer. +- **`TestPackagePublishRefusesAChecksumItDidNotSend`**. +- **`TestPackagePublishPassesTheServersCodeThrough`**: a 409 `ALREADY_EXISTS` keeps its code, exits 5, and is sent once. + +| Mutation | Test that fails | +|---|---| +| a 404 from validate treated as a refusal | `TestPackageBuildOnAServerWithoutTheRouteStillBuilds` | +| `build` skips validate | `TestPackageBuildValidatesThenWrites`, `...WritesNothingTheServerRefuses` | +| assignment URLs not sent to applies | `TestPackageRunAsksTheServersMatcher` | + +--- + +### Documentation + +**Summary**: R24. The README's "Commands (sketch)" gains the four `package` lines. The researcher guide gains a section, "Developing and publishing a package", with these parts: +- the loop (`init`, edit, `run`, `build`, `publish`); +- the scope file's four keys, with an example, kept outside the package or under a dot name such as `.scope.json` so `build` never ships a class hash; +- what `run` reproduces and what it cannot (the sandbox, the passthrough variables, python3.11); +- that `build` and `run` check against report-server, and the warning an older report-server gives; +- `.cc-data-run/` and `.cc-data-build/` (gitignored, and holding student data); +- the 422 "publishing is not configured" answer a portal gives before its bucket exists. + +`skill_header.md` gains a short "Packages" list naming the four commands. The `--help` texts above already state the differences between a local run and a VM. + +**Files affected**: +- `README.md`. +- `docs/researcher-guide.md`. +- `internal/guidance/src/skill_header.md`. + +**Estimated diff size**: ~90 lines + +## Open Questions + + + +### RESOLVED: Judgment call: how a raw zip body reaches the server +**Options considered**: +- A) Generalize `Client.do` into `send` with a content type. +- B) Build a separate `http.Request` for the zip routes. + +**Decision**: A. Publish and validate then share the bearer, the cross-origin redirect refusal, the no-retry rule for POST and the error-envelope decoding with every other call. B would re-implement each of them. + +### RESOLVED: Judgment call: the exit class for a refused or failed package +**Options considered**: +- A) Exit 1 with codes `PACKAGE_REFUSED`, `PACKAGE_FAILED` and `PACKAGE_OUTPUT_REFUSED`. +- B) A new exit code 7. + +**Decision**: A. The root help defines exit 1 as "internal/other error", and the JSON envelope's code already distinguishes the cases. A new exit class is a contract change for every script that reads cc-data's exit codes. + +### RESOLVED: Judgment call: following no link without `O_NOFOLLOW` +**Options considered**: +- A) `Lstat`, then open, then `os.SameFile`. +- B) `syscall.O_NOFOLLOW`. + +**Decision**: A. CI tests on `windows-2022`, where `syscall.O_NOFOLLOW` does not exist. On a laptop the race A leaves open would need the author's own package to swap a file mid-read, and the runner, which runs as root, is the boundary that matters. + +### RESOLVED: Judgment call: what a 404 from the new routes means +**Options considered**: +- A) Any 404 from validate or applies means the route is not deployed: warn and proceed. +- B) Require a specific error code before treating it as "not deployed". + +**Decision**: A. report-server's router has no route either path could match today (`/packages/validate` and `/packages/applies` are one segment, and the only other `POST /packages/...` route takes four). So a 404 can only mean the route is missing, and B would need a code that an older server, by definition, cannot send. + +### RESOLVED: Low confidence: the process-group kill on macOS +**Context**: `TestRunKillsAtTheBound` ran on Linux only. + +**Decision**: `Setpgid` and `kill(-pgid, SIGKILL)` are POSIX and behave the same on Darwin, and `GOOS=darwin go build ./internal/packages/` compiles. CI's `macos-15` and `macos-15-intel` jobs run the test, so a Darwin difference would fail there. + +## Self-Review + +Roles: commit reviewer, test writer, senior engineer, operator, security engineer. Each finding was checked against the scratch build before it was written down. + +### Senior Engineer + +#### RESOLVED: A shell entrypoint would run locally and be refused on the VM +The zip originally gave every entry mode 0644, and the runner executes a non-`.py` entrypoint directly. A Go zip with a 0644 `plain.sh` and a 0755 `exec.sh`, unpacked with `unzip`, gives "Permission denied" for the first and runs the second, so `unzip` restores the zip's modes. Locally, the staged copy kept the source's 0755, so `package run` passed. Fixed: +- `Files` records a ship mode per file (0755 if any execute bit is set, else 0644). +- `Zip` and the staging copy both use it. +- `CheckEntrypointMode` refuses a non-executable non-`.py` entrypoint in `run` and `build`. + +R17 states the rule. + +#### RESOLVED: `RunOptions.Now` was never read +The field and its assignments were dead, so they are removed. Elapsed time is measured in `cmd/package.go`, where it is reported. + +### Commit Reviewer + +#### RESOLVED: Shared test helpers would not compile step by step +The scratch build kept every test in one file. The plan places each helper in the first step whose tests use it: `goodManifest` and `writeTree` in `files_test.go`, and the run helpers in `run_test.go`. + +### Operator + +#### RESOLVED: A scope file inside the package ships with it +An author naming the file `scope.json` in the package directory would publish a class hash in every archive. The documentation step says to keep it outside the package or under a dot name, which `Collect` already excludes. + +#### RESOLVED: The applicability call would time out before the deriver finishes +The deriver may spend up to 240 seconds on a scope's activities, and the client's per-attempt timeout for JSON calls is 60 seconds. `appliesFor` copies the client with a 5-minute `RequestTimeout`, which also keeps the shared client's default for every other call. diff --git a/specs/REPORT-146-package-commands-and-dataset-views/requirements.md b/specs/REPORT-146-package-commands-and-dataset-views/requirements.md new file mode 100644 index 0000000..6833484 --- /dev/null +++ b/specs/REPORT-146-package-commands-and-dataset-views/requirements.md @@ -0,0 +1,347 @@ +# cc-data-cli: the package commands and the dataset views that replace `_lib` + +**Jira**: https://concord-consortium.atlassian.net/browse/REPORT-146 +**Repo**: https://github.com/concord-consortium/cc-data-cli +**Implementation Spec**: [implementation.md](implementation.md) +**Status**: **In Development** + +## Overview + +Researchers get four `cc-data package` commands (`init`, `run`, `build`, `publish`) for writing a Researcher Dashboard package on a laptop, testing it the way the dashboard's runner will run it, and publishing it to the package catalog with the cc-data token they already have. Two new dataset views (`run_answers` and `learner_endpoints`) replace the counting SQL the spike copied into every package. + +## Project Owner Overview + +The Researcher Dashboard runs analysis "packages" for a class on a short-lived virtual machine. In the spike, a package was built and uploaded by a script that called AWS directly. Each package also carried a shared library of SQL describing how cc-data lays out a dataset. As a result, developing a package needed AWS credentials, and the only way to find a mistake was to run the package on a VM. + +This story moves the whole loop onto the researcher's laptop and onto the token they already use for cc-data. `cc-data package run` runs a package against their own dataset under the runner's rules: the same `scope.json`, the same output files and the same size limit on the result. It asks report-server, rather than a copy of its rules, whether the package is valid and whether it applies to the class. An author therefore hits these failures on their laptop, not on a VM. `cc-data package publish` registers the package in the catalog in one step. The shared SQL becomes two named views in cc-data itself, so a package writes plain SQL against stable names and carries no library. The first package built this way is the Wildfire package (REPORT-147), whose spec waits on this one. + +**One rule, one home.** The package rules (what a valid package is) and applicability (whether a package suits a class) are decided by report-server alone. cc-data, the dashboard app and the runner ask it and do not keep their own copies. The new report-service routes this needs are one companion story (see Dependencies). + +## Background + +The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) sections 5.5 (applicability by URL patterns), 5.7 (publishing and iterating), 10 (`manifest.json`, `scope.json`, the result a package writes) and 13 (data sources). The section numbers below are that document's. + +**What `_lib` was.** On cc-data-studies' unmerged spike branch `RIGSE-365-analysis-packages` (`838a603`), `studies/_lib/reports.py` held three queries. `studies/_lib/ccdata.py` held the cc-data calls a class study makes: it creates the Student Answers and Student Actions runs, pulls them, and keeps run ids in `runs.json`. `publish-package.sh` copied `_lib/` into every archive. REPORT-147 builds on cc-data-studies `main`, where `_lib` never landed, so this story adds new views rather than replacing existing code. `_lib` is only the reference for what the views must cover (stream channel, 2026-10-07). The three queries are: + +1. **Answers scoped to a run**: `answers JOIN run_membership USING (source_key, remote_endpoint, question_id) WHERE m.run_id = AND m.type = 'answers'`. The query is written this way, and not against `answers_`, because the per-run view exists only once the run has membership on disk. A class with no answers would then fail to bind instead of counting zero. +2. **Learners with at least one answer**: a `UNION ALL` over every `res__remote_endpoint` column of `report_` (paired with `user_id`), joined to that run's answers, then `count(DISTINCT user_id)`. The column list differs from class to class, so the spike read it with `DESCRIBE` before building the SQL. +3. **Log freshness**: `count(*), max(event_time) FROM logs WHERE run_id = `. + +`ccdata.py`'s run creation and pulling stay package code. Section 13 says a package makes its own pulls, and the Jira story says the runner does not create a package's report runs. Nothing replaces that half in cc-data. + +**What cc-data already has** (`internal/duck/views.go`, released in v0.2.0, which is `main` at `58c9172`): + +- `run_membership` (run_id, type, identity columns), `answers`, `history`, `reports`, `report_`, `answers_`. Each has a typed-empty fallback, so they always bind. +- `logs`, with `run_id` and `event_time` (REPORT-112). The freshness query is already plain SQL against stable names, so query 3 needs no new view. +- A drift guard (`internal/guidance/guard_test.go`): every name in `StaticViewNames()` must be documented in `internal/guidance/src/core.md`'s Views section and in `docs/researcher-guide.md`'s "How datasets are organized" table, and the reverse. `cc-data query --help` and the MCP `query` tool description list the static views from the same function. + +**What the runner does to a package.** This is RD-4 pass 2 as spec'd and built in its stage-verification throwaway build (`stage5-throwaway-build.patch`, researcher-dashboard branch oob `RD-4-pass-2-pull-loop`). It is not implemented on any branch yet. + +- It validates `manifest.json`: + - required fields; + - `expected_duration_seconds` a positive integer and at most `PACKAGE_MAX_DURATION_SECONDS` (default 7200); + - `clue_prepull` boolean if present; + - `urls` keys only `all`, `any` and `none`, each a list of non-empty strings; + - the entrypoint inside the package; + - `name` equal to the last segment of the identity. +- When `urls` declares any pattern, it matches the patterns against the class profile's `assignment_urls` ∪ `interactive_urls`, using the glob of section 5.5. It refuses with "the package does not apply to this class: ". It also refuses a package that declares patterns on a class with no profile yet. +- It writes `scope.json` with exactly `kind`, `id`, `classes`, `assignments`, `clue_source`, `dataset` (the bare name `pkg-`) and `output_dir`. It names the file in `RD_SCOPE_FILE`. +- It replaces the package's environment with `HOME`, `PATH=/usr/local/bin:/usr/bin:/bin`, `CC_DATA_ROOT`, `CC_DATA_PORTAL`, `CC_DATA_LOCAL`, `RD_DATASET` (`/pkg-`), `RD_SCOPE_FILE`, `RD_DATA_DIR` and `RD_OUTPUT_DIR`, plus `RD_REPORT_SERVER_URL`, `RD_BUCKET` and `RD_STORAGE_PREFIX` when set. +- It runs a `.py` entrypoint with `python3.11`, and any other entrypoint directly. The bound is `expected_duration_seconds` plus a 10-minute margin. +- It empties the output directory before each run. Afterwards it reads, without following links: + - `display.md`, required and refused over 65,536 bytes (never truncated); + - `summary.txt`, whose first line, trimmed, is the summary (or null); + - `counts.json`, for `answers`, `logs` and `log_freshness_at` only. + +**What report-server accepts** (`server/lib/report_server/packages/*.ex` and `package_controller.ex` on report-service `master`, `067a1c4`, live on staging as `1.12.0-pre.2`): + +- **The request.** `POST /api/v1/packages` behind the ordinary API token (the cc-data token). The body is the zip, with `Content-Type: application/zip` and a declared `Content-Length`. Query parameters are `origin=projects/` (absent means the caller's own `users/`) and `official=true|false`. Only a publisher may set `official`. +- **Archive limits.** At most 10 MiB, at most 50 MiB declared uncompressed in total, no absolute or `..` entry, exactly one root `manifest.json` of at most 64 KiB. +- **Manifest rules.** + - `name` matches `^[a-z0-9][a-z0-9-]{0,62}$`. + - `version` is `MAJOR.MINOR.PATCH[-prerelease]`, at most 64 characters. + - `title` is non-empty, at most 200 characters. + - `description` is optional: one line, at most 500 characters. + - `urls` has only `all`/`any`/`none` arrays, each pattern non-empty with no whitespace or control characters and at most 256 characters, and at most 20 patterns in total. + - `clue_prepull` is boolean. + - `entrypoint` is a file in the archive. + - `expected_duration_seconds` is 1 to 28,800. + - No `owner`, `maintainer`, `origin`, `visibility`, `project` or `official` key. +- **On success (201).** The response is `{catalog_id, identity, version, checksum, visibility, official, current_version}`, where `checksum` is `sha256:` of the body. +- **On failure.** The flat envelope `{"error", "message"}` with `BAD_REQUEST`, `UNPROCESSABLE` (422), `FORBIDDEN`, `ALREADY_EXISTS` (409), `NOT_AUTHENTICATED` or `SERVICE_UNAVAILABLE`. +- **Visibility and the version pointer.** A first publish creates the package `private`. For a private package, every publish moves `current_version` to the new version. + +## Dependencies + +**A companion report-service story (to be created)** owns the routes cc-data calls. REPORT-146 codes against the contract below, which that story's spec may refine. Until those routes are deployed, `build` and `run` warn and proceed (R6, R12), so REPORT-146 is not blocked on it. + +- **`POST /api/v1/packages/validate`** (cc-data token). The body is the raw zip, sent exactly as for publish, with `origin` as an optional query parameter. It runs every check `POST /api/v1/packages` runs, including permission and origin, and stops before writing any row or object. It answers 200 with `{identity, version, checksum, visibility}` (what a publish would record), or the same coded error a publish would give. + - It is a separate route, not a `dry_run` parameter on publish. Today's publish ignores parameters it does not know, so a `dry_run=true` sent to a server without the feature would publish for real. A missing route answers 404. +- **`POST /api/v1/packages/applies`** (cc-data token, or the researcher token the runner holds). The body is JSON: `{urls: {all, any, none}, assignment_urls: [...], scope_urls: [...]}`. + - **Assignment URLs** are followed by report-service's existing deriver (`deriveProfile` in `functions/src/researcher-dashboard/derive-profile.ts`). report-server reaches it through a new `derive_urls` route on the `api` function, the way it already calls `bulk_read`. + - **Scope URLs** are matched as given; that is the runner's case, since it already holds the class profile. + - **The answer** is `{applies, reason, interactive_urls, unread: [{url, reason}], truncated}`. The `reason` uses the runner's wording. + - **The matcher** behind it is the only glob matcher in the system. The app gets the same answer through the package-list call it already makes. +- **report-server adopts the runner's stricter rules,** so that anything it accepts the runner will run: `expected_duration_seconds` at most 7,200 (today 28,800), and no symbolic-link entries in an archive. + +## Requirements + +### Dataset views + +- **R1. `run_answers`.** A static view with every row of `answers`, plus `run_id`: one row for each Student Answers run whose membership (`run_membership.type = 'answers'`) holds that answer. Its columns are `run_id` (BIGINT) followed by the `answers` columns. Like every static view, it registers on every dataset: with no answers store or no membership it is empty with the same columns, never a bind error. +- **R2. `learner_endpoints`.** A static view with one row per learner per assignment per answers-type report CSV. Its columns are: + - `run_id` (BIGINT); + - `user_id` (BIGINT), the Portal user; + - `offering_id` (BIGINT), from the same assignment's `res__offering_id`, NULL where the CSV has none; + - `remote_endpoint` (VARCHAR), from `res__remote_endpoint`. + + It is built from every `res__remote_endpoint` column that each answers-type download records. It excludes the pseudo-header rows (`Prompt`, `Correct answer`) and rows whose endpoint is NULL. An endpoint ending in `/`, a learner with no secure key, is emitted as NULL, by the same rule `student_id_mapping` applies to `run_remote_endpoint`. With no answers-type report it is empty with the same four columns. A missing CSV contributes zero rows, with the existing missing-file warning. +- **R3. The documented counts reproduce `_lib`'s.** For a run that has both its report and its answers pulled, these queries return what `_lib.reports.answers_sql` and `learners_sql` return: + - answers: `SELECT count(*) FROM run_answers WHERE run_id = `; + - learners: `SELECT count(DISTINCT e.user_id) FROM learner_endpoints e JOIN run_answers a USING (run_id, remote_endpoint) WHERE run_id = `. + + The tests use a synthetic dataset built so that each wrong answer differs from the right one. It has: + - a learner with answers in two assignments, so counting endpoints gives 2 where counting users gives 1; + - a learner with a bare `.../` endpoint, who must not join; + - an answer whose membership puts it in two runs, so it counts once per run; + - a second answers-type run for another class, which must not leak into the first run's counts. +- **R4. Freshness needs no new view.** `logs.run_id` and `logs.event_time` already exist. The guidance documents the per-run form `SELECT count(*) AS logs, max(event_time) AS log_freshness_at FROM logs WHERE run_id = `, the same names `counts.json` carries. +- **R5. Both views are documented where the drift guard checks.** That is `core.md`'s Views section and the researcher guide's table. Each entry gives the counting query of R3 and says why `learner_endpoints` counts by `user_id`. The `run_answers` entry also says that its `run_id` is membership (every run that holds the answer), while the store's own `_run_id` is only the run that last fetched it. Filtering `answers` by `_run_id` undercounts a run whose answers a later run re-fetched. + +### Package contract checks + +- **R6. The package rules are report-server's, and cc-data asks for them.** + - **Where the check runs.** `build` and `run` send the zip they built to `POST /api/v1/packages/validate` before writing or running anything. A refusal keeps report-server's code and message and exits 5, and nothing is written or run. + - **What cc-data keeps.** cc-data checks only what it must know to act: the manifest parses, the entrypoint is a file in the package, `expected_duration_seconds` is a positive integer (it sets `run`'s time bound), and `urls` and `clue_prepull` are read as written. It keeps no copy of report-server's limits. + - **Before the route exists.** If report-server answers 404 (the route is not deployed yet), `build` and `run` print a warning that the package is unchecked until publish, and continue. +- **R7. Applicability is report-server's, and cc-data asks for it.** cc-data has no glob matcher and no copy of the pattern fixture. `run` asks `POST /api/v1/packages/applies` (R12). +- **R8. The display cap is 65,536 bytes.** The cap is counted in UTF-8 bytes of `display.md`. One Go constant holds it, and its comment cites `final-design.md` section 10 as the only place the number is decided. A test pins both edges: 65,536 bytes passes and 65,537 is refused. This is one of the runner behaviors `run` reproduces (R11 to R14), which have no server to ask, since the runner lives on a VM. + +### `cc-data package init` + +- **R9.** `cc-data package init [dir]` (default `.`) creates `dir` if needed and writes `manifest.json` and `run.py`. It reads nothing in the directory except to check that neither file already exists; it refuses, changing nothing, if either does. + - The manifest has `name` (from `--name`, else the directory's base name when that matches the name grammar; otherwise `--name` is required), `title` (the name), `version` `0.1.0`, a placeholder one-line `description`, `urls` with empty `all`, `any` and `none`, `clue_prepull` `false`, `entrypoint` `run.py` and `expected_duration_seconds` `300`. report-server accepts it. + - The stub is stdlib-only Python and is the smallest complete package. It reads `RD_SCOPE_FILE` and `RD_DATASET`. + - **Runs.** It reuses a Student Answers run already in its dataset for the scope's class (a report row with that `class_id` in `reports`, joined type-qualified to `downloads`). Otherwise it creates one with `cc-data reports create --report-slug student-answers --report-filter '{"class":[]}'`. + - **Pulls and counts.** It pulls with `get report --refresh` and `get answers`, then counts with the R3 queries through `cc-data query`, never by locating dataset files. + - **Guard.** It keeps the empty-scope guard that section 13 places in package code. + - **Output.** It writes `display.md`, `summary.txt` and `counts.json` to the scope's `output_dir`. + - **Limits it states.** A comment notes that it filters the whole class in one run (Athena fails above about four assignments; chunking is REPORT-149), and that it creates a run on the server the first time it meets a class. + - **Why the stub creates runs.** On the VM its dataset starts empty, so a stub that only read an existing run would report nothing there. The VM path is what a package exists for. + +### `cc-data package run` + +- **R10. Inputs.** `cc-data package run [dir] --dataset --scope `. + - `--dataset` names the researcher's own dataset, resolved like every other dataset ref. `package run` neither requires that it exists nor creates it, since a package creates and fills its own dataset on the VM too. + - `--scope` names a local scope file. It holds the four keys rigse supplies on the VM: `kind`, `id`, `classes` (`[{class_hash, class_id}]`) and `assignments` (`[{offering_id, runnable_id, name, url}]`). The shape is checked; any other key is refused. + - `package init` does not write a scope file, so the file comes from the author. +- **R11. The package sees the runner's layout.** Before the entrypoint runs: + - **A fresh run directory** under `dir/.cc-data-run/`, with `in/` and `out/`, both emptied. `.cc-data-run` and everything `package run` empties or creates under it must be a real directory, never a link. A link is refused before anything is removed, so a stray link can never point the emptying at a directory outside the package. + - **A `.gitignore` containing `*`** written into `dir/.cc-data-run/`. A package directory is usually a git checkout (cc-data-studies ignores only `local-data/`, `__pycache__/` and `*.pyc`), and `out/display.md` holds counts or text drawn from student data. + - **`in/scope.json`** in the runner's exact shape: the scope file's four keys; `clue_source` `"firebase"`; `dataset` as the ref's bare name, as the runner writes `pkg-`; and `output_dir` as the absolute `out/` path. + - **The runner's environment:** + - `RD_DATASET`: the full ref. + - `RD_SCOPE_FILE`. + - `RD_OUTPUT_DIR`. + - `RD_DATA_DIR` and `CC_DATA_LOCAL`: `dir/.cc-data-run/data/`. + - `CC_DATA_ROOT`: the data root `package run` itself resolved. + - `CC_DATA_PORTAL`: the ref's portal. + - `RD_REPORT_SERVER_URL`: the portal's stored credential's server, when one is stored. + - **A short passthrough list a laptop needs.** It covers `HOME`, `PATH`, `USER`, `LOGNAME`, `LANG`, `LC_*`, `TZ`, `TMPDIR`, `DBUS_SESSION_BUS_ADDRESS` and `XDG_RUNTIME_DIR`, plus Windows's `SYSTEMROOT`, `USERPROFILE`, `APPDATA`, `LOCALAPPDATA`, `PATHEXT` and `COMSPEC`. There is no Windows release, but CI builds and tests on `windows-2022`, and these variables are absent on Linux and macOS. These are the variables the package's own `cc-data` calls need to find the researcher's stored credential (the keychain or the credentials file) and to run at all. Nothing else is inherited. + - **No storage variables.** `RD_BUCKET` and `RD_STORAGE_PREFIX` are never set. +- **R12. Applicability before the entrypoint.** + - **The question asked.** When the manifest declares any pattern, `package run` sends the patterns and the scope's assignment URLs to `POST /api/v1/packages/applies`. report-server derives the interactive URLs inside those assignments, so the URLs matched are the ones the runner matches. + - **A refusal.** If the package does not apply, `run` refuses with report-server's reason, prefixed as the runner prefixes it ("the package does not apply to this class: "), and does not start the entrypoint. + - **Partial profiles.** Each URL the deriver could not read, and a truncated profile, is reported on stderr. + - **The call's timeout** is 5 minutes, since the deriver may spend up to 240 seconds; the client's 60-second default would cut it off. + - **Before the route exists (404).** `run` warns that applicability cannot be checked yet and that the VM's runner is the check, and runs the package. +- **R13. Execution.** + - **The interpreter.** A `.py` entrypoint runs with `python3.11` when it is on `PATH`, else with `python3`, after a one-line note that the VM runs 3.11. Any other entrypoint is executed directly. + - **The working directory** is a staged copy of exactly the files R17's `build` would include, at `dir/.cc-data-run/pkg/`, rebuilt on every run. The runner's working directory is the unpacked archive (`steps.js` `runPackage`, `cwd: manifest.dir`). A package that reads a file `build` leaves out, such as something under `local-data/`, therefore fails locally just as it would on the VM. The same collection refuses a link, so `run` and `build` cannot disagree about what the package is. + - **Output streams.** The package's stdout and stderr stream to `package run`'s stderr. + - **The time bound.** A run longer than `expected_duration_seconds` plus 10 minutes is killed and reported as having run past its bound, as the runner does. + - **No sandbox.** No network namespace, no other uid and no egress proxy: those are the VM's, and the story forbids needing researcher-dashboard infrastructure locally. +- **R14. The result, read as the runner reads it.** After the entrypoint exits 0, the output is read without following links: + - `display.md` is required, and is refused over R8's cap with its size and the limit (never truncated). + - `summary.txt` is optional, and its first line, trimmed, is the summary. + - `counts.json`, when present and valid, contributes `answers`, `logs` and `log_freshness_at` only. + + On success, stdout carries one JSON line: the display path and size in bytes, the summary, the counts and the elapsed seconds. +- **R15. A `clue_prepull: true` package is refused locally.** The message says CLUE data cannot be fetched by cc-data yet (section 13), so such a package runs only on the VM. +- **R16. Failures follow the CLI's exit-code contract.** + - A manifest cc-data cannot act on, an invalid scope file, or a package tree `build` cannot collect (a link, a non-executable entrypoint), is a usage error (exit 2, codes `INVALID_MANIFEST`, `INVALID_SCOPE`, `INVALID_PACKAGE`). + - A refusal from report-server's validate or applies route keeps the server's code and exits 5, as every server contract error does. + - A package the scope does not satisfy, or a `clue_prepull` package, is `PACKAGE_REFUSED`. + - A non-zero exit or a timeout is `PACKAGE_FAILED`. + - A missing, linked or oversized `display.md` is `PACKAGE_OUTPUT_REFUSED`. + + The last three exit 1 with the single JSON error envelope on stdout. + +### `cc-data package build` + +- **R17. Contents.** + - **What goes in:** `cc-data package build [dir] [--out ]` zips the regular files under `dir`. + - **What is left out:** any path with a segment beginning with `.` (which covers `.cc-data-run/`, `.git/` and `.DS_Store`), `__pycache__/`, `*.pyc` and `local-data/`. + - **What is refused:** a symbolic link, or any file that is neither regular nor a directory. Nothing outside `dir` is ever included, so no shared library can be vendored by the tool. + - **Before writing:** `manifest.json` must be at the root, it must pass R6, and the entrypoint must be one of the files collected. + - **Modes:** each file ships 0755 when any execute bit is set on disk, else 0644. `unzip` on the VM restores these, and the runner executes a non-`.py` entrypoint directly, so such an entrypoint must be executable or `run` and `build` refuse it. `run`'s staged copy uses the same modes. (Windows records no execute bit, so the check is skipped there.) +- **R18. Reproducible.** The same tree builds to the same bytes: entries in sorted order, a fixed modification time, and no extra attributes. The same package therefore always has the same checksum. +- **R19. `build` refuses what `publish` would refuse,** by asking report-server (R6). It takes `--portal` (default: the configured portal) and `--origin projects/` so the check is made as the publish will be. A refused archive is not written. +- **R20. Output.** + - The default output path is `dir/.cc-data-build/-.zip`. It sits inside an excluded dot directory, with the same `.gitignore` as R11's, so building from inside the package directory never feeds one build's zip into the next. + - An explicit `--out` inside `dir` is refused unless R17 would exclude that path anyway. + - stdout carries one JSON line: the path, `sha256:`, the size and the file count. + +### `cc-data package publish` + +- **R21. The request.** + - **The command:** `cc-data package publish [--portal ] [--origin projects/] [--official] [--json]`. + - **No local check:** publish sends the zip as it is. report-server's publish applies every rule, including the runner's stricter ones once the companion story lands, so a hand-made zip is held to the same rules as a built one. + - **The upload:** `POST /api/v1/packages` on the portal's report server, with the portal's stored cc-data token. The body is the zip bytes, sent with `Content-Type: application/zip` and `Content-Length`. `origin` and `official=true` are sent only when given. + - **Retries:** the request is never retried. + - **Never S3:** it never talks to S3. +- **R22. Success is checked, not assumed.** On 201, the response's `checksum` must equal the local `sha256:` of the zip; a mismatch is an error. Output is the response: as JSON with `--json`, otherwise one line each for identity, version, visibility, official, current version and checksum. +- **R23. Errors pass through.** A server error keeps its code and message: `ALREADY_EXISTS`, `FORBIDDEN`, `UNPROCESSABLE`, `BAD_REQUEST` and `SERVICE_UNAVAILABLE` exit 5, and `NOT_AUTHENTICATED` exits 3. A failure where the server never answered carries an action: publishing again is safe, and `ALREADY_EXISTS` then means the first attempt landed. + +### Documentation + +- **R24.** The following are updated: the README's command sketch, a new researcher-guide section on developing and publishing a package, and the skill header's command list. `cc-data package --help` explains the local/VM differences R11 and R13 leave (the passthrough variables, the interpreter, no sandbox). + +## Done when + +- A package made with `init`, edited and run with `package run` against a real dataset, produces `display.md`, `summary.txt` and `counts.json` under the runner's layout. +- `display.md` over 65,536 bytes fails `package run` at the same size the runner refuses. +- With the companion routes deployed: `build` and `run` refuse what report-server refuses, and `run` refuses a package report-server's matcher says does not apply. Before they are deployed, both warn and proceed. +- `cc-data package publish` creates a private package and a version row on staging, once staging's `PackageBuckets` is set. +- No package imports a shared library from its archive: `build` ships only the package's own files. +- Moved to REPORT-147 (Doug, 2026-10-08): a package built on a laptop runs on the VM and produces the same output. + +## Technical Notes + +- **Views live in `internal/duck/views.go`.** Add two builders to `viewSet.statements()`, after `runMembershipView` (`run_answers` reads `answers` and `run_membership`, so it must register after both). Use `vs.prefix` for every referenced view so multi-dataset sessions resolve to the right schema, as `perStoreView` does. `learner_endpoints` is built from the manifest (`dl.Columns`) at session open, so the per-class column list is known without a `DESCRIBE`. This is the one thing `_lib` needed a round trip for. +- **What the tests can see.** `StaticViewNames()` and the guard pick up both views automatically. `cmd/query.go`'s help and `internal/mcpserver/tools.go`'s `query` description list them with no edit. +- **Verified against real staging data.** On `learn.portal.staging.concord.org/staging-smoketest`, run 164: `_lib`'s answers SQL gives 5 and its learners SQL gives 2. The R3 forms, written as plain SQL over a hand-built `learner_endpoints` (`SELECT 164 AS run_id, user_id, res_1_offering_id, res_1_remote_endpoint FROM report_164`) and `run_answers`, give 5 and 2. `user_id` and `res__offering_id` scan as BIGINT and `res__remote_endpoint` as VARCHAR. Casts in the view keep that true for a CSV where a column is empty throughout. +- **The answers-type CSV also carries the class's shape**: `class_id`, and per assignment `res__name`, `res__offering_id` and `res__resource_url`. `runnable_id` and `class_hash` are not in it. A local scope therefore cannot be derived from a pulled report alone, which is why R10 takes the scope from a file. +- **report-server already calls report-service's `api` function** through `ReportServer.ReportService` (`server/lib/report_server/report_service.ex`), with the shared `REPORT_SERVICE_TOKEN`, for `bulk_read`, `fetch_attachment_meta`, `answer`, `plugin_states` and `resource`. The companion story's `derive_urls` route follows that same path, so `deriveProfile` stays the only copy of the walk and no new credential is created. +- **The publish request needs a new client method.** `internal/api/client.go`'s `do` always sends `Content-Type: application/json` with a body, so publishing needs a sibling that sends raw bytes with `application/zip`. `net/http` sets `Content-Length` for a `*bytes.Reader` body. `isIdempotent` already makes POST non-retried. `AsWriteCLIError` is the existing pattern for a write that must not invite a blind retry. +- **The Go `archive/zip` writer sets data-descriptor flag `0x8` and zero local-header sizes.** report-server's `Archive` reads offsets and sizes from the central directory for this reason (REPORT-142). The runner unpacks with `unzip`, which handles it. A reproducible build sets each header's `Modified` to a fixed time and uses `Deflate`. +- **cc-data releases.** The release target is `cc-data-cli 0.3.0` (the Jira fix version). The runner image pins `CC_DATA_REF=v0.2.0` (`runner/Dockerfile`), so the views reach a VM only when RD-4/RD-1 bump that pin to the release that carries them. The R11 environment adds nothing the runner does not already set. +- **What staging answers before `PackageBuckets` is set.** report-server validates the archive and manifest before it looks up the bucket (`Packages.publish`). Until the parameter is set for a portal server, a publish that passes every other check answers 422 `UNPROCESSABLE` "publishing is not configured for ". That answer is itself evidence that `publish`'s request was well formed. +- **Checked with throwaway code (stage 4), then deleted.** + - **The pattern fixture, for the companion story.** RD-4 pass 2's `fixtures/url-patterns.json` holds 27 match cases and 12 applicability cases (sha256 `35a994a1...c420`); the stream docs say 26. A Go port of the runner's matcher passed all of them, and making `?` a wildcard failed the `a?c` case, so the fixture catches that mutation. The port was then dropped with the decision to match only in report-server, where this fixture belongs. + - **The archive.** A Go `archive/zip` build with sorted entries, `Deflate` and a fixed `Modified` gives identical bytes after a source file's mtime changes. `unzip` restores each entry's mode, so a 0644 shell entrypoint is "Permission denied" after unpacking (R17's mode rule). `unzip -p` and Python's `zipfile` read it, and a symbolic link in the tree is refused during the walk. report-server's own tests already cover reading a Go-written zip (`server/test/support/fixtures/packages/class-counts-go-1.0.6.zip`). + - **The views.** Both views were added to a scratch copy of `views.go` and run through the real engine: + - On `staging-smoketest` they give 5 and 2 for run 164, as `_lib` does. + - They register empty, with the stated columns, on a dataset with no answers (`log-test`). + - They resolve per schema in a two-dataset session. + - `dataset materialize` writes `learner_endpoints.parquet` (it declares CSV files). `run_answers` stays a join over the materialized `answers` and `run_membership`, and the counts are unchanged after materializing. + - On a dataset whose CSV is malformed (`rd-inspect`, a pseudo-header row with one column too many), `learner_endpoints` fails at query time exactly as the existing `reports` view does. + - The full test suite passes apart from the two documentation guards, which name exactly the two new views. + - **One correction the prototype forced.** A CSV member's `run_id` is an integer literal and comes out INTEGER, so the view casts it to BIGINT to keep R2's types. +- **Out-of-band references.** The stream docs (`plan.md`, `fy26-sprint-26.md`, `speccing.md`, `final-design.md`) are global oob files under `streams/researcher-dashboard/`. RD-4 pass 2's throwaway build is branch oob `researcher-dashboard/RD-4-pass-2-pull-loop/stage-verification/`. + +## Out of Scope + +- The companion report-service story's routes and rule changes (Dependencies), and the dashboard app's and runner's use of them (RD-3, RD-4). +- MCP tools for the package commands; an agent can run the CLI. +- Changing a package's visibility, `official`, `archived` or `current_version` after publish (report-server's `POST /api/v1/packages/:kind/:owner_id/:name/:state`), and listing the catalog. +- Creating report runs or pulling data on a package's behalf, locally or on the VM. +- The VM's sandbox (network namespace, package uid, egress proxy) and the CLUE pre-pull. +- Bumping the runner image's cc-data pin; deleting `_lib` or the spike packages; a dashboard control that hands an author a ready-made scope file (RD-3). +- Recording the cc-data version in a package's result, which is the runner's image version (section 13). + +## Open Questions + + + +### RESOLVED: Which URLs does `package run` match a package's patterns against? +**Context**: The runner matches against the class profile's `assignment_urls` ∪ `interactive_urls`, while `scope.json` carries only assignment URLs, so a pattern on an interactive (the Wildfire case) would be refused locally and accepted on the VM. The profile is derived by report-service's function from public authoring JSON. +**Options considered**: +- A) A Go port of the deriver's walk in cc-data. +- B) An author-supplied `profile_urls` list in the scope file. +- C) Assignment URLs only. +- D) Ask report-server, which reaches report-service's existing deriver. + +**Decision**: D, taken further (Doug, 2026-10-08): report-server holds the only matcher, and cc-data, the dashboard app and the runner all ask it (Dependencies, `POST /api/v1/packages/applies`). The deriver keeps its one copy, and there is no glob matcher anywhere else. The app and the runner already call report-service at startup, so the call adds no new dependency for either. Before the route is deployed, `run` warns and proceeds (R12). + +### RESOLVED: Where does the shared pattern fixture live? +**Decision**: The question dissolves under the decision above (Doug, 2026-10-08). With one matcher in report-server, `url-patterns.json` is an ordinary test file beside it in report-service, and nothing vendors a copy. cc-data has none. + +### RESOLVED: How is "a package built on a laptop runs in the VM and produces the same output" verified? +**Decision**: REPORT-147 owns it (Doug, 2026-10-08). Its done-when already requires the Wildfire package to run "identically under `cc-data package run` on a laptop and on a VM", and it is the first story that can put a package on a VM. REPORT-146 closes on the criteria in "Done when" below. + +### RESOLVED: Judgment call: doing applicability and the rules in report-server rather than in cc-data +**Options considered**: +- A) cc-data keeps Go copies of the manifest rules, the archive limits and the matcher, guarded by fixtures. +- B) cc-data asks report-server's validate and applies routes, keeping only what it needs to stage and start a package. + +**Decision**: B (Doug, 2026-10-08, to remove DRY violations without multiplying stories). The copies already disagreed: report-server allows 28,800 seconds where the runner refuses over 7,200. The companion story makes report-server adopt the runner's stricter rules, so its answer is the runner's. + +### RESOLVED: Judgment call: where a local scope comes from +**Options considered**: +- A) An author-supplied file (`--scope`) carrying the four keys rigse supplies. +- B) Synthesized from a pulled Student Answers report (`class_id`, and `res__offering_id`, `_name` and `_resource_url` per assignment). + +**Decision**: A. The report has no `class_hash` or `runnable_id`, and the scope contract has no room for nulls in either. Any synthesized scope would differ from the VM's in exactly the fields a package might key on. A dashboard control that hands out the real scope is the eventual source (RD-3, out of scope). + +### RESOLVED: Judgment call: a `clue_prepull` package under `package run` +**Options considered**: +- A) Refuse it locally (R15). +- B) Run it with `clue_source: "firebase"` and an empty corpus. + +**Decision**: A. B tells the package something false, and the package would produce numbers the VM would not. Section 13 already says a CLUE package is the one package that cannot be developed locally until REPORT-109 to 111. + +### RESOLVED: Low confidence: the passthrough environment in R11 +**Context**: The runner replaces the environment entirely. Locally, a package's own `cc-data` calls must find the researcher's credential: the OS keychain via `go-keyring`, or `~/.config/cc-data/credentials.json`. +**Options considered**: +- A) The R11 allowlist. +- B) Inherit the whole environment and set the runner's variables over it. + +**Decision**: A, verified on this machine (Linux, keyring backend). `env -i` with the R11 list, then `cc-data reports list --portal learn.concord.org --json`, reads the keyring token and lists runs. So does `env -i HOME PATH USER` alone: go-keyring reaches the session bus without `DBUS_SESSION_BUS_ADDRESS` here, so the D-Bus and XDG variables are kept only for setups that need them. On macOS go-keyring calls `/usr/bin/security`, which needs nothing beyond `HOME` and a `PATH` reaching `/usr/bin`. A cc-data query under the same `env -i` reads a dataset. B would let a package pass locally while depending on a variable the VM never sets. + +### RESOLVED: Low confidence: `scope.json`'s `dataset` is a bare name +**Context**: `final-design.md` section 10 shows `"dataset": ""`. RD-4 pass 2's `scopeFile` writes the bare `pkg-`, while `RD_DATASET` carries the full ref. A package that passes `scope["dataset"]` to `cc-data query --dataset` resolves it under the configured default portal, which may not be the dataset's. +**Options considered**: +- A) Mirror the runner (bare name), have the `init` stub read `RD_DATASET`, and record the discrepancy for RD-4 and the design to settle. +- B) Write the full ref locally, and ask RD-4 to change `scopeFile` to match section 10. + +**Decision**: A. This story's contract is "exactly as the runner", and B changes a contract RD-4 owns from a story that consumes it. If RD-4 changes `scopeFile`, R11's `dataset` follows it. Either way the `init` stub reads `RD_DATASET`, the one name that is a full ref on both sides. + +## Self-Review + +Roles: senior engineer, security engineer, QA engineer, education researcher, package author, operator. Every finding below was checked against the code before it was written down. Each had one defensible correction, which has been applied to the requirements above. + +### Security Engineer + +#### RESOLVED: Emptying the run directory could follow a link out of the package +R11 empties `dir/.cc-data-run/in` and `out` on every run. If `.cc-data-run` were a symbolic link to another directory, the emptying would delete under that target. Fixed in R11: every directory `package run` empties or creates must be a real directory, and a link is refused before anything is removed. + +#### RESOLVED: `publish` would upload a hand-made zip the runner then refuses +report-server checks entry paths but not entry types, and the runner refuses a symbolic link after unpacking (RD-4's `checkTree`). So a zip made outside `build` could publish and then fail on every VM. Resolved by the companion story: report-server refuses link entries at publish, so the one check covers every zip. + +### Education Researcher + +#### RESOLVED: Run output written inside a git checkout +`out/display.md` is derived from student data and lands inside the package directory. cc-data-studies' `.gitignore` ignores only `local-data/`, `__pycache__/` and `*.pyc` (read 2026-10-07), so `git add -A` would stage it. Fixed in R11 and R20: each working directory gets a `.gitignore` of `*`. + +#### RESOLVED: `_run_id` reads like run scoping and is not +The answers store stamps `_run_id` with the run that fetched each stored version (`internal/store/segment.go` `AppendPage`), and a later run's re-fetch overwrites it. Next to `run_answers.run_id`, a researcher could filter `answers` by `_run_id` and silently undercount. Fixed in R5's documentation requirement. + +### Package Author + +#### RESOLVED: The default build output would be swept into the next build +With the output in the current directory and `build` run from inside the package, as authors do, R20 as written either refused its own default or zipped the last build into the next. Fixed in R20: the default is `dir/.cc-data-build/`, which R17 already excludes. + +#### RESOLVED: An `init` stub that only reads existing runs reports nothing on the VM +On the VM a package's dataset (`pkg-`) starts empty, and section 13 makes run creation package code. A stub that assumed a pulled run would pass locally and count zero on its first VM run. Fixed in R9: the stub reuses or creates its Student Answers run, as `_lib/ccdata.py` and RD-4's smoke package do. + +### QA Engineer + +#### RESOLVED: R3 could pass with a fixture that cannot tell the queries apart +With one assignment per learner, counting endpoints and counting users give the same number, so a broken learner view would pass. Fixed in R3: the synthetic dataset includes a two-assignment learner, a bare endpoint, an answer in two runs and a second class's run. + +### Operator + +#### RESOLVED: Staging's answer before `PackageBuckets` exists was unstated +`Store.bucket_for` returns 422 "publishing is not configured for " after every archive and manifest check has passed. Recorded in Technical Notes, so that answer reads as a well-formed request rather than a failure of this story. From 962d7f58a5ec72ffb46c41eccc56ed99590af8b2 Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 05:42:00 -0400 Subject: [PATCH 02/17] docs: align the package spec with REPORT-167 and REPORT-147 [REPORT-146] The companion report-service story now exists as REPORT-167, so the spec names it where it said the story was still to be created. Dependencies gain the one contract fixture REPORT-167 holds, and init's name check is recorded as the one copy of that grammar cc-data keeps on purpose. REPORT-167's spec refined the validate route. A version that is already published is reported with already_published, and a portal that cannot store packages yet is reported with publishing_unavailable; neither is a refusal, so package run keeps working on a published version and on a portal without a bucket, staging included. Validate takes official as publish does. build now validates with --origin and --official as the publish will, warns on either condition and still writes the zip; run ignores both. The applies body always carries urls, which REPORT-167 requires, and neither route answers 404 on a server that has it, so a 404 still means only that the route is not deployed. REPORT-147 measured on production that a whole-class Student Answers run fails above three or four assignments and that a reused run's learner list goes stale, and counts learners through a Student ID Mapping run. The learner_endpoints view is therefore dropped: nothing would use it, and the existing student_id_mapping view joined to run_answers already counts learners by user_id. The guidance documents that count, and the init stub now reuses or creates a Student ID Mapping run for its class, re-reads it live and counts through run_answers and student_id_mapping. A second review round, checked against the plan's code in a scratch build, fixed six things. package run keeps an applies error's server code and exit class instead of reporting INTERNAL. Ctrl-C now cancels the run and kills the package's process group, which the terminal's own interrupt cannot reach. build refuses a manifest name or version that would turn its default output into a path. The proxy variables join the passthrough list, as the runner sets them. R18 and the zip comment admit the fixed extended-timestamp field, and publish drops its local copy of report-server's origin rule. The runner and REPORT-147 now pin a pre-release tag, v0.3.0-pre.1, cut after this story merges, rather than waiting for 0.3.0 (Doug, stream channel #21 and #22). R25 and a seventh plan step make that tag safe: release.yml classifies the tag once, marks a pre-release as one on GitHub with every archive and checksum attached, and skips the Homebrew formula for it, with a test pinning both uses. Following RD-4 pass 2's re-spec (stream channel #28 and #29): once the entrypoint exits, package run kills what is left of its process group before reading the output, as the runner kills every process of the package's uid. The passthrough's NO_PROXY is credited to the laptop, since the runner sets only the HTTP(S)_PROXY spellings, scope.json's bare dataset name is recorded as settled, and Done when no longer waits on staging's PackageBuckets, which is now set. --- .../implementation.md | 515 ++++++++++-------- .../requirements.md | 116 ++-- 2 files changed, 359 insertions(+), 272 deletions(-) diff --git a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md index d6800a0..a99fa5c 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md +++ b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md @@ -6,47 +6,38 @@ ## Implementation Plan -The plan comes in six commits. Each one builds and passes `go test ./...` on its own, in this order. All of the code below was compiled and tested in a scratch copy of `main` (`58c9172`) while this plan was written: the full suite passes, `go vet` is clean for Linux, and `GOOS=windows go vet` is clean for `internal/packages` and `internal/api`. A table under each step names mutations its tests catch; every one listed was applied and made a test fail. The scratch copy has been deleted. +The plan comes in seven commits. Each one builds and passes `go test ./...` on its own, in this order. All of the code below was compiled and tested in a scratch copy of `main` (`58c9172`) while this plan was written: the full suite passes, `go vet` is clean for Linux, and `GOOS=windows go vet` is clean for `internal/packages` and `internal/api`. A table under each step names mutations its tests catch; every one listed was applied and made a test fail. The scratch copy has been deleted. -The package rules and applicability are report-server's (requirements, Dependencies). cc-data calls `POST /api/v1/packages/validate` and `POST /api/v1/packages/applies`, which the companion report-service story adds. A 404 from either means an older report-server, and is a warning, not a failure. So this plan ships and works before that story lands, and gains its checks when it does. +The package rules and applicability are report-server's (requirements, Dependencies). cc-data calls `POST /api/v1/packages/validate` and `POST /api/v1/packages/applies`, which REPORT-167 adds. A 404 from either means an older report-server, and is a warning, not a failure. So this plan ships and works before that story lands, and gains its checks when it does. -### The run-scoped views: `run_answers` and `learner_endpoints` +### The run-scoped view `run_answers`, and the documented counts -**Summary**: R1 to R5. Adds two static views to `internal/duck/views.go`, documents them where the drift guards check, and tests the counts on a dataset where each wrong way of counting gives a different number. Static views are derived everywhere else, so `cmd/query.go`'s help and the MCP `query` description pick both up with no edit. +**Summary**: R1, R3 to R5. Adds the static view `run_answers` to `internal/duck/views.go`, and documents it with the learner count through the existing `student_id_mapping` view and the log-freshness query, where the drift guards check. It tests both counts on a dataset where each wrong way of counting gives a different number. Static views are derived everywhere else, so `cmd/query.go`'s help and the MCP `query` description pick it up with no edit. **Files affected**: -- `internal/duck/views.go`: two builders and a helper, registered after `runMembershipView`. +- `internal/duck/views.go`: one builder, registered after `runMembershipView`. - `internal/duck/run_views_test.go`: new. -- `internal/guidance/src/core.md`: two Views entries and the freshness query. -- `docs/researcher-guide.md`: two table rows. +- `internal/guidance/src/core.md`: the `run_answers` entry, the learner count, and the freshness query. +- `docs/researcher-guide.md`: one table row. -**Estimated diff size**: ~230 lines +**Estimated diff size**: ~170 lines ```diff ---- /tmp/o 2026-10-08 05:22:45.429149300 -0400 -+++ internal/duck/views.go 2026-10-07 17:23:24.424211223 -0400 -@@ -8,6 +8,7 @@ - "io" - "os" - "path/filepath" -+ "regexp" - "sort" - "strings" - "time" -@@ -71,6 +72,8 @@ +--- /tmp/o 2026-10-08 09:00:24.920084911 -0400 ++++ internal/duck/views.go 2026-10-08 09:00:14.224272861 -0400 +@@ -71,6 +71,7 @@ stmts = append(stmts, vs.storeView(store.TypeAnswers)) stmts = append(stmts, vs.storeView(store.TypeHistory)) stmts = append(stmts, vs.runMembershipView()) + stmts = append(stmts, vs.runAnswersView()) -+ stmts = append(stmts, vs.learnerEndpointsView()) stmts = append(stmts, vs.downloadsView()) stmts = append(stmts, vs.attachmentFilesView()) stmts = append(stmts, vs.attachmentStatesView()) -@@ -360,6 +363,88 @@ +@@ -360,6 +361,20 @@ return viewStmt{name: name, primary: primary, fallback: fallback, files: files} } -+// runAnswersView is every answer once per Student Answers run whose membership holds it: the ++// runAnswersView is every answer once per run whose answers fetch holds it in membership: the +// type-qualified membership join, registered on every dataset so a run with no answers counts +// zero rather than failing to bind the way answers_ does. It must follow both views it +// reads, and it declares no files, so materializing reads it through theirs. @@ -59,81 +50,13 @@ The package rules and applicability are report-server's (requirements, Dependenc + fallback := fmt.Sprintf("CREATE VIEW %s AS SELECT CAST(NULL AS BIGINT) AS run_id, s.* FROM %s s WHERE false", name, answers) + return viewStmt{name: name, primary: primary, fallback: fallback} +} -+ -+// resEndpointColumn is an answers report's per-assignment endpoint column; N is the -+// assignment's position in that run's list. -+var resEndpointColumn = regexp.MustCompile(`^res_([0-9]+)_remote_endpoint$`) -+ -+// learnerEndpointsView is one row per learner per assignment of every answers-type report CSV: -+// the Portal user, the assignment's offering and the endpoint that joins them to run_answers. -+// A student has one endpoint per assignment, which is why learners are counted by user_id. -+// Each CSV is scanned once, its assignment columns unnested as structs, because a real class -+// carries eight to fourteen of them. -+func (vs viewSet) learnerEndpointsView() viewStmt { -+ name := vs.prefix + `"learner_endpoints"` -+ standIn := fmt.Sprintf("CREATE VIEW %s AS SELECT CAST(NULL AS BIGINT) AS run_id, CAST(NULL AS BIGINT) AS user_id, CAST(NULL AS BIGINT) AS offering_id, CAST(NULL AS VARCHAR) AS remote_endpoint WHERE false", name) -+ var members, files []string -+ for _, dl := range vs.m.Downloads { -+ if dl.Type != "report" || dl.ReportType != dataset.ReportTypeAnswers || len(dl.Files) == 0 || dl.Columns == nil { -+ continue -+ } -+ structs := learnerStructs(dl) -+ if _, ok := dl.Columns["user_id"]; !ok || len(structs) == 0 { -+ continue -+ } -+ if fileMissing(filepath.Join(vs.canonDir, dl.Files[0])) { -+ vs.warnf("report CSV %s is missing on disk; contributing zero rows to learner_endpoints (run cc-data dataset reindex)", dl.Files[0]) -+ continue -+ } -+ members = append(members, fmt.Sprintf( -+ "SELECT CAST(run_id AS BIGINT) AS run_id, TRY_CAST(user_id AS BIGINT) AS user_id, unnest([%s]) AS a FROM (%s)", -+ strings.Join(structs, ", "), vs.csvScan(dl, true, nil))) -+ files = append(files, dl.Files...) -+ } -+ if len(members) == 0 { -+ return viewStmt{name: name, primary: standIn, fallback: standIn} -+ } -+ // A bare endpoint ending in "/" is a learner with no secure key, shared by every such -+ // learner, so it is withheld as in the dimension views rather than joined. -+ primary := fmt.Sprintf("CREATE VIEW %s AS SELECT run_id, user_id, a.offering_id AS offering_id, "+ -+ "CASE WHEN a.remote_endpoint LIKE %s THEN NULL ELSE a.remote_endpoint END AS remote_endpoint FROM (\n%s\n) WHERE a.remote_endpoint IS NOT NULL", -+ name, sqlStr("%/"), strings.Join(members, "\nUNION ALL\n")) -+ return viewStmt{name: name, primary: primary, fallback: standIn, files: files} -+} -+ -+// learnerStructs is one {offering_id, remote_endpoint} struct per assignment the CSV records, -+// in assignment order. A CSV without an assignment's offering_id gives that struct a NULL one. -+func learnerStructs(dl dataset.Download) []string { -+ type res struct { -+ n int -+ col string -+ } -+ var found []res -+ for col := range dl.Columns { -+ if m := resEndpointColumn.FindStringSubmatch(col); m != nil { -+ var n int -+ fmt.Sscanf(m[1], "%d", &n) -+ found = append(found, res{n, col}) -+ } -+ } -+ sort.Slice(found, func(i, j int) bool { return found[i].n < found[j].n }) -+ structs := make([]string, 0, len(found)) -+ for _, r := range found { -+ offering := "CAST(NULL AS BIGINT)" -+ if col := fmt.Sprintf("res_%d_offering_id", r.n); dl.Columns[col] != "" { -+ offering = fmt.Sprintf("TRY_CAST(%s AS BIGINT)", sqlIdent(col)) -+ } -+ structs = append(structs, fmt.Sprintf("{'offering_id': %s, 'remote_endpoint': CAST(%s AS VARCHAR)}", offering, sqlIdent(r.col))) -+ } -+ return structs -+} + // downloadsView is a VALUES dimension table from the manifest. // // hide_names separates the two meanings of a name column: where a run hid names, student_name ``` -`learner_endpoints` scans each CSV once and unnests one `{offering_id, remote_endpoint}` struct per assignment. Scanning once per assignment column would read a real class's CSV eight to fourteen times. The view declares its CSVs as files, so `dataset materialize` writes it. `run_answers` declares none, so materializing reads it through the materialized `answers` and `run_membership`. Both behaviors were run against a copy of `staging-smoketest`. +`run_answers` declares no files, so `dataset materialize` leaves it a join over the materialized `answers` and `run_membership` (checked against a copy of `staging-smoketest`). The learner count is tested here even though it adds no view, because the guidance presents it as the way to count, and a guidance query that silently stopped counting correctly is the failure this test exists to catch. ```go package duck @@ -147,122 +70,106 @@ import ( "github.com/concord-consortium/cc-data-cli/internal/store" ) +// The documented learner count, run-scoped on both sides, as the guidance gives it. +const learnersOfRun = "SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = %d AND m.run_id = %d" + // Each wrong way to count gives a different number here: counting endpoints instead of users -// (learner 1 answered in two assignments), joining the bare-endpoint learner (who would -// borrow learner 4's answer), counting an answer once instead of once per run (e3/q1 is in -// runs 584 and 585), leaking class 2's run into class 1's, and joining membership untyped -// (run 584's history holds e1/q1 too). -func TestRunAnswersAndLearnerEndpointsCountLikeLib(t *testing.T) { +// (user 1 answered in two assignments), joining the bare-endpoint learner (who would borrow +// another bare answer), counting an answer once instead of once per run (e3/q1 is in runs 700 +// and 800), and joining membership untyped (run 700's history holds e1a/q1 too). +func TestRunAnswersAndTheDocumentedLearnerCount(t *testing.T) { d := newDS(t, "ds") - buildStore(t, d, 584, [][]byte{ - answerRec("s", "https://p/d/e1", "q1", "a"), answerRec("s", "https://p/d/e1b", "q1", "b"), + buildStore(t, d, 700, [][]byte{ + answerRec("s", "https://p/d/e1a", "q1", "a"), answerRec("s", "https://p/d/e1b", "q1", "b"), answerRec("s", "https://p/d/e3", "q1", "c"), answerRec("s", "https://p/d/", "q1", "bare"), }) - hist, _ := json.Marshal(map[string]any{"source_key": "s", "remote_endpoint": "https://p/d/e1", "question_id": "q1", "history_id": "h1"}) - seg := store.OpenSegment(d.Dir, store.TypeHistory, 584) - if err := seg.AppendPage([][]byte{hist}, time.Unix(584, 0).UTC(), 584); err != nil { + hist, _ := json.Marshal(map[string]any{"source_key": "s", "remote_endpoint": "https://p/d/e1a", "question_id": "q1", "history_id": "h1"}) + seg := store.OpenSegment(d.Dir, store.TypeHistory, 700) + if err := seg.AppendPage([][]byte{hist}, time.Unix(700, 0).UTC(), 700); err != nil { t.Fatal(err) } seg.WriteCursor(&store.Cursor{Items: 1}) - if _, err := d.MergeCompact(store.TypeHistory, 584, seg); err != nil { + if _, err := d.MergeCompact(store.TypeHistory, 700, seg); err != nil { t.Fatal(err) } - buildStore(t, d, 585, [][]byte{answerRec("s", "https://p/d/e3", "q1", "c"), answerRec("s", "https://p/d/e9", "q1", "z")}) - addReportCSV(t, d, 584, "answers", "student_id,user_id,class_id,res_1_offering_id,res_1_remote_endpoint,res_2_offering_id,res_2_remote_endpoint\n"+ - "Prompt,,,,,,\nCorrect answer,,,,,,\n"+ - "10,1,1,7,https://p/d/e1,8,https://p/d/e1b\n"+ - "30,3,1,7,https://p/d/e3,8,\n"+ - "40,4,1,7,https://p/d/,8,\n") - addReportCSV(t, d, 585, "answers", "student_id,user_id,class_id,res_1_offering_id,res_1_remote_endpoint\n"+ - "Prompt,,,,\nCorrect answer,,,,\n30,3,2,9,https://p/d/e3\n90,9,2,9,https://p/d/e9\n") + buildStore(t, d, 800, [][]byte{answerRec("s", "https://p/d/e3", "q1", "c"), answerRec("s", "https://p/d/e9", "q1", "z")}) + row := func(learner, user, class, offering int, endpoint string) string { + return fmt.Sprintf("%d,%d,%d,s%d,%d,%d,https://act/1,%s", learner, user, user, user, class, offering, endpoint) + } + addDimensionCSV(t, d, dimFixture{run: 700, slug: "student-id-mapping", fetchedAt: at(1), filter: `{}`, csv: mappingCSV( + row(101, 1, 1, 70, "https://p/d/e1a"), row(102, 1, 1, 71, "https://p/d/e1b"), + row(103, 3, 1, 70, "https://p/d/e3"), row(104, 4, 1, 70, "https://p/d/"))}) + addDimensionCSV(t, d, dimFixture{run: 800, slug: "student-id-mapping", fetchedAt: at(2), filter: `{}`, csv: mappingCSV( + row(109, 9, 2, 90, "https://p/d/e9"))}) e := openEngine(t, []DatasetSpec{{DS: d}}, nil) - if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 584"); n != 4 { - t.Errorf("run 584 answers = %d, want 4", n) - } - if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 585"); n != 2 { - t.Errorf("run 585 answers = %d, want 2", n) + if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 700"); n != 4 { + t.Errorf("run 700 answers = %d, want 4", n) } - learners := "SELECT count(DISTINCT e.user_id) FROM learner_endpoints e JOIN run_answers a USING (run_id, remote_endpoint) WHERE run_id = %d" - if n := queryInt(t, e, fmt.Sprintf(learners, 584)); n != 2 { - t.Errorf("run 584 learners = %d, want 2 (users 1 and 3; user 4's endpoint is bare)", n) + if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 800"); n != 2 { + t.Errorf("run 800 answers = %d, want 2", n) } - if n := queryInt(t, e, fmt.Sprintf(learners, 585)); n != 2 { - t.Errorf("run 585 learners = %d, want 2", n) + if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 700, 700)); n != 2 { + t.Errorf("run 700 learners = %d, want 2 (users 1 and 3; user 4's endpoint is bare)", n) } - if n := queryInt(t, e, "SELECT count(*) FROM learner_endpoints WHERE remote_endpoint IS NULL AND user_id = 4"); n != 1 { - t.Errorf("the bare endpoint was not nulled") - } - if n := queryInt(t, e, "SELECT count(*) FROM learner_endpoints WHERE run_id = 584 AND offering_id = 8"); n != 1 { - t.Errorf("assignment 2's offering_id did not follow its endpoint") - } - var types string - rows, _ := e.Query(t.Context(), "SELECT string_agg(column_type, ',' ORDER BY column_name) FROM (DESCRIBE learner_endpoints)") - rows.Next() - rows.Scan(&types) - rows.Close() - if types != "BIGINT,VARCHAR,BIGINT,BIGINT" { - t.Errorf("learner_endpoints types (offering_id, remote_endpoint, run_id, user_id) = %s", types) + if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 800, 800)); n != 1 { + t.Errorf("run 800 learners = %d, want 1", n) } } -func TestRunViewsAreEmptyNotMissingOnAFreshDataset(t *testing.T) { +func TestRunAnswersIsEmptyNotMissingOnAFreshDataset(t *testing.T) { e := openEngine(t, []DatasetSpec{{DS: newDS(t, "ds")}}, nil) - for _, v := range []string{"run_answers", "learner_endpoints"} { - if n := queryInt(t, e, "SELECT count(*) FROM "+v); n != 0 { - t.Errorf("%s has %d rows", v, n) - } + if n := queryInt(t, e, "SELECT count(*) FROM run_answers"); n != 0 { + t.Errorf("run_answers has %d rows", n) } - if n := queryInt(t, e, "SELECT count(*) FROM (DESCRIBE learner_endpoints)"); n != 4 { - t.Errorf("learner_endpoints stand-in has %d columns, want 4", n) + if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 1, 1)); n != 0 { + t.Errorf("the learner count binds to %d on a fresh dataset, want 0", n) } } ``` | Mutation | Test that fails | |---|---| -| bare endpoints not nulled | learners for run 584 = 3 | -| `m.type = 'answers'` dropped | run 584 answers = 5 (the history membership joins) | -| offering taken from assignment 1 | "assignment 2's offering_id did not follow its endpoint" | -| `run_id` left uncast | the DESCRIBE type check (INTEGER) | +| `m.type = 'answers'` dropped | run 700 answers = 5 (the history membership joins) | +| the documented count by endpoint instead of `user_id` | run 700 learners = 3 | -Documentation (the two guards in `internal/guidance/guard_test.go` fail until both files carry the names; the dashes in `core.md` match that list's existing entries): +Documentation (the two guards in `internal/guidance/guard_test.go` fail until both files carry the name; the dashes in `core.md` match that list's existing entries): ```diff ---- /tmp/o 2026-10-08 05:22:45.441149321 -0400 -+++ internal/guidance/src/core.md 2026-10-07 17:21:28.050911252 -0400 -@@ -122,6 +122,23 @@ +--- /tmp/o 2026-10-08 09:00:24.902084889 -0400 ++++ internal/guidance/src/core.md 2026-10-08 09:00:14.224676445 -0400 +@@ -122,6 +122,24 @@ **type-qualified**: `answers a JOIN run_membership m USING (source_key, remote_endpoint, question_id) WHERE m.run_id = 584 AND m.type = 'answers'`. History joins add `history_id` to the USING list. -+- `run_answers` — every `answers` row with `run_id`, once for each Student Answers -+ run whose membership holds it, so one run's answers are `SELECT count(*) FROM ++- `run_answers` — every `answers` row with `run_id`, once for each run whose ++ answers fetch holds it (a Student Answers or a Student ID Mapping run), so one ++ run's answers are `SELECT count(*) FROM + run_answers WHERE run_id = 584`. It is the type-qualified membership join above, + always present, so a run with no answers counts zero rather than failing to bind + as `answers_` does. Its `run_id` is membership; the store's own `_run_id` is + only the run that last fetched the record, so filtering `answers` by `_run_id` + undercounts a run whose answers a later run re-fetched. -+- `learner_endpoints` — one row per learner per assignment in each answers-type -+ report CSV: `run_id`, `user_id` (the Portal user), `offering_id` and -+ `remote_endpoint`, built from every `res__remote_endpoint` column. A bare -+ endpoint ending in `/` (a learner with no secure key) is NULL, as in -+ `student_id_mapping`. Count a run's learners with an answer by `user_id`, never by -+ endpoint, since a student has one endpoint per assignment: `SELECT count(DISTINCT -+ e.user_id) FROM learner_endpoints e JOIN run_answers a USING (run_id, -+ remote_endpoint) WHERE run_id = 584`. ++- A run's learners with at least one answer come from its Student ID Mapping ++ report: `SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN ++ run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = 584 ++ AND m.run_id = 584`, where 584 is a Student ID Mapping run whose answers were ++ fetched. Count by `user_id`, never by endpoint, since a student has one endpoint ++ per assignment. A Student ID Mapping run is computed live, so re-reading it with ++ `--refresh` picks up learners who joined since; a Student Answers run is fixed ++ when its query ran, and a whole-class one fails above three or four assignments. +- A run's log freshness is `SELECT count(*) AS logs, max(event_time) AS + log_freshness_at FROM logs WHERE run_id = `. - Reports-to-stores join: `reports.res__remote_endpoint = answers.remote_endpoint`, with `res___*` pairing to `answers.question_id`. ---- /tmp/o 2026-10-08 05:22:45.444149326 -0400 -+++ docs/researcher-guide.md 2026-10-07 17:21:28.050652765 -0400 -@@ -346,6 +346,8 @@ +--- /tmp/o 2026-10-08 09:00:24.914084903 -0400 ++++ docs/researcher-guide.md 2026-10-08 08:57:22.784965017 -0400 +@@ -346,6 +346,7 @@ | `attachment_content` | The text/JSON content of every saved CODAP/SageModeler snapshot, queryable and diffable. | | `student_id_mapping` | One row per learner from Student ID Mapping runs, with the ids that join them to their answers and history. | | `student_metadata` | One row per learner from Student Metadata runs: name, username, class, school, teachers, permission forms. | -+| `run_answers` | Every answer with the `run_id` of each run that holds it: `SELECT count(*) FROM run_answers WHERE run_id = 584` counts one run's answers. | -+| `learner_endpoints` | One row per learner per assignment in each Student Answers report: `user_id`, `offering_id` and the `remote_endpoint` that joins them to `run_answers`. Count a run's learners by `user_id`, since one student has an endpoint per assignment. | ++| `run_answers` | Every answer with the `run_id` of each run that holds it: `SELECT count(*) FROM run_answers WHERE run_id = 584` counts one run's answers. Join it to `student_id_mapping` on `remote_endpoint = run_remote_endpoint` to count learners by `user_id`. | | `run_membership`, `downloads` | Provenance: which run's fetch covered which records, and what each download was, including whether its run hid names. | ### Speeding up a large dataset @@ -478,8 +385,9 @@ func Collect(dir string) (Files, error) { return files, err } -// Zip writes the files as a reproducible archive: sorted entries, Deflate, a fixed time, each -// file's ship mode, and no extra fields. The size limits are report-server's to apply. +// Zip writes the files as a reproducible archive: sorted entries, Deflate, a fixed time, and +// each file's ship mode. archive/zip also writes that time as an extended-timestamp extra +// field, which is as fixed as the time itself. The size limits are report-server's to apply. func Zip(files Files) ([]byte, error) { var buf bytes.Buffer w := zip.NewWriter(&buf) @@ -666,7 +574,7 @@ Tests (the link and mode tests skip on Windows, where links need privileges and - **Layout.** Rebuilds `.cc-data-run/` (`pkg/`, `in/`, `out/`, `data/` and a `.gitignore` of `*`) and writes the runner's `scope.json`. - **Applicability.** Asks an injected `Applies` function, which the commands step wires to report-server. - **Environment and interpreter.** Builds the runner's environment plus the passthrough list, and picks the interpreter. -- **Execution.** Runs the entrypoint from the staged copy in its own process group, and kills the group at the bound. +- **Execution.** Runs the entrypoint from the staged copy in its own process group, kills the group at the bound or on an interrupt, and kills what is left of it once the entrypoint exits. - **Result.** Reads it with `ReadResult`. **Files affected**: @@ -803,11 +711,14 @@ const TimeoutMargin = 10 * time.Minute const RunDirName = ".cc-data-run" // passthrough is what a laptop adds to the runner's environment so the package's own cc-data -// calls find the researcher's stored credential (keychain or credentials file) and run at -// all. The runner sets nothing beyond its own variables, so neither does this. +// calls find the researcher's stored credential (keychain or credentials file), reach the +// server, and run at all. The runner sets HTTPS_PROXY and HTTP_PROXY when it has a proxy, and +// NO_PROXY keeps a laptop's exemptions with them. Matching is case-insensitive, so the +// lower-case spellings pass too. var passthrough = []string{ "HOME", "PATH", "USER", "LOGNAME", "LANG", "TZ", "TMPDIR", "DBUS_SESSION_BUS_ADDRESS", "XDG_RUNTIME_DIR", + "HTTPS_PROXY", "HTTP_PROXY", "NO_PROXY", // Windows has no release, but CI builds and tests there. "SYSTEMROOT", "USERPROFILE", "APPDATA", "LOCALAPPDATA", "PATHEXT", "COMSPEC", } @@ -912,9 +823,16 @@ func Run(ctx context.Context, o RunOptions) (Result, error) { cmd.Stdout = o.Stderr cmd.Stderr = o.Stderr killGroup(cmd) - if err := cmd.Run(); err != nil { - if errors.Is(runCtx.Err(), context.DeadlineExceeded) { + err = cmd.Run() + // The runner kills everything the package left behind before it reads the output, so a + // background process can neither outlive the run nor write a later run's files. + reapGroup(cmd) + if err != nil { + switch { + case errors.Is(runCtx.Err(), context.DeadlineExceeded): return Result{}, &Failed{Reason: fmt.Sprintf("the package ran past its %ds bound", int(timeout.Seconds()))} + case ctx.Err() != nil: + return Result{}, &Failed{Reason: "the package was interrupted"} } return Result{}, &Failed{Reason: fmt.Sprintf("the package exited unsuccessfully: %v", err)} } @@ -1064,6 +982,14 @@ func killGroup(cmd *exec.Cmd) { cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) } } + +// reapGroup kills whatever is left of the package's process group once the entrypoint has +// exited. The group outlives its leader while any member does, so the kill still finds it. +func reapGroup(cmd *exec.Cmd) { + if cmd.Process != nil { + syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) + } +} ``` ```go //go:build windows @@ -1072,20 +998,24 @@ package packages import "os/exec" -// killGroup kills only the entrypoint on Windows, which has no release. +// killGroup and reapGroup kill only the entrypoint on Windows, which has no release. func killGroup(cmd *exec.Cmd) {} + +func reapGroup(cmd *exec.Cmd) {} ``` Tests (the entrypoint is a `/bin/sh` script, so these skip on Windows): - **`TestLocalScopeRefusesTheRunnersKeys`**: a scope file setting `dataset` is refused. - **`TestRunGivesThePackageTheRunnersLayout`**: - The package dumps its environment, copies `scope.json` and lists its working directory. - - The environment carries `RD_DATASET` (the full ref), `CC_DATA_PORTAL`, `RD_SCOPE_FILE` and a passed-through `LC_ALL`, and not an injected `AWS_SECRET_ACCESS_KEY`. + - The environment carries `RD_DATASET` (the full ref), `CC_DATA_PORTAL`, `RD_SCOPE_FILE` a passed-through `LC_ALL` and lower-case `https_proxy`, and not an injected `AWS_SECRET_ACCESS_KEY`. - `scope.json` has the bare `dataset`, `clue_source: "firebase"` and the absolute `output_dir`. - The working directory has no `local-data/`, and `.gitignore` is `*`. - **`TestRunRefusesBeforeStarting`**: when `Applies` says no, the error is the runner's prefix plus the reason, and the entrypoint never starts. A `clue_prepull` package is refused. - **`TestRunWarnsAndRunsWhenApplicabilityIsUnconfirmed`**: an unconfirmed verdict prints the warning and runs. - **`TestRunKillsAtTheBound`**: `sleep 30 & wait` under a 300 ms bound fails as "past its bound" within 5 seconds. +- **`TestRunLeavesNothingBehind`**: a package that backgrounds `(sleep 1; echo late > "$RD_OUTPUT_DIR/late.txt")` and exits 0 succeeds, and 1.5 seconds later `late.txt` does not exist. +- **`TestRunKillsTheGroupWhenInterrupted`**: the same package, with the parent context canceled after 300 ms, fails as "interrupted" within 5 seconds. - **`TestRunRefusesALinkedRunDirectory`**: `.cc-data-run` linked to another directory is refused, and a file in that directory survives. | Mutation | Test that fails | @@ -1095,6 +1025,8 @@ Tests (the entrypoint is a `/bin/sh` script, so these skip on Windows): | the whole environment inherited | `TestRunGivesThePackageTheRunnersLayout` | | run from the source directory, not the staged copy | `TestRunGivesThePackageTheRunnersLayout` | | a "does not apply" verdict ignored | `TestRunRefusesBeforeStarting` | +| a canceled context reported as an ordinary failure | `TestRunKillsTheGroupWhenInterrupted` | +| no `reapGroup` after the entrypoint exits | `TestRunLeavesNothingBehind` (the background job writes `late.txt`) | | an unconfirmed verdict treated as a refusal | `TestRunWarnsAndRunsWhenApplicabilityIsUnconfirmed` | --- @@ -1201,21 +1133,29 @@ func (c *Client) PublishPackage(ctx context.Context, archive []byte, origin stri } // ValidatedPackage is what a publish of the archive would record, as report-server's validate -// route answers it without recording anything. +// route answers it without recording anything. Neither of the last two is a refusal: both say +// a publish of this zip would be refused right now (the version exists, or the portal cannot +// store packages yet), while running it is fine. PublishingUnavailable is publish's own message. type ValidatedPackage struct { - Identity string `json:"identity"` - Version string `json:"version"` - Checksum string `json:"checksum"` - Visibility string `json:"visibility"` + Identity string `json:"identity"` + Version string `json:"version"` + Checksum string `json:"checksum"` + Visibility string `json:"visibility"` + AlreadyPublished bool `json:"already_published"` + PublishingUnavailable *string `json:"publishing_unavailable"` } // ValidatePackage asks report-server to apply every publish check to the archive and stop -// before storing it. A refusal is the same coded error a publish would answer. -func (c *Client) ValidatePackage(ctx context.Context, archive []byte, origin string) (*ValidatedPackage, error) { +// before storing it, with the same origin and official a publish would send. A refusal is the +// same coded error a publish would answer. +func (c *Client) ValidatePackage(ctx context.Context, archive []byte, origin string, official bool) (*ValidatedPackage, error) { q := url.Values{} if origin != "" { q.Set("origin", origin) } + if official { + q.Set("official", "true") + } data, err := c.send(ctx, http.MethodPost, "/api/v1/packages/validate", q, archive, "application/zip") if err != nil { return nil, err @@ -1270,7 +1210,7 @@ func RouteMissing(err error) bool { **Summary**: R9, R12, R16, R19 to R23. - **Wiring.** Adds the four subcommands to the root, and `Init` with the embedded stub. -- **The report-server calls.** `validate` is shared by `build` and `run`. `appliesFor` gives `run` its `Applies` function, with a 5-minute timeout for the deriver. +- **The report-server calls.** `validate` is shared by `build` and `run` and returns report-server's answer: `build` warns when `already_published` is true or `publishing_unavailable` names a reason, and `run` ignores both, since running a package needs neither a new version nor a bucket. `appliesFor` gives `run` its `Applies` function, with a 5-minute timeout for the deriver. - **Testability.** `build` and `publish` carry a `run(ctx, client, ...)` method, as `reportCreateFlags` does, so both are tested against `httptest` with no stored credential. **Files affected**: @@ -1291,6 +1231,7 @@ import ( "fmt" "os" "os/exec" + "os/signal" "path/filepath" "strings" "time" @@ -1408,12 +1349,17 @@ func newPackageRunCmd() *cobra.Command { if err != nil { return err } - ctx := context.Background() + // The package runs in its own process group, out of reach of the terminal's Ctrl-C, + // so an interrupt has to cancel the run for the group to be killed with it. + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) + defer stop() archive, err := packages.Zip(files) if err != nil { return output.Internalf("%v", err) } - if err := validate(ctx, client, archive, ""); err != nil { + // A version that is already published, or a portal that cannot store packages yet, + // still runs; only a publish is refused. + if _, err := validate(ctx, client, archive, "", false); err != nil { return err } start := time.Now() @@ -1440,19 +1386,19 @@ func newPackageRunCmd() *cobra.Command { // fetching a scope's activities; the client's 60-second default would cut it off. const derivationTimeout = 5 * time.Minute -// validate asks report-server whether a publish of the archive would be accepted. A -// report-server without the route is not a refusal: the archive is unchecked, and saying so -// is the whole answer. -func validate(ctx context.Context, client *api.Client, archive []byte, origin string) error { - _, err := client.ValidatePackage(ctx, archive, origin) +// validate asks report-server whether a publish of the archive would be accepted, and returns +// its answer. A report-server without the route is not a refusal: the archive is unchecked, +// the answer is nil, and saying so is the whole result. +func validate(ctx context.Context, client *api.Client, archive []byte, origin string, official bool) (*api.ValidatedPackage, error) { + ans, err := client.ValidatePackage(ctx, archive, origin, official) switch { case api.RouteMissing(err): output.Warnf("report-server has no validate route yet, so this package is not checked against the catalog's rules until publish") - return nil + return nil, nil case err != nil: - return api.AsCLIError(err) + return nil, api.AsCLIError(err) } - return nil + return ans, nil } // appliesFor is package run's applicability check: report-server's one matcher, against the @@ -1485,7 +1431,10 @@ func packageRunError(err error) error { var refused *packages.Refused var failed *packages.Failed var out *packages.OutputRefused + var cliErr *output.CLIError switch { + case errors.As(err, &cliErr): + return cliErr case errors.As(err, &refused): return &output.CLIError{ExitCode: output.ExitInternal, Code: "PACKAGE_REFUSED", Message: err.Error()} case errors.As(err, &failed): @@ -1498,6 +1447,7 @@ func packageRunError(err error) error { type packageBuildFlags struct { out, portal, origin string + official bool } // run builds, validates and writes; it takes the client so it can be exercised against a test @@ -1509,7 +1459,13 @@ func (f packageBuildFlags) run(ctx context.Context, client *api.Client, dir stri } out := f.out if out == "" { - out = filepath.Join(dir, packages.BuildDirName, fmt.Sprintf("%s-%s.zip", m.Name, m.Version)) + // The manifest is unchecked until validate answers, and unchecked for good on a server + // without the route, so its name and version must not be able to steer the path. + name := fmt.Sprintf("%s-%s.zip", m.Name, m.Version) + if strings.ContainsAny(name, `/\`) { + return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_MANIFEST", Message: fmt.Sprintf("the manifest's name and version make %q, which is not a file name; fix them or pass --out", name)} + } + out = filepath.Join(dir, packages.BuildDirName, name) } else if err := packages.CheckOutsidePackage(dir, out); err != nil { return output.Usagef("%v", err) } @@ -1517,9 +1473,16 @@ func (f packageBuildFlags) run(ctx context.Context, client *api.Client, dir stri if err != nil { return output.Internalf("%v", err) } - if err := validate(ctx, client, archive, f.origin); err != nil { + ans, err := validate(ctx, client, archive, f.origin, f.official) + if err != nil { return err } + if ans != nil && ans.AlreadyPublished { + output.Warnf("%s %s is already published; publish will refuse it until the version changes", ans.Identity, ans.Version) + } + if ans != nil && ans.PublishingUnavailable != nil { + output.Warnf("this package cannot be published yet: %s", *ans.PublishingUnavailable) + } if err := packages.WriteBuild(out, archive); err != nil { return output.Internalf("%v", err) } @@ -1545,6 +1508,7 @@ func newPackageBuildCmd() *cobra.Command { cmd.Flags().StringVar(&f.out, "out", "", "where to write the zip (default: /.cc-data-build/-.zip)") cmd.Flags().StringVar(&f.portal, "portal", "", "portal whose catalog rules to check against: an environment alias or a hostname") cmd.Flags().StringVar(&f.origin, "origin", "", "projects/, when the package will be published as a project's") + cmd.Flags().BoolVar(&f.official, "official", false, "check it as an official publish will be checked (publisher role only)") return cmd } @@ -1581,9 +1545,6 @@ func newPackagePublishCmd() *cobra.Command { "--official needs the publisher role. The request is never retried.", Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { - if f.origin != "" && !strings.HasPrefix(f.origin, "projects/") { - return output.Usagef("--origin must be projects/; your own packages need no origin") - } archive, err := os.ReadFile(args[0]) if err != nil { return output.Usagef("reading %s: %v", args[0], err) @@ -1718,11 +1679,10 @@ reads the scope from RD_SCOPE_FILE, pulls its own data into RD_DATASET with cc-d through cc-data's views, and writes display.md, summary.txt and counts.json to the scope's output_dir. Stdlib only: the VM installs no Python packages. -Two limits to know before building on it: -- It creates a Student Answers run for the class the first time it meets one, on the server - the dataset's portal names. Later runs reuse that run and re-pull it. -- It filters the whole class in one run. Athena fails a whole-class Student Answers run above - about four assignments; chunking by scope["assignments"] is the fix. +It reads the class through a Student ID Mapping run, which the portal computes on every +download, so re-reading it picks up learners who joined since, and which has no limit on how +many assignments a class has. It reuses the researcher's own run for the class when there is +one and otherwise creates one, on the server the dataset's portal names. """ import json import os @@ -1742,19 +1702,18 @@ def query(dataset, sql): return json.loads(cc_data("query", "--dataset", dataset, sql, "--format", "json")) -def answers_run(dataset, class_id): - """The class's Student Answers run: one already in the dataset, else a new one.""" - # downloads first: until a report is pulled, the reports view has no class_id to filter on. - pulled = query(dataset, "SELECT run_id FROM downloads WHERE type = 'report' AND slug = 'student-answers'") - if pulled: - ids = ", ".join(str(int(row["run_id"])) for row in pulled) - found = query(dataset, f"SELECT max(run_id) AS run_id FROM reports WHERE run_id IN ({ids}) AND class_id = {int(class_id)}") - if found and found[0]["run_id"] is not None: - return int(found[0]["run_id"]) +def mapping_run(portal, class_id): + """The newest Student ID Mapping run filtered to exactly this class, else a new one.""" + runs = json.loads(cc_data("reports", "list", "--portal", portal, "--json"))["runs"] + mine = [r["run_id"] for r in runs + if r.get("slug") == "student-id-mapping" + and (r.get("report_filter") or {}).get("filters") == ["class"] + and (r.get("report_filter") or {}).get("class") == [class_id]] + if mine: + return max(int(run) for run in mine) created = json.loads(cc_data( - "reports", "create", "--portal", os.environ["CC_DATA_PORTAL"], - "--report-slug", "student-answers", - "--report-filter", json.dumps({"class": [int(class_id)]}), "--json")) + "reports", "create", "--portal", portal, "--report-slug", "student-id-mapping", + "--report-filter", json.dumps({"class": [class_id]}), "--json")) return int(created["run"]["run_id"]) @@ -1763,24 +1722,26 @@ def main(): scope = json.load(handle) # RD_DATASET is the full / ref; scope["dataset"] is the bare name. dataset = os.environ["RD_DATASET"] - class_id = scope["classes"][0]["class_id"] + class_id = int(scope["classes"][0]["class_id"]) try: cc_data("dataset", "create", dataset) except RuntimeError as err: if "exists" not in str(err).lower(): raise - run = answers_run(dataset, class_id) + run = mapping_run(os.environ["CC_DATA_PORTAL"], class_id) cc_data("get", "report", str(run), "--dataset", dataset, "--refresh") cc_data("get", "answers", str(run), "--dataset", dataset) - rows = query(dataset, f"SELECT count(*) AS n FROM report_{run}")[0]["n"] + rows = query(dataset, f"SELECT count(*) AS n FROM student_id_mapping WHERE run_id = {run}")[0]["n"] if rows == 0: raise RuntimeError("no report-service access to this class, or the class has no data") answers = query(dataset, f"SELECT count(*) AS n FROM run_answers WHERE run_id = {run}")[0]["n"] + # By user_id, never by endpoint: a student has one endpoint per assignment. learners = query(dataset, f""" - SELECT count(DISTINCT e.user_id) AS n FROM learner_endpoints e - JOIN run_answers a USING (run_id, remote_endpoint) WHERE run_id = {run}""")[0]["n"] + SELECT count(DISTINCT m.user_id) AS n FROM student_id_mapping m + JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint + WHERE a.run_id = {run} AND m.run_id = {run}""")[0]["n"] out = scope["output_dir"] with open(os.path.join(out, "display.md"), "w", encoding="utf-8") as handle: @@ -1799,10 +1760,10 @@ if __name__ == "__main__": sys.exit(1) ``` -The stub was run through `Run` against a copy of `staging-smoketest`, using a wrapper `cc-data` that forwards `dataset` and `query` to the scratch build and turns `get` into a no-op. -- **The result.** It wrote "5 answers / 2 learners with at least one answer" and `counts.json` `{"answers": 5}`. -- **On an empty dataset** it called `reports create --report-slug student-answers --report-filter {"class": [6]}`. -- **Why it reads `downloads` first.** An earlier draft queried `reports.class_id` before any report existed, and the empty `reports` stand-in has no such column. +The stub was run through `Run` against a synthetic dataset holding a Student ID Mapping run and its answers, with a wrapper `cc-data` that fakes `reports list`, `reports create` and `get`, and forwards `dataset` and `query` to the scratch build. +- **The run it picks.** Out of a `reports list` holding a Student ID Mapping run filtered by class and school, one filtered by the class alone, a Student Answers run and another class's run, it took the class-only Student ID Mapping run. +- **The result.** It wrote "3 answers / 2 learners with at least one answer" and `counts.json` `{"answers": 3}`: a learner in two assignments counted once, and a learner with no answers not at all. +- **With no matching run** it called `reports create --report-slug student-id-mapping --report-filter {"class": [6]}`. With nothing pulled, it stopped at the empty-scope guard with its message. Tests (against an `httptest` server that answers 404 for any route it was not given): - **`TestPackageInitRefusesASecondInit`**. @@ -1811,10 +1772,18 @@ Tests (against an `httptest` server that answers 404 for any route it was not gi - The zip is `wildfire-responses-0.1.0.zip` with two files, and the printed checksum is the archive's own. - A second build from inside the package gives the same checksum. - **`TestPackageBuildWritesNothingTheServerRefuses`**: a 422 `UNPROCESSABLE` keeps its code, exits 5, and no zip is written. +- **`TestPackageBuildWarnsOnAPublishedVersionAndStillWrites`**: `already_published: true` prints "users/7/p 0.1.0 is already published; publish will refuse it until the version changes", and the zip is still written. **`TestPackageBuildIsQuietOnANewVersion`** checks the converse. +- **`TestPackageBuildValidatesAsThePublishWillBe`**: `--origin projects/20 --official` reach validate's query, and `official` is absent without the flag. +- **`TestValidateReturnsAPublishedVersionAsAnAnswerNotAnError`**: the shared `validate` returns the answer, so `run` proceeds on a published version. +- **`TestPackageBuildWarnsWhenThePortalCannotPublishAndStillWrites`**: `publishing_unavailable: "publishing is not configured for x"` prints "this package cannot be published yet: publishing is not configured for x", and the zip is still written. **`TestPackageBuildIsQuietWhenPublishingIsAvailable`** checks `null`. +- **`TestValidateReturnsAnUnavailablePortalAsAnAnswerNotAnError`**: so `run` proceeds on a portal with no bucket yet, such as staging before `PackageBuckets` is set. - **`TestPackageBuildOnAServerWithoutTheRouteStillBuilds`**: a 404 is a warning. - **`TestPackageBuildRefusesAnOutputItWouldCollect`**. +- **`TestPackageBuildRefusesANameThatIsAPath`**: on a server without the validate route, a manifest named `../../escaped` is `INVALID_MANIFEST`, and no zip is written anywhere. +- **`TestPackageRunKeepsTheServersCode`**: `packageRunError` given applies' 503 `SERVICE_UNAVAILABLE` returns that code with exit 5, and given `NOT_AUTHENTICATED` exit 3. - **`TestPackageRunAsksTheServersMatcher`**: the request carries the patterns and the assignment URLs, and a "does not apply" answer comes back as the verdict. - **`TestPackageRunOnAServerWithoutTheRouteIsUnconfirmed`**. +- **`TestPackageRunAlwaysSendsURLs`**: the applies body always carries `urls`, which REPORT-167 requires (a missing or `null` `urls` is 400 there). - **`TestPackagePublishSendsTheZipAsTheBody`**: checks POST, `application/zip`, `Content-Length`, `origin=projects/20` present, `official` absent, and the bearer. - **`TestPackagePublishRefusesAChecksumItDidNotSend`**. - **`TestPackagePublishPassesTheServersCodeThrough`**: a 409 `ALREADY_EXISTS` keeps its code, exits 5, and is sent once. @@ -1824,6 +1793,14 @@ Tests (against an `httptest` server that answers 404 for any route it was not gi | a 404 from validate treated as a refusal | `TestPackageBuildOnAServerWithoutTheRouteStillBuilds` | | `build` skips validate | `TestPackageBuildValidatesThenWrites`, `...WritesNothingTheServerRefuses` | | assignment URLs not sent to applies | `TestPackageRunAsksTheServersMatcher` | +| no already-published warning | `TestPackageBuildWarnsOnAPublishedVersionAndStillWrites` | +| no unavailable-portal warning | `TestPackageBuildWarnsWhenThePortalCannotPublishAndStillWrites` | +| the warning printed when `publishing_unavailable` is `null` | `TestPackageBuildIsQuietWhenPublishingIsAvailable` | +| `publishing_unavailable` read under the wrong name | `TestPackageBuildWarnsWhenThePortalCannotPublishAndStillWrites`, `TestValidateReturnsAnUnavailablePortalAsAnAnswerNotAnError` | +| `official` not sent to validate | `TestPackageBuildValidatesAsThePublishWillBe` | +| `packageRunError` without its `CLIError` case | `TestPackageRunKeepsTheServersCode` (exit 1, `INTERNAL`) | +| the default output name not checked | `TestPackageBuildRefusesANameThatIsAPath` (the zip lands two directories up) | +| `already_published` read under the wrong name | `TestPackageBuildWarnsOnAPublishedVersionAndStillWrites`, `TestValidateReturnsAPublishedVersionAsAnAnswerNotAnError` | --- @@ -1846,6 +1823,77 @@ Tests (against an `httptest` server that answers 404 for any route it was not gi **Estimated diff size**: ~90 lines +--- + +### Pre-release tags in the release workflow + +**Summary**: R25. `.github/workflows/release.yml` classifies the tag once, and the release and formula steps both read that answer: a pre-release tag (a `-` after the version, as in `v0.3.0-pre.1`) creates a GitHub pre-release with every archive and checksum, and skips the Homebrew formula. The archives upload inside `gh release create` itself, before the formula step, so skipping the formula cannot lose them. A test in `release_test.go`, beside the workflow's other guards, pins both uses. + +**Files affected**: +- `.github/workflows/release.yml`: one step, one flag, one condition. +- `release_test.go`: one test. +- `README.md`: one sentence in the release paragraph. + +**Estimated diff size**: ~40 lines + +```diff + - name: Create GitHub release + env: + GH_TOKEN: ${{ github.token }} + run: | + # Attach the checksum manifests alongside the tarballs so consumers can + # verify downloads independently. + gh release create "${GITHUB_REF_NAME}" dist/*.tar.gz dist/*.sha256 \ +- --title "${GITHUB_REF_NAME}" --generate-notes ++ --title "${GITHUB_REF_NAME}" --generate-notes \ ++ --prerelease=${{ steps.tag.outputs.prerelease }} + + - name: Render and push the Homebrew formula ++ if: steps.tag.outputs.prerelease == 'false' + env: +``` + +and, before "Create GitHub release": + +```yaml + # A tag with a semver pre-release suffix (v0.3.0-pre.1) is published for pinning, + # by the runner image and package workflows, and never offered as the current + # release: GitHub does not mark it latest and Homebrew never sees it. + - name: Classify the tag + id: tag + run: | + if [[ "${GITHUB_REF_NAME}" == *-* ]]; then + echo "prerelease=true" >> "${GITHUB_OUTPUT}" + else + echo "prerelease=false" >> "${GITHUB_OUTPUT}" + fi +``` + +```go +// A pre-release tag exists to be pinned, so it must not become GitHub's latest release or +// reach researchers through brew upgrade. Both steps read the one classification. +func TestReleaseWorkflowKeepsPreReleasesBack(t *testing.T) { + rel := readFile(t, ".github/workflows/release.yml") + for _, want := range []string{ + `if [[ "${GITHUB_REF_NAME}" == *-* ]]; then`, + "--prerelease=${{ steps.tag.outputs.prerelease }}", + "if: steps.tag.outputs.prerelease == 'false'", + } { + if !strings.Contains(rel, want) { + t.Fatalf("release workflow must hold a pre-release tag back from latest and from Homebrew; missing %q", want) + } + } +} +``` + +README, after the sentence naming the Homebrew formula: "A tag with a pre-release suffix, such as `v0.3.0-pre.1`, publishes a GitHub pre-release with the same archives and no formula, for pinning by the runner image and package workflows." + +Checked in a scratch copy: the workflow still parses, with the new step between "Verify checksums" and "Create GitHub release"; the classification writes `prerelease=true` for `v0.3.0-pre.1` and `false` for `v0.3.0`; and `gh release create` (2.100.0) parses `--prerelease=false` as a boolean, where `--prerelease=bogus` is refused. + +| Mutation | Test that fails | +|---|---| +| the formula step's condition removed | `TestReleaseWorkflowKeepsPreReleasesBack` | + ## Open Questions @@ -1912,3 +1960,28 @@ An author naming the file `scope.json` in the package directory would publish a #### RESOLVED: The applicability call would time out before the deriver finishes The deriver may spend up to 240 seconds on a scope's activities, and the client's per-attempt timeout for JSON calls is 60 seconds. `appliesFor` copies the client with a 5-minute `RequestTimeout`, which also keeps the shared client's default for every other call. + +### Second round + +Roles: senior engineer, package author, security engineer, operator, and a reviewer for one source of truth. Every finding was confirmed against `main`, against the plan's code extracted into a scratch copy, or against the runner and REPORT-167 sources, and each had one defensible fix, which has been applied above. The scratch copy built and passed `go test ./...` on Linux with the fixes, and `go vet` was clean for `internal/packages` under `GOOS=windows` and `GOOS=darwin`. + +#### RESOLVED: An applies failure lost its code and exit class (Senior Engineer) +`Run` returns `appliesFor`'s `*output.CLIError` unchanged, and `packageRunError` matched only the three package error types, so it fell through to `Internalf`. A throwaway test fed it applies' 503 `SERVICE_UNAVAILABLE` and `NOT_AUTHENTICATED`, and both came back exit 1 `INTERNAL`, against R16. `packageRunError` now returns a `CLIError` as it is, and `TestPackageRunKeepsTheServersCode` pins it. + +#### RESOLVED: Ctrl-C left the package running (Package Author) +`Setpgid` takes the package out of the terminal's foreground group, and cc-data had no interrupt handling outside `dataset materialize`. Sending `SIGINT` to cc-data's group, as a terminal does, ended cc-data while the package's `sleep 30` kept running. `package run` now cancels on `os.Interrupt` (the `materialize` idiom), which kills the group through the existing `cmd.Cancel`, and `Run` reports "the package was interrupted". Rerun with the fix, the package was gone a second after the signal. + +#### RESOLVED: `build`'s default path came from unchecked manifest fields (Security Engineer) +On a server without the validate route, a manifest named `../../escaped` built to `escaped-1.0.0.zip` two directories above the package, outside `.cc-data-build` and its `.gitignore`. `build` now refuses a name and version that make a path. It is a path check, not a copy of report-server's name grammar. + +#### RESOLVED: The proxy variables were not passed through (Operator) +cc-data's client uses Go's default transport, which reads `HTTPS_PROXY`, and the runner sets the proxy variables when it has a proxy (`package-env.js`). Behind a proxy, `package run`'s own requests would succeed while the package's `cc-data` calls failed. They are now on the passthrough list. + +#### RESOLVED: "No extra fields" in the zip was untrue (Senior Engineer) +`archive/zip` writes `Modified` as an extended-timestamp extra field (`5554...` on every entry of a scratch build). It is as fixed as the time, so R18 holds; the comment and R18 now say so. + +#### RESOLVED: `publish` kept a local copy of the origin rule (One Source of Truth) +`publish` refused an `--origin` without the `projects/` prefix, `build` did not, and report-server answers the same case with 422 "origin must be projects/; a user origin is always your own". The local check is deleted, so both commands give report-server's answer. + +Checked and found sound: the R3 test (dropping `m.run_id = ` fails it at run 800, 2 learners against 1); a second `get answers` on a fetched run re-fetches and merges new answers, so the stub stays current; `reports list --json`'s `report_filter` carries `filters` and `class` as the stub reads them; `query --format json` answers an array of objects; report-server's 270-second wait sits between the deriver's 240 and cc-data's 300; an old report-server answers both new routes with a JSON 404 even without a token; and no publish check refuses an ordinary user's own origin, so `run`'s validate call cannot lock out a researcher who may use the API. + diff --git a/specs/REPORT-146-package-commands-and-dataset-views/requirements.md b/specs/REPORT-146-package-commands-and-dataset-views/requirements.md index 6833484..f5c3321 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/requirements.md +++ b/specs/REPORT-146-package-commands-and-dataset-views/requirements.md @@ -7,15 +7,15 @@ ## Overview -Researchers get four `cc-data package` commands (`init`, `run`, `build`, `publish`) for writing a Researcher Dashboard package on a laptop, testing it the way the dashboard's runner will run it, and publishing it to the package catalog with the cc-data token they already have. Two new dataset views (`run_answers` and `learner_endpoints`) replace the counting SQL the spike copied into every package. +Researchers get four `cc-data package` commands (`init`, `run`, `build`, `publish`) for writing a Researcher Dashboard package on a laptop, testing it the way the dashboard's runner will run it, and publishing it to the package catalog with the cc-data token they already have. A new dataset view, `run_answers`, and documented joins on views cc-data already has replace the counting SQL the spike copied into every package. ## Project Owner Overview The Researcher Dashboard runs analysis "packages" for a class on a short-lived virtual machine. In the spike, a package was built and uploaded by a script that called AWS directly. Each package also carried a shared library of SQL describing how cc-data lays out a dataset. As a result, developing a package needed AWS credentials, and the only way to find a mistake was to run the package on a VM. -This story moves the whole loop onto the researcher's laptop and onto the token they already use for cc-data. `cc-data package run` runs a package against their own dataset under the runner's rules: the same `scope.json`, the same output files and the same size limit on the result. It asks report-server, rather than a copy of its rules, whether the package is valid and whether it applies to the class. An author therefore hits these failures on their laptop, not on a VM. `cc-data package publish` registers the package in the catalog in one step. The shared SQL becomes two named views in cc-data itself, so a package writes plain SQL against stable names and carries no library. The first package built this way is the Wildfire package (REPORT-147), whose spec waits on this one. +This story moves the whole loop onto the researcher's laptop and onto the token they already use for cc-data. `cc-data package run` runs a package against their own dataset under the runner's rules: the same `scope.json`, the same output files and the same size limit on the result. It asks report-server, rather than a copy of its rules, whether the package is valid and whether it applies to the class. An author therefore hits these failures on their laptop, not on a VM. `cc-data package publish` registers the package in the catalog in one step. The shared SQL becomes a named view and documented joins in cc-data itself, so a package writes plain SQL against stable names and carries no library. The first package built this way is the Wildfire package (REPORT-147), whose spec waits on this one. -**One rule, one home.** The package rules (what a valid package is) and applicability (whether a package suits a class) are decided by report-server alone. cc-data, the dashboard app and the runner ask it and do not keep their own copies. The new report-service routes this needs are one companion story (see Dependencies). +**One rule, one home.** The package rules (what a valid package is) and applicability (whether a package suits a class) are decided by report-server alone. cc-data, the dashboard app and the runner ask it and do not keep their own copies. The new report-service routes this needs are one companion story, REPORT-167 (see Dependencies). ## Background @@ -33,6 +33,9 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - `run_membership` (run_id, type, identity columns), `answers`, `history`, `reports`, `report_`, `answers_`. Each has a typed-empty fallback, so they always bind. - `logs`, with `run_id` and `event_time` (REPORT-112). The freshness query is already plain SQL against stable names, so query 3 needs no new view. +- `student_id_mapping`, one row per learner from Student ID Mapping runs, with `user_id` and `run_remote_endpoint` (a bare endpoint already nulled). Joined to a run's answers, it counts learners without double-counting a student who answered in two assignments, which is query 2's goal, so query 2 needs no new view either. + +**Why learners come from Student ID Mapping, not the Student Answers CSV** (REPORT-147's measurements on production, `bca55aa`, decided by Doug 2026-10-08). A whole-class Student Answers run fails above three or four assignments, and real Wildfire classes have 8 to 14. A reused Student Answers run's CSV is fixed when its query ran, so a learner who joins later has answers but no row. A Student ID Mapping run is a Portal report computed on every download, with no assignment limit, and its id fetches answers like any run. So the package route this story teaches is Student ID Mapping, and `_lib`'s Student Answers union (query 2) does not carry over. - A drift guard (`internal/guidance/guard_test.go`): every name in `StaticViewNames()` must be documented in `internal/guidance/src/core.md`'s Views section and in `docs/researcher-guide.md`'s "How datasets are organized" table, and the reverse. `cc-data query --help` and the MCP `query` tool description list the static views from the same function. **What the runner does to a package.** This is RD-4 pass 2 as spec'd and built in its stage-verification throwaway build (`stage5-throwaway-build.patch`, researcher-dashboard branch oob `RD-4-pass-2-pull-loop`). It is not implemented on any branch yet. @@ -73,46 +76,45 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se ## Dependencies -**A companion report-service story (to be created)** owns the routes cc-data calls. REPORT-146 codes against the contract below, which that story's spec may refine. Until those routes are deployed, `build` and `run` warn and proceed (R6, R12), so REPORT-146 is not blocked on it. +**REPORT-167** (report-service: one home for the package rules and applicability, created 2026-10-08) owns the routes cc-data calls. REPORT-146 codes against the contract below, which that story's spec may refine. Until those routes are deployed, `build` and `run` warn and proceed (R6, R12), so REPORT-146 is not blocked on it. -- **`POST /api/v1/packages/validate`** (cc-data token). The body is the raw zip, sent exactly as for publish, with `origin` as an optional query parameter. It runs every check `POST /api/v1/packages` runs, including permission and origin, and stops before writing any row or object. It answers 200 with `{identity, version, checksum, visibility}` (what a publish would record), or the same coded error a publish would give. +- **`POST /api/v1/packages/validate`** (cc-data token). The body is the raw zip, sent exactly as for publish, with `origin` and `official` as optional query parameters, as publish takes them. It runs every check `POST /api/v1/packages` runs, including permission and origin, and stops before writing any row or object. It answers 200 with `{identity, version, checksum, visibility, already_published}` (what a publish would record now), or the same coded error a publish would give for any other refusal. **A version that is already published is reported, not refused:** `already_published` is true and the answer is still 200, while only the publish itself answers `ALREADY_EXISTS` (REPORT-167's R12, refined after REPORT-146's first spec commit). **A portal that cannot store packages yet is reported, not refused:** `publishing_unavailable` is `null` or publish's own message (today only "publishing is not configured for ", before the portal has a bucket in `PackageBuckets`), the answer is still 200, and every other check still runs; only the publish answers that 422. Without this, `package run` would refuse every package on staging until its bucket is set (REPORT-167, `1ff2fc0`). The answer is thus `{identity, version, checksum, visibility, already_published, publishing_unavailable}`. - It is a separate route, not a `dry_run` parameter on publish. Today's publish ignores parameters it does not know, so a `dry_run=true` sent to a server without the feature would publish for real. A missing route answers 404. - **`POST /api/v1/packages/applies`** (cc-data token, or the researcher token the runner holds). The body is JSON: `{urls: {all, any, none}, assignment_urls: [...], scope_urls: [...]}`. - **Assignment URLs** are followed by report-service's existing deriver (`deriveProfile` in `functions/src/researcher-dashboard/derive-profile.ts`). report-server reaches it through a new `derive_urls` route on the `api` function, the way it already calls `bulk_read`. - **Scope URLs** are matched as given; that is the runner's case, since it already holds the class profile. + - **`urls` is required**: a missing or `null` `urls` is 400 `BAD_REQUEST`, while a `null` group inside it counts as empty. cc-data always sends it. - **The answer** is `{applies, reason, interactive_urls, unread: [{url, reason}], truncated}`. The `reason` uses the runner's wording. - **The matcher** behind it is the only glob matcher in the system. The app gets the same answer through the package-list call it already makes. +- **One contract fixture** of identity, version and package-key cases lives in report-service, asserted by report-server, the function, the runner and the app. cc-data's only copy of that grammar is `init`'s name check (R9), kept on purpose. +- **Neither route answers 404 on a report-server that has it** (REPORT-167). So a 404 still means only that the route is not deployed. A deriver failure answers 400 or 503, and report-server waits up to 270 seconds for the deriver, inside cc-data's 5-minute timeout. - **report-server adopts the runner's stricter rules,** so that anything it accepts the runner will run: `expected_duration_seconds` at most 7,200 (today 28,800), and no symbolic-link entries in an archive. ## Requirements ### Dataset views -- **R1. `run_answers`.** A static view with every row of `answers`, plus `run_id`: one row for each Student Answers run whose membership (`run_membership.type = 'answers'`) holds that answer. Its columns are `run_id` (BIGINT) followed by the `answers` columns. Like every static view, it registers on every dataset: with no answers store or no membership it is empty with the same columns, never a bind error. -- **R2. `learner_endpoints`.** A static view with one row per learner per assignment per answers-type report CSV. Its columns are: - - `run_id` (BIGINT); - - `user_id` (BIGINT), the Portal user; - - `offering_id` (BIGINT), from the same assignment's `res__offering_id`, NULL where the CSV has none; - - `remote_endpoint` (VARCHAR), from `res__remote_endpoint`. - - It is built from every `res__remote_endpoint` column that each answers-type download records. It excludes the pseudo-header rows (`Prompt`, `Correct answer`) and rows whose endpoint is NULL. An endpoint ending in `/`, a learner with no secure key, is emitted as NULL, by the same rule `student_id_mapping` applies to `run_remote_endpoint`. With no answers-type report it is empty with the same four columns. A missing CSV contributes zero rows, with the existing missing-file warning. -- **R3. The documented counts reproduce `_lib`'s.** For a run that has both its report and its answers pulled, these queries return what `_lib.reports.answers_sql` and `learners_sql` return: +- **R1. `run_answers`.** A static view with every row of `answers`, plus `run_id`: one row for each run whose answers fetch holds that answer in membership (`run_membership.type = 'answers'`), whether a Student Answers or a Student ID Mapping run. Its columns are `run_id` (BIGINT) followed by the `answers` columns. Like every static view, it registers on every dataset: with no answers store or no membership it is empty with the same columns, never a bind error. +- **R2. Dropped (Doug, 2026-10-08): no `learner_endpoints` view.** It would have unioned each Student Answers CSV's `res__remote_endpoint` columns, so it is empty for a Student ID Mapping run and rests on the route REPORT-147 measured as failing on real classes. Nothing uses it, so nothing builds it. +- **R3. The documented counts.** For a run whose answers were fetched: - answers: `SELECT count(*) FROM run_answers WHERE run_id = `; - - learners: `SELECT count(DISTINCT e.user_id) FROM learner_endpoints e JOIN run_answers a USING (run_id, remote_endpoint) WHERE run_id = `. + - learners with at least one answer, for a Student ID Mapping run: `SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = AND m.run_id = `, by `user_id` and never by endpoint, since a student has one endpoint per assignment. - The tests use a synthetic dataset built so that each wrong answer differs from the right one. It has: + The tests run both on a synthetic dataset built so that each wrong answer differs from the right one. It has: - a learner with answers in two assignments, so counting endpoints gives 2 where counting users gives 1; - a learner with a bare `.../` endpoint, who must not join; - an answer whose membership puts it in two runs, so it counts once per run; - - a second answers-type run for another class, which must not leak into the first run's counts. + - a history membership for one of the answers, which an untyped join would count; + - a second Student ID Mapping run for another class, which must not leak into the first run's counts. + - **R4. Freshness needs no new view.** `logs.run_id` and `logs.event_time` already exist. The guidance documents the per-run form `SELECT count(*) AS logs, max(event_time) AS log_freshness_at FROM logs WHERE run_id = `, the same names `counts.json` carries. -- **R5. Both views are documented where the drift guard checks.** That is `core.md`'s Views section and the researcher guide's table. Each entry gives the counting query of R3 and says why `learner_endpoints` counts by `user_id`. The `run_answers` entry also says that its `run_id` is membership (every run that holds the answer), while the store's own `_run_id` is only the run that last fetched it. Filtering `answers` by `_run_id` undercounts a run whose answers a later run re-fetched. +- **R5. `run_answers` is documented where the drift guard checks,** in `core.md`'s Views section and the researcher guide's table. The guidance gives R3's learner count through `student_id_mapping`, says why it counts by `user_id`, and says why a Student ID Mapping run is the route for answers and learners (computed live, no assignment limit). The `run_answers` entry also says that its `run_id` is membership (every run that holds the answer), while the store's own `_run_id` is only the run that last fetched it. Filtering `answers` by `_run_id` undercounts a run whose answers a later run re-fetched. ### Package contract checks - **R6. The package rules are report-server's, and cc-data asks for them.** - **Where the check runs.** `build` and `run` send the zip they built to `POST /api/v1/packages/validate` before writing or running anything. A refusal keeps report-server's code and message and exits 5, and nothing is written or run. - - **What cc-data keeps.** cc-data checks only what it must know to act: the manifest parses, the entrypoint is a file in the package, `expected_duration_seconds` is a positive integer (it sets `run`'s time bound), and `urls` and `clue_prepull` are read as written. It keeps no copy of report-server's limits. + - **What cc-data keeps.** Besides `init`'s name check (R9), which runs before report-server has seen anything and is one of the copies `final-design.md` section 17 records as deliberate, cc-data checks only what it must know to act: the manifest parses, the entrypoint is a file in the package, `expected_duration_seconds` is a positive integer (it sets `run`'s time bound), and `urls` and `clue_prepull` are read as written. It keeps no copy of report-server's limits. - **Before the route exists.** If report-server answers 404 (the route is not deployed yet), `build` and `run` print a warning that the package is unchecked until publish, and continue. - **R7. Applicability is report-server's, and cc-data asks for it.** cc-data has no glob matcher and no copy of the pattern fixture. `run` asks `POST /api/v1/packages/applies` (R12). - **R8. The display cap is 65,536 bytes.** The cap is counted in UTF-8 bytes of `display.md`. One Go constant holds it, and its comment cites `final-design.md` section 10 as the only place the number is decided. A test pins both edges: 65,536 bytes passes and 65,537 is refused. This is one of the runner behaviors `run` reproduces (R11 to R14), which have no server to ask, since the runner lives on a VM. @@ -122,11 +124,10 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - **R9.** `cc-data package init [dir]` (default `.`) creates `dir` if needed and writes `manifest.json` and `run.py`. It reads nothing in the directory except to check that neither file already exists; it refuses, changing nothing, if either does. - The manifest has `name` (from `--name`, else the directory's base name when that matches the name grammar; otherwise `--name` is required), `title` (the name), `version` `0.1.0`, a placeholder one-line `description`, `urls` with empty `all`, `any` and `none`, `clue_prepull` `false`, `entrypoint` `run.py` and `expected_duration_seconds` `300`. report-server accepts it. - The stub is stdlib-only Python and is the smallest complete package. It reads `RD_SCOPE_FILE` and `RD_DATASET`. - - **Runs.** It reuses a Student Answers run already in its dataset for the scope's class (a report row with that `class_id` in `reports`, joined type-qualified to `downloads`). Otherwise it creates one with `cc-data reports create --report-slug student-answers --report-filter '{"class":[]}'`. - - **Pulls and counts.** It pulls with `get report --refresh` and `get answers`, then counts with the R3 queries through `cc-data query`, never by locating dataset files. - - **Guard.** It keeps the empty-scope guard that section 13 places in package code. + - **Runs.** It takes the researcher's newest Student ID Mapping run filtered to exactly the scope's class (from `cc-data reports list --json`: slug `student-id-mapping`, `report_filter.filters` equal to `["class"]` and `report_filter.class` equal to `[]`), as REPORT-147's R7 does. Otherwise it creates one with `cc-data reports create --report-slug student-id-mapping --report-filter '{"class":[]}'`. + - **Pulls and counts.** It re-reads the run with `get report --refresh` (computed live, so learners who joined since appear) and pulls `get answers` on it, then counts with the R3 queries through `cc-data query`, never by locating dataset files. + - **Guard.** It keeps the empty-scope guard that section 13 places in package code: no `student_id_mapping` rows for the run is an error. - **Output.** It writes `display.md`, `summary.txt` and `counts.json` to the scope's `output_dir`. - - **Limits it states.** A comment notes that it filters the whole class in one run (Athena fails above about four assignments; chunking is REPORT-149), and that it creates a run on the server the first time it meets a class. - **Why the stub creates runs.** On the VM its dataset starts empty, so a stub that only read an existing run would report nothing there. The VM path is what a package exists for. ### `cc-data package run` @@ -147,7 +148,7 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - `CC_DATA_ROOT`: the data root `package run` itself resolved. - `CC_DATA_PORTAL`: the ref's portal. - `RD_REPORT_SERVER_URL`: the portal's stored credential's server, when one is stored. - - **A short passthrough list a laptop needs.** It covers `HOME`, `PATH`, `USER`, `LOGNAME`, `LANG`, `LC_*`, `TZ`, `TMPDIR`, `DBUS_SESSION_BUS_ADDRESS` and `XDG_RUNTIME_DIR`, plus Windows's `SYSTEMROOT`, `USERPROFILE`, `APPDATA`, `LOCALAPPDATA`, `PATHEXT` and `COMSPEC`. There is no Windows release, but CI builds and tests on `windows-2022`, and these variables are absent on Linux and macOS. These are the variables the package's own `cc-data` calls need to find the researcher's stored credential (the keychain or the credentials file) and to run at all. Nothing else is inherited. + - **A short passthrough list a laptop needs.** It covers `HOME`, `PATH`, `USER`, `LOGNAME`, `LANG`, `LC_*`, `TZ`, `TMPDIR`, `DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR` and the proxy variables `HTTPS_PROXY`, `HTTP_PROXY` and `NO_PROXY` in either case (the runner sets the first two when it has a proxy, and `NO_PROXY` keeps a laptop's exemptions with them), plus Windows's `SYSTEMROOT`, `USERPROFILE`, `APPDATA`, `LOCALAPPDATA`, `PATHEXT` and `COMSPEC`. There is no Windows release, but CI builds and tests on `windows-2022`, and these variables are absent on Linux and macOS. These are the variables the package's own `cc-data` calls need to find the researcher's stored credential (the keychain or the credentials file), to reach the server, and to run at all. Nothing else is inherited. - **No storage variables.** `RD_BUCKET` and `RD_STORAGE_PREFIX` are never set. - **R12. Applicability before the entrypoint.** - **The question asked.** When the manifest declares any pattern, `package run` sends the patterns and the scope's assignment URLs to `POST /api/v1/packages/applies`. report-server derives the interactive URLs inside those assignments, so the URLs matched are the ones the runner matches. @@ -160,6 +161,8 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - **The working directory** is a staged copy of exactly the files R17's `build` would include, at `dir/.cc-data-run/pkg/`, rebuilt on every run. The runner's working directory is the unpacked archive (`steps.js` `runPackage`, `cwd: manifest.dir`). A package that reads a file `build` leaves out, such as something under `local-data/`, therefore fails locally just as it would on the VM. The same collection refuses a link, so `run` and `build` cannot disagree about what the package is. - **Output streams.** The package's stdout and stderr stream to `package run`'s stderr. - **The time bound.** A run longer than `expected_duration_seconds` plus 10 minutes is killed and reported as having run past its bound, as the runner does. + - **Nothing left behind.** When the entrypoint exits, anything still running in its process group is killed before the output is read, as the runner kills every process of the package's uid (RD-4 pass 2's R28a). A background process the package started therefore neither outlives the run nor writes into a later one. A process that leaves the group (`setsid`, a daemon) escapes this locally, as it escapes the time bound. + - **An interrupt.** Ctrl-C (or `SIGINT`) during the run kills the package and everything it started, and is reported as interrupted. The package runs in its own process group so the bound can kill its children, which also puts it out of reach of the terminal's own Ctrl-C. - **No sandbox.** No network namespace, no other uid and no egress proxy: those are the VM's, and the story forbids needing researcher-dashboard infrastructure locally. - **R14. The result, read as the runner reads it.** After the entrypoint exits 0, the output is read without following links: - `display.md` is required, and is refused over R8's cap with its size and the limit (never truncated). @@ -170,9 +173,9 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - **R15. A `clue_prepull: true` package is refused locally.** The message says CLUE data cannot be fetched by cc-data yet (section 13), so such a package runs only on the VM. - **R16. Failures follow the CLI's exit-code contract.** - A manifest cc-data cannot act on, an invalid scope file, or a package tree `build` cannot collect (a link, a non-executable entrypoint), is a usage error (exit 2, codes `INVALID_MANIFEST`, `INVALID_SCOPE`, `INVALID_PACKAGE`). - - A refusal from report-server's validate or applies route keeps the server's code and exits 5, as every server contract error does. + - A refusal or failure from report-server's validate or applies route keeps the server's code and exit class: 5 for a contract error, 3 for `NOT_AUTHENTICATED`, as every server call does. - A package the scope does not satisfy, or a `clue_prepull` package, is `PACKAGE_REFUSED`. - - A non-zero exit or a timeout is `PACKAGE_FAILED`. + - A non-zero exit, a timeout or an interrupt is `PACKAGE_FAILED`. - A missing, linked or oversized `display.md` is `PACKAGE_OUTPUT_REFUSED`. The last three exit 1 with the single JSON error envelope on stdout. @@ -185,10 +188,10 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - **What is refused:** a symbolic link, or any file that is neither regular nor a directory. Nothing outside `dir` is ever included, so no shared library can be vendored by the tool. - **Before writing:** `manifest.json` must be at the root, it must pass R6, and the entrypoint must be one of the files collected. - **Modes:** each file ships 0755 when any execute bit is set on disk, else 0644. `unzip` on the VM restores these, and the runner executes a non-`.py` entrypoint directly, so such an entrypoint must be executable or `run` and `build` refuse it. `run`'s staged copy uses the same modes. (Windows records no execute bit, so the check is skipped there.) -- **R18. Reproducible.** The same tree builds to the same bytes: entries in sorted order, a fixed modification time, and no extra attributes. The same package therefore always has the same checksum. -- **R19. `build` refuses what `publish` would refuse,** by asking report-server (R6). It takes `--portal` (default: the configured portal) and `--origin projects/` so the check is made as the publish will be. A refused archive is not written. +- **R18. Reproducible.** The same tree builds to the same bytes: entries in sorted order, a fixed modification time (which Go's writer also records as an extended-timestamp extra field), and nothing that varies between builds. The same package therefore always has the same checksum. +- **R19. `build` refuses what `publish` would refuse,** by asking report-server (R6). It takes `--portal` (default: the configured portal), `--origin projects/` and `--official` so the check is made as the publish will be. A refused archive is not written. When validate answers `already_published`, `build` still writes the zip and warns " is already published; publish will refuse it until the version changes". When `publishing_unavailable` names a reason, `build` warns "this package cannot be published yet: " and still writes the zip. `run` ignores both, since running a package needs neither a new version nor a bucket. - **R20. Output.** - - The default output path is `dir/.cc-data-build/-.zip`. It sits inside an excluded dot directory, with the same `.gitignore` as R11's, so building from inside the package directory never feeds one build's zip into the next. + - The default output path is `dir/.cc-data-build/-.zip`. It sits inside an excluded dot directory, with the same `.gitignore` as R11's, so building from inside the package directory never feeds one build's zip into the next. A name or version holding a path separator is `INVALID_MANIFEST` rather than a path, since it is unchecked until validate answers, and for good on a server without the route. - An explicit `--out` inside `dir` is refused unless R17 would exclude that path anyway. - stdout carries one JSON line: the path, `sha256:`, the size and the file count. @@ -196,7 +199,7 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - **R21. The request.** - **The command:** `cc-data package publish [--portal ] [--origin projects/] [--official] [--json]`. - - **No local check:** publish sends the zip as it is. report-server's publish applies every rule, including the runner's stricter ones once the companion story lands, so a hand-made zip is held to the same rules as a built one. + - **No local check:** publish sends the zip as it is. report-server's publish applies every rule, including the runner's stricter ones once REPORT-167 lands, so a hand-made zip is held to the same rules as a built one. - **The upload:** `POST /api/v1/packages` on the portal's report server, with the portal's stored cc-data token. The body is the zip bytes, sent with `Content-Type: application/zip` and `Content-Length`. `origin` and `official=true` are sent only when given. - **Retries:** the request is never retried. - **Never S3:** it never talks to S3. @@ -207,42 +210,45 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - **R24.** The following are updated: the README's command sketch, a new researcher-guide section on developing and publishing a package, and the skill header's command list. `cc-data package --help` explains the local/VM differences R11 and R13 leave (the passthrough variables, the interpreter, no sandbox). +### Release + +- **R25. A pre-release tag is safe to cut** (Doug, 2026-10-08, stream channel #21 and #22). After this story merges, `main` is tagged `v0.3.0-pre.1`, and RD-4 pass 2's runner image and REPORT-147's publish workflow pin that exact tag, rather than waiting for 0.3.0, whose fix version also holds the CLUE export chain. + - **The release workflow tells the two kinds of tag apart.** A tag with a semver pre-release suffix (a `-` after the version) still builds, checks and uploads every archive and its `.sha256`, and creates a GitHub release, marked as a pre-release so GitHub never shows it as the latest. + - **Homebrew never sees it.** The formula step is skipped for a pre-release, so `brew upgrade` never hands it to researchers. + - **One classification.** Both steps read one decision made once in the workflow, and a test pins both uses. + - The change ships in this story's PR, so the first tag after merge is already safe. + ## Done when - A package made with `init`, edited and run with `package run` against a real dataset, produces `display.md`, `summary.txt` and `counts.json` under the runner's layout. - `display.md` over 65,536 bytes fails `package run` at the same size the runner refuses. -- With the companion routes deployed: `build` and `run` refuse what report-server refuses, and `run` refuses a package report-server's matcher says does not apply. Before they are deployed, both warn and proceed. -- `cc-data package publish` creates a private package and a version row on staging, once staging's `PackageBuckets` is set. +- With REPORT-167's routes deployed: `build` and `run` refuse what report-server refuses, and `run` refuses a package report-server's matcher says does not apply. Before they are deployed, both warn and proceed. +- `cc-data package publish` creates a private package and a version row on staging (whose `PackageBuckets` was set 2026-10-08, stream channel #34). - No package imports a shared library from its archive: `build` ships only the package's own files. +- After merge, `main` is tagged `v0.3.0-pre.1`, and its GitHub release is marked as a pre-release, carries `cc-data_0.3.0-pre.1_linux_amd64.tar.gz` and its `.sha256` (which REPORT-147's workflow downloads), and pushed no Homebrew formula. - Moved to REPORT-147 (Doug, 2026-10-08): a package built on a laptop runs on the VM and produces the same output. ## Technical Notes -- **Views live in `internal/duck/views.go`.** Add two builders to `viewSet.statements()`, after `runMembershipView` (`run_answers` reads `answers` and `run_membership`, so it must register after both). Use `vs.prefix` for every referenced view so multi-dataset sessions resolve to the right schema, as `perStoreView` does. `learner_endpoints` is built from the manifest (`dl.Columns`) at session open, so the per-class column list is known without a `DESCRIBE`. This is the one thing `_lib` needed a round trip for. -- **What the tests can see.** `StaticViewNames()` and the guard pick up both views automatically. `cmd/query.go`'s help and `internal/mcpserver/tools.go`'s `query` description list them with no edit. -- **Verified against real staging data.** On `learn.portal.staging.concord.org/staging-smoketest`, run 164: `_lib`'s answers SQL gives 5 and its learners SQL gives 2. The R3 forms, written as plain SQL over a hand-built `learner_endpoints` (`SELECT 164 AS run_id, user_id, res_1_offering_id, res_1_remote_endpoint FROM report_164`) and `run_answers`, give 5 and 2. `user_id` and `res__offering_id` scan as BIGINT and `res__remote_endpoint` as VARCHAR. Casts in the view keep that true for a CSV where a column is empty throughout. +- **Views live in `internal/duck/views.go`.** Add one builder to `viewSet.statements()`, after `runMembershipView` (`run_answers` reads `answers` and `run_membership`, so it must register after both). Use `vs.prefix` for every referenced view so multi-dataset sessions resolve to the right schema, as `perStoreView` does. +- **What the tests can see.** `StaticViewNames()` and the guard pick up the view automatically. `cmd/query.go`'s help and `internal/mcpserver/tools.go`'s `query` description list it with no edit. +- **Verified against real staging data.** On `learn.portal.staging.concord.org/staging-smoketest`, run 164: `_lib`'s answers SQL gives 5, and `run_answers` gives 5. REPORT-147 measured the Student ID Mapping learner join against the Student Answers union on production for both Wildfire classes (runs 2390 to 2393): the two count the same today. - **The answers-type CSV also carries the class's shape**: `class_id`, and per assignment `res__name`, `res__offering_id` and `res__resource_url`. `runnable_id` and `class_hash` are not in it. A local scope therefore cannot be derived from a pulled report alone, which is why R10 takes the scope from a file. -- **report-server already calls report-service's `api` function** through `ReportServer.ReportService` (`server/lib/report_server/report_service.ex`), with the shared `REPORT_SERVICE_TOKEN`, for `bulk_read`, `fetch_attachment_meta`, `answer`, `plugin_states` and `resource`. The companion story's `derive_urls` route follows that same path, so `deriveProfile` stays the only copy of the walk and no new credential is created. +- **report-server already calls report-service's `api` function** through `ReportServer.ReportService` (`server/lib/report_server/report_service.ex`), with the shared `REPORT_SERVICE_TOKEN`, for `bulk_read`, `fetch_attachment_meta`, `answer`, `plugin_states` and `resource`. REPORT-167's `derive_urls` route follows that same path, so `deriveProfile` stays the only copy of the walk and no new credential is created. - **The publish request needs a new client method.** `internal/api/client.go`'s `do` always sends `Content-Type: application/json` with a body, so publishing needs a sibling that sends raw bytes with `application/zip`. `net/http` sets `Content-Length` for a `*bytes.Reader` body. `isIdempotent` already makes POST non-retried. `AsWriteCLIError` is the existing pattern for a write that must not invite a blind retry. - **The Go `archive/zip` writer sets data-descriptor flag `0x8` and zero local-header sizes.** report-server's `Archive` reads offsets and sizes from the central directory for this reason (REPORT-142). The runner unpacks with `unzip`, which handles it. A reproducible build sets each header's `Modified` to a fixed time and uses `Deflate`. -- **cc-data releases.** The release target is `cc-data-cli 0.3.0` (the Jira fix version). The runner image pins `CC_DATA_REF=v0.2.0` (`runner/Dockerfile`), so the views reach a VM only when RD-4/RD-1 bump that pin to the release that carries them. The R11 environment adds nothing the runner does not already set. +- **cc-data releases.** The Jira fix version stays `cc-data-cli 0.3.0`, but the view and the package commands reach a VM through the pre-release tag `v0.3.0-pre.1` (R25). The runner image pins `CC_DATA_REF=v0.2.0` (`runner/Dockerfile`) and builds cc-data from the git tag, so RD-4 pass 2 raises that pin to the tag (channel #15 and #21). REPORT-147's publish workflow downloads the tag's linux/amd64 archive instead, which is why the pre-release must carry its assets. The R11 environment adds nothing the runner does not already set. - **What staging answers before `PackageBuckets` is set.** report-server validates the archive and manifest before it looks up the bucket (`Packages.publish`). Until the parameter is set for a portal server, a publish that passes every other check answers 422 `UNPROCESSABLE` "publishing is not configured for ". That answer is itself evidence that `publish`'s request was well formed. - **Checked with throwaway code (stage 4), then deleted.** - - **The pattern fixture, for the companion story.** RD-4 pass 2's `fixtures/url-patterns.json` holds 27 match cases and 12 applicability cases (sha256 `35a994a1...c420`); the stream docs say 26. A Go port of the runner's matcher passed all of them, and making `?` a wildcard failed the `a?c` case, so the fixture catches that mutation. The port was then dropped with the decision to match only in report-server, where this fixture belongs. + - **The pattern fixture, for REPORT-167.** RD-4 pass 2's `fixtures/url-patterns.json` holds 27 match cases and 12 applicability cases (sha256 `35a994a1...c420`); the stream docs say 26. A Go port of the runner's matcher passed all of them, and making `?` a wildcard failed the `a?c` case, so the fixture catches that mutation. The port was then dropped with the decision to match only in report-server, where this fixture belongs. - **The archive.** A Go `archive/zip` build with sorted entries, `Deflate` and a fixed `Modified` gives identical bytes after a source file's mtime changes. `unzip` restores each entry's mode, so a 0644 shell entrypoint is "Permission denied" after unpacking (R17's mode rule). `unzip -p` and Python's `zipfile` read it, and a symbolic link in the tree is refused during the walk. report-server's own tests already cover reading a Go-written zip (`server/test/support/fixtures/packages/class-counts-go-1.0.6.zip`). - - **The views.** Both views were added to a scratch copy of `views.go` and run through the real engine: - - On `staging-smoketest` they give 5 and 2 for run 164, as `_lib` does. - - They register empty, with the stated columns, on a dataset with no answers (`log-test`). - - They resolve per schema in a two-dataset session. - - `dataset materialize` writes `learner_endpoints.parquet` (it declares CSV files). `run_answers` stays a join over the materialized `answers` and `run_membership`, and the counts are unchanged after materializing. - - On a dataset whose CSV is malformed (`rd-inspect`, a pseudo-header row with one column too many), `learner_endpoints` fails at query time exactly as the existing `reports` view does. - - The full test suite passes apart from the two documentation guards, which name exactly the two new views. - - **One correction the prototype forced.** A CSV member's `run_id` is an integer literal and comes out INTEGER, so the view casts it to BIGINT to keep R2's types. + - **The view.** `run_answers` was added to a scratch copy of `views.go` and run through the real engine: 5 answers for run 164 on `staging-smoketest`, as `_lib` gives; empty with its columns on a dataset with no answers (`log-test`); resolved per schema in a two-dataset session; and, after `dataset materialize`, a join over the materialized `answers` and `run_membership` with unchanged counts. The full test suite passed apart from the two documentation guards, which named exactly the new view. (A `learner_endpoints` view was also prototyped and verified, then dropped with R2.) + - **The stub.** It was run through `package run`'s engine against a synthetic dataset with a Student ID Mapping run and its answers, with a wrapper `cc-data` faking only the server calls. Out of a `reports list` holding a run filtered by class and school, a Student Answers run and another class's run, it picked the class-only Student ID Mapping run, and wrote 3 answers and 2 learners (a learner in two assignments counted once, a learner with no answers not at all). With no matching run it created one with the class filter, and with no rows it stopped at the empty-scope guard's message. - **Out-of-band references.** The stream docs (`plan.md`, `fy26-sprint-26.md`, `speccing.md`, `final-design.md`) are global oob files under `streams/researcher-dashboard/`. RD-4 pass 2's throwaway build is branch oob `researcher-dashboard/RD-4-pass-2-pull-loop/stage-verification/`. ## Out of Scope -- The companion report-service story's routes and rule changes (Dependencies), and the dashboard app's and runner's use of them (RD-3, RD-4). +- REPORT-167's routes and rule changes (Dependencies), and the dashboard app's and runner's use of them (RD-3, RD-4). - MCP tools for the package commands; an agent can run the CLI. - Changing a package's visibility, `official`, `archived` or `current_version` after publish (report-server's `POST /api/v1/packages/:kind/:owner_id/:name/:state`), and listing the catalog. - Creating report runs or pulling data on a package's behalf, locally or on the VM. @@ -276,7 +282,15 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - A) cc-data keeps Go copies of the manifest rules, the archive limits and the matcher, guarded by fixtures. - B) cc-data asks report-server's validate and applies routes, keeping only what it needs to stage and start a package. -**Decision**: B (Doug, 2026-10-08, to remove DRY violations without multiplying stories). The copies already disagreed: report-server allows 28,800 seconds where the runner refuses over 7,200. The companion story makes report-server adopt the runner's stricter rules, so its answer is the runner's. +**Decision**: B (Doug, 2026-10-08, to remove DRY violations without multiplying stories). The copies already disagreed: report-server allows 28,800 seconds where the runner refuses over 7,200. REPORT-167 makes report-server adopt the runner's stricter rules, so its answer is the runner's. + +### RESOLVED: Keep `learner_endpoints`, and which route the `init` stub teaches +**Context**: REPORT-147 (`bca55aa`) counts learners through a Student ID Mapping run with `run_answers` and `student_id_mapping`, after measuring on production that whole-class Student Answers runs fail above three or four assignments and that a reused run's learner list goes stale. That left `learner_endpoints` with no consumer, and the stub teaching the failing route. +**Options considered**: +- A) Drop `learner_endpoints`, document the Student ID Mapping learner count, and switch the stub to Student ID Mapping. +- B) Keep the view for researchers who pull Student Answers runs, and keep the stub. + +**Decision**: A (Doug, 2026-10-08). Nothing uses the view, the route it rests on fails on real classes, and the existing `student_id_mapping` view already makes the count without double-counting. The stub is the template authors copy, so it teaches the route that works. ### RESOLVED: Judgment call: where a local scope comes from **Options considered**: @@ -306,7 +320,7 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - A) Mirror the runner (bare name), have the `init` stub read `RD_DATASET`, and record the discrepancy for RD-4 and the design to settle. - B) Write the full ref locally, and ask RD-4 to change `scopeFile` to match section 10. -**Decision**: A. This story's contract is "exactly as the runner", and B changes a contract RD-4 owns from a story that consumes it. If RD-4 changes `scopeFile`, R11's `dataset` follows it. Either way the `init` stub reads `RD_DATASET`, the one name that is a full ref on both sides. +**Decision**: A. This story's contract is "exactly as the runner", and B changes a contract RD-4 owns from a story that consumes it. RD-4 pass 2 has since settled it as the bare name, and `final-design.md` section 10 says so (stream channel #28). The `init` stub reads `RD_DATASET`, the one name that is a full ref on both sides. ## Self-Review @@ -318,7 +332,7 @@ Roles: senior engineer, security engineer, QA engineer, education researcher, pa R11 empties `dir/.cc-data-run/in` and `out` on every run. If `.cc-data-run` were a symbolic link to another directory, the emptying would delete under that target. Fixed in R11: every directory `package run` empties or creates must be a real directory, and a link is refused before anything is removed. #### RESOLVED: `publish` would upload a hand-made zip the runner then refuses -report-server checks entry paths but not entry types, and the runner refuses a symbolic link after unpacking (RD-4's `checkTree`). So a zip made outside `build` could publish and then fail on every VM. Resolved by the companion story: report-server refuses link entries at publish, so the one check covers every zip. +report-server checks entry paths but not entry types, and the runner refuses a symbolic link after unpacking (RD-4's `checkTree`). So a zip made outside `build` could publish and then fail on every VM. Resolved by REPORT-167: report-server refuses link entries at publish, so the one check covers every zip. ### Education Researcher @@ -334,7 +348,7 @@ The answers store stamps `_run_id` with the run that fetched each stored version With the output in the current directory and `build` run from inside the package, as authors do, R20 as written either refused its own default or zipped the last build into the next. Fixed in R20: the default is `dir/.cc-data-build/`, which R17 already excludes. #### RESOLVED: An `init` stub that only reads existing runs reports nothing on the VM -On the VM a package's dataset (`pkg-`) starts empty, and section 13 makes run creation package code. A stub that assumed a pulled run would pass locally and count zero on its first VM run. Fixed in R9: the stub reuses or creates its Student Answers run, as `_lib/ccdata.py` and RD-4's smoke package do. +On the VM a package's dataset (`pkg-`) starts empty, and section 13 makes run creation package code. A stub that assumed a pulled run would pass locally and count zero on its first VM run. Fixed in R9: the stub reuses or creates its own run, as `_lib/ccdata.py` and RD-4's smoke package do (a Student ID Mapping run since 2026-10-08). ### QA Engineer From c2aba060b749fd9bc249df88794110fa93952f21 Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 15:55:14 -0400 Subject: [PATCH 03/17] feat: add the run_answers view and document run-scoped counts [REPORT-146] run_answers is every answers row with the run_id of each run whose answers fetch holds it in membership. It is registered on every dataset, so a run with no answers counts zero rather than failing to bind as answers_ does, and it declares no files, so a materialized dataset reads it through the materialized answers and run_membership. The guidance and the researcher guide document it with the learner count through student_id_mapping, counted by user_id, and the per-run log freshness query. The tests run both counts on a dataset where each wrong way of counting gives a different number. --- README.md | 2 +- docs/researcher-guide.md | 1 + internal/duck/run_views_test.go | 67 +++++++++++++++++++++++++++++++++ internal/duck/views.go | 14 +++++++ internal/guidance/src/core.md | 18 +++++++++ 5 files changed, 101 insertions(+), 1 deletion(-) create mode 100644 internal/duck/run_views_test.go diff --git a/README.md b/README.md index beac3b8..8aac7ca 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ A command-line tool for [Concord Consortium](https://concord.org) researchers to - `get history `: the interactive state history series (full series by default), as JSONL. - `get attachments `: S3 file attachments referenced by answers/history (open-response audio, offloaded CODAP/SageModeler state). - **See what's in a dataset**: `cc-data dataset list` and `dataset show` render instant summaries from the manifest (downloads by type, answer/history/attachment counts, coverage, size, age). `--json` emits the same summary for Claude and scripts, and every `get` ends with a machine-readable result line so tooling can confirm what happened. -- **Query everything with SQL** (`cc-data query`, `cc-data repl`): an ephemeral in-memory DuckDB exposes unified `reports`, `answers`, and `history` views across everything in the dataset, plus `run_membership` (which runs' fetches covered which records), `attachment_files` (per-attachment metadata and local paths), `attachment_states` (queryable offloaded CODAP/SageModeler state), `student_id_mapping` and `student_metadata` (the Portal student reports as deduplicated join dimensions, one row per learner), and a `downloads` provenance dimension table. Audio attachment content itself isn't SQL-queryable; `attachment_files` gives you its local path instead. +- **Query everything with SQL** (`cc-data query`, `cc-data repl`): an ephemeral in-memory DuckDB exposes unified `reports`, `answers`, and `history` views across everything in the dataset, plus `run_membership` (which runs' fetches covered which records), `run_answers` (every answer with the `run_id` of each run whose answers fetch holds it), `attachment_files` (per-attachment metadata and local paths), `attachment_states` (queryable offloaded CODAP/SageModeler state), `student_id_mapping` and `student_metadata` (the Portal student reports as deduplicated join dimensions, one row per learner), and a `downloads` provenance dimension table. Audio attachment content itself isn't SQL-queryable; `attachment_files` gives you its local path instead. - **Teach Claude about itself**: on install, `cc-data` writes a Claude Code skill (and a one-line `~/.claude/CLAUDE.md` pointer) so Claude can download and query data for you, and ships `cc-data mcp`, an MCP stdio server for Claude Desktop using the same stored credential. ## Datasets diff --git a/docs/researcher-guide.md b/docs/researcher-guide.md index 3c0d112..77fafcb 100644 --- a/docs/researcher-guide.md +++ b/docs/researcher-guide.md @@ -346,6 +346,7 @@ When you (or Claude) query a dataset, the data is exposed as a set of SQL | `attachment_content` | The text/JSON content of every saved CODAP/SageModeler snapshot, queryable and diffable. | | `student_id_mapping` | One row per learner from Student ID Mapping runs, with the ids that join them to their answers and history. | | `student_metadata` | One row per learner from Student Metadata runs: name, username, class, school, teachers, permission forms. | +| `run_answers` | Every answer with the `run_id` of each run that holds it: `SELECT count(*) FROM run_answers WHERE run_id = 584` counts one run's answers. Join it to `student_id_mapping` on `remote_endpoint = run_remote_endpoint` to count learners by `user_id`. | | `run_membership`, `downloads` | Provenance: which run's fetch covered which records, and what each download was, including whether its run hid names. | ### Speeding up a large dataset diff --git a/internal/duck/run_views_test.go b/internal/duck/run_views_test.go new file mode 100644 index 0000000..86486d5 --- /dev/null +++ b/internal/duck/run_views_test.go @@ -0,0 +1,67 @@ +package duck + +import ( + "encoding/json" + "fmt" + "testing" + "time" + + "github.com/concord-consortium/cc-data-cli/internal/store" +) + +// The documented learner count, run-scoped on both sides, as the guidance gives it. +const learnersOfRun = "SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = %d AND m.run_id = %d" + +// Each wrong way to count gives a different number here: counting endpoints instead of users +// (user 1 answered in two assignments), joining the bare-endpoint learner (who would borrow +// another bare answer), counting an answer once instead of once per run (e3/q1 is in runs 700 +// and 800), and joining membership untyped (run 700's history holds e1a/q1 too). +func TestRunAnswersAndTheDocumentedLearnerCount(t *testing.T) { + d := newDS(t, "ds") + buildStore(t, d, 700, [][]byte{ + answerRec("s", "https://p/d/e1a", "q1", "a"), answerRec("s", "https://p/d/e1b", "q1", "b"), + answerRec("s", "https://p/d/e3", "q1", "c"), answerRec("s", "https://p/d/", "q1", "bare"), + }) + hist, _ := json.Marshal(map[string]any{"source_key": "s", "remote_endpoint": "https://p/d/e1a", "question_id": "q1", "history_id": "h1"}) + seg := store.OpenSegment(d.Dir, store.TypeHistory, 700) + if err := seg.AppendPage([][]byte{hist}, time.Unix(700, 0).UTC(), 700); err != nil { + t.Fatal(err) + } + seg.WriteCursor(&store.Cursor{Items: 1}) + if _, err := d.MergeCompact(store.TypeHistory, 700, seg); err != nil { + t.Fatal(err) + } + buildStore(t, d, 800, [][]byte{answerRec("s", "https://p/d/e3", "q1", "c"), answerRec("s", "https://p/d/e9", "q1", "z")}) + row := func(learner, user, class, offering int, endpoint string) string { + return fmt.Sprintf("%d,%d,%d,s%d,%d,%d,https://act/1,%s", learner, user, user, user, class, offering, endpoint) + } + addDimensionCSV(t, d, dimFixture{run: 700, slug: "student-id-mapping", fetchedAt: at(1), filter: `{}`, csv: mappingCSV( + row(101, 1, 1, 70, "https://p/d/e1a"), row(102, 1, 1, 71, "https://p/d/e1b"), + row(103, 3, 1, 70, "https://p/d/e3"), row(104, 4, 1, 70, "https://p/d/"))}) + addDimensionCSV(t, d, dimFixture{run: 800, slug: "student-id-mapping", fetchedAt: at(2), filter: `{}`, csv: mappingCSV( + row(109, 9, 2, 90, "https://p/d/e9"))}) + e := openEngine(t, []DatasetSpec{{DS: d}}, nil) + + if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 700"); n != 4 { + t.Errorf("run 700 answers = %d, want 4", n) + } + if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 800"); n != 2 { + t.Errorf("run 800 answers = %d, want 2", n) + } + if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 700, 700)); n != 2 { + t.Errorf("run 700 learners = %d, want 2 (users 1 and 3; user 4's endpoint is bare)", n) + } + if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 800, 800)); n != 1 { + t.Errorf("run 800 learners = %d, want 1", n) + } +} + +func TestRunAnswersIsEmptyNotMissingOnAFreshDataset(t *testing.T) { + e := openEngine(t, []DatasetSpec{{DS: newDS(t, "ds")}}, nil) + if n := queryInt(t, e, "SELECT count(*) FROM run_answers"); n != 0 { + t.Errorf("run_answers has %d rows", n) + } + if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 1, 1)); n != 0 { + t.Errorf("the learner count binds to %d on a fresh dataset, want 0", n) + } +} diff --git a/internal/duck/views.go b/internal/duck/views.go index 51db436..f4a5386 100644 --- a/internal/duck/views.go +++ b/internal/duck/views.go @@ -71,6 +71,7 @@ func (vs viewSet) statements() []viewStmt { stmts = append(stmts, vs.storeView(store.TypeAnswers)) stmts = append(stmts, vs.storeView(store.TypeHistory)) stmts = append(stmts, vs.runMembershipView()) + stmts = append(stmts, vs.runAnswersView()) stmts = append(stmts, vs.downloadsView()) stmts = append(stmts, vs.attachmentFilesView()) stmts = append(stmts, vs.attachmentStatesView()) @@ -360,6 +361,19 @@ func (vs viewSet) runMembershipView() viewStmt { return viewStmt{name: name, primary: primary, fallback: fallback, files: files} } +// runAnswersView is answers with the run_id of each answers membership. Always bound, unlike +// answers_; it must follow both views it reads and declares no files, so materialize reads +// it through theirs. +func (vs viewSet) runAnswersView() viewStmt { + name := vs.prefix + `"run_answers"` + answers := vs.prefix + sqlIdent(store.TypeAnswers) + membership := vs.prefix + `"run_membership"` + primary := fmt.Sprintf("CREATE VIEW %s AS SELECT m.run_id, s.* FROM %s s JOIN %s m USING (source_key, remote_endpoint, question_id) WHERE m.type = %s", + name, answers, membership, sqlStr(store.TypeAnswers)) + fallback := fmt.Sprintf("CREATE VIEW %s AS SELECT CAST(NULL AS BIGINT) AS run_id, s.* FROM %s s WHERE false", name, answers) + return viewStmt{name: name, primary: primary, fallback: fallback} +} + // downloadsView is a VALUES dimension table from the manifest. // // hide_names separates the two meanings of a name column: where a run hid names, student_name diff --git a/internal/guidance/src/core.md b/internal/guidance/src/core.md index a77a57c..987cce2 100644 --- a/internal/guidance/src/core.md +++ b/internal/guidance/src/core.md @@ -122,6 +122,24 @@ like `wildfire_2026.answers`): **type-qualified**: `answers a JOIN run_membership m USING (source_key, remote_endpoint, question_id) WHERE m.run_id = 584 AND m.type = 'answers'`. History joins add `history_id` to the USING list. +- `run_answers` — every `answers` row with `run_id`, once for each run whose + answers fetch holds it (a Student Answers or a Student ID Mapping run), so one + run's answers are `SELECT count(*) FROM + run_answers WHERE run_id = 584`. It is the type-qualified membership join above, + always present, so a run with no answers counts zero rather than failing to bind + as `answers_` does. Its `run_id` is membership; the store's own `_run_id` is + only the run that last fetched the record, so filtering `answers` by `_run_id` + undercounts a run whose answers a later run re-fetched. +- A run's learners with at least one answer come from its Student ID Mapping + report: `SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN + run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = 584 + AND m.run_id = 584`, where 584 is a Student ID Mapping run whose answers were + fetched. Count by `user_id`, never by endpoint, since a student has one endpoint + per assignment. A Student ID Mapping run is computed live, so re-reading it with + `--refresh` picks up learners who joined since; a Student Answers run is fixed + when its query ran, and a whole-class one fails above three or four assignments. +- A run's log freshness is `SELECT count(*) AS logs, max(event_time) AS + log_freshness_at FROM logs WHERE run_id = `. - Reports-to-stores join: `reports.res__remote_endpoint = answers.remote_endpoint`, with `res___*` pairing to `answers.question_id`. From 38620594d18a0c295caa716068d68cb880f700a2 Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 15:56:52 -0400 Subject: [PATCH 04/17] feat: read, collect, zip and read back a package [REPORT-146] internal/packages gains the local side of a Researcher Dashboard package, with no callers yet: - ReadManifest reads only the fields a run acts on: the entrypoint, which must be a collected file, and a positive integer duration. Every other rule stays report-server's. - Collect gathers what build ships and run stages, leaving out dot paths, __pycache__, *.pyc and local-data, refusing links and other non-regular files, and recording each file's ship mode (0755 when any execute bit is set, else 0644). - Zip writes a reproducible archive, and CheckEntrypointMode refuses a non-.py entrypoint that would not be executable after unzip. - ReadResult reads display.md, summary.txt and counts.json as the runner does, following no link and refusing a display.md over the 65,536-byte cap. The spec's manifest check drops its raw-bytes duration test, which json.Unmarshal already makes redundant. --- internal/packages/archive.go | 145 +++++++++++ internal/packages/files_test.go | 226 ++++++++++++++++++ internal/packages/manifest.go | 77 ++++++ internal/packages/result.go | 113 +++++++++ .../implementation.md | 47 ++-- 5 files changed, 579 insertions(+), 29 deletions(-) create mode 100644 internal/packages/archive.go create mode 100644 internal/packages/files_test.go create mode 100644 internal/packages/manifest.go create mode 100644 internal/packages/result.go diff --git a/internal/packages/archive.go b/internal/packages/archive.go new file mode 100644 index 0000000..08a2619 --- /dev/null +++ b/internal/packages/archive.go @@ -0,0 +1,145 @@ +package packages + +import ( + "archive/zip" + "bytes" + "crypto/sha256" + "encoding/hex" + "fmt" + "io" + "io/fs" + "os" + "path" + "path/filepath" + "runtime" + "sort" + "strings" + "time" +) + +// buildTime is every entry's modification time, so the same tree always zips to the same +// bytes and the same checksum. +var buildTime = time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC) + +// Files is a package's contents: slash-separated paths relative to its directory, sorted, +// each with the mode it ships with. +type Files struct { + Dir string + Paths []string + Modes map[string]os.FileMode + Bytes int64 +} + +// shipMode is the one mode a file keeps through build and unzip: 0755 when any execute bit is +// set, else 0644. unzip restores it on the VM, where a non-.py entrypoint is executed directly. +func shipMode(m fs.FileMode) os.FileMode { + if m.Perm()&0o111 != 0 { + return 0o755 + } + return 0o644 +} + +// Set is the paths as a lookup, the shape ReadManifest checks the entrypoint against. +func (f Files) Set() map[string]bool { + set := make(map[string]bool, len(f.Paths)) + for _, p := range f.Paths { + set[p] = true + } + return set +} + +// excluded is what a build leaves out: any dot path (.git, .cc-data-run, .cc-data-build), +// __pycache__, *.pyc and local-data. +func excluded(rel string, isDir bool) bool { + base := path.Base(rel) + if strings.HasPrefix(base, ".") || base == "__pycache__" || (isDir && base == "local-data") { + return true + } + return !isDir && strings.HasSuffix(base, ".pyc") +} + +// Collect walks dir for the files a package is made of. A link or other non-regular file is +// refused rather than skipped, since the runner refuses an archive holding one. +func Collect(dir string) (Files, error) { + files := Files{Dir: dir, Modes: map[string]os.FileMode{}} + err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if p == dir { + return nil + } + rel, err := filepath.Rel(dir, p) + if err != nil { + return err + } + rel = filepath.ToSlash(rel) + if excluded(rel, d.IsDir()) { + if d.IsDir() { + return filepath.SkipDir + } + return nil + } + switch { + case d.Type()&fs.ModeSymlink != 0: + return fmt.Errorf("%s is a symbolic link; a package may hold only regular files", rel) + case d.IsDir(): + return nil + case !d.Type().IsRegular(): + return fmt.Errorf("%s is not a regular file", rel) + } + info, err := d.Info() + if err != nil { + return err + } + files.Paths = append(files.Paths, rel) + files.Modes[rel] = shipMode(info.Mode()) + files.Bytes += info.Size() + return nil + }) + sort.Strings(files.Paths) + return files, err +} + +// Zip writes a reproducible archive: sorted entries, Deflate, a fixed time and each file's ship +// mode. The size limits are report-server's. +func Zip(files Files) ([]byte, error) { + var buf bytes.Buffer + w := zip.NewWriter(&buf) + for _, rel := range files.Paths { + h := &zip.FileHeader{Name: rel, Method: zip.Deflate, Modified: buildTime} + h.SetMode(files.Modes[rel]) + dst, err := w.CreateHeader(h) + if err != nil { + return nil, err + } + src, err := os.Open(filepath.Join(files.Dir, filepath.FromSlash(rel))) + if err != nil { + return nil, err + } + _, err = io.Copy(dst, src) + src.Close() + if err != nil { + return nil, err + } + } + if err := w.Close(); err != nil { + return nil, err + } + return buf.Bytes(), nil +} + +// CheckEntrypointMode refuses a non-.py entrypoint that would not be executable once unzipped, +// since the runner executes it directly. Windows records no execute bit to check. +func (f Files) CheckEntrypointMode(entrypoint string) error { + if runtime.GOOS == "windows" || strings.HasSuffix(entrypoint, ".py") || f.Modes[entrypoint]&0o111 != 0 { + return nil + } + return fmt.Errorf("the entrypoint %s is not executable; the runner executes a non-.py entrypoint directly (chmod +x it)", entrypoint) +} + +// Checksum is the catalog's spelling of an archive's digest. +func Checksum(b []byte) string { + sum := sha256.Sum256(b) + return "sha256:" + hex.EncodeToString(sum[:]) +} diff --git a/internal/packages/files_test.go b/internal/packages/files_test.go new file mode 100644 index 0000000..c7cb4bb --- /dev/null +++ b/internal/packages/files_test.go @@ -0,0 +1,226 @@ +package packages + +import ( + "archive/zip" + "bytes" + "errors" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + "time" +) + +const goodManifest = `{ + "name": "wildfire-responses", + "title": "Wildfire responses", + "version": "0.1.0", + "urls": {"all": [], "any": [], "none": []}, + "clue_prepull": false, + "entrypoint": "run.py", + "expected_duration_seconds": 300 +}` + +// writeTree writes each path's content under dir, creating parents; a path ending in "/" is an +// empty directory. +func writeTree(t *testing.T, dir string, files map[string]string) { + t.Helper() + for rel, content := range files { + p := filepath.Join(dir, filepath.FromSlash(rel)) + if strings.HasSuffix(rel, "/") { + if err := os.MkdirAll(p, 0o755); err != nil { + t.Fatal(err) + } + continue + } + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } +} + +func skipOnWindows(t *testing.T, why string) { + t.Helper() + if runtime.GOOS == "windows" { + t.Skip(why) + } +} + +func TestReadManifestNeedsWhatTheRunActsOn(t *testing.T) { + files := map[string]bool{"run.py": true} + if _, err := ReadManifest([]byte(goodManifest), files); err != nil { + t.Fatalf("good manifest refused: %v", err) + } + if _, err := ReadManifest([]byte(goodManifest), map[string]bool{"other.py": true}); err == nil { + t.Error("a missing entrypoint was accepted") + } + for _, duration := range []string{`"300"`, `1.5`, `0`} { + m := strings.Replace(goodManifest, `"expected_duration_seconds": 300`, `"expected_duration_seconds": `+duration, 1) + if _, err := ReadManifest([]byte(m), files); err == nil { + t.Errorf("duration %s was accepted", duration) + } + } +} + +func TestCollectLeavesOutWhatBuildMustNotShip(t *testing.T) { + dir := t.TempDir() + writeTree(t, dir, map[string]string{ + "manifest.json": goodManifest, + "run.py": "print(1)", + "lib/q.sql": "SELECT 1", + "lib/y.pyc": "x", + ".git/HEAD": "ref", + ".cc-data-run/out/display.md": "x", + ".cc-data-build/p-0.1.0.zip": "x", + "__pycache__/cache.txt": "x", + "local-data/answers.csv": "x", + ".DS_Store": "x", + }) + files, err := Collect(dir) + if err != nil { + t.Fatal(err) + } + if got := strings.Join(files.Paths, ","); got != "lib/q.sql,manifest.json,run.py" { + t.Errorf("collected %s", got) + } +} + +func TestCollectRefusesALink(t *testing.T) { + skipOnWindows(t, "links need privileges on Windows") + dir := t.TempDir() + writeTree(t, dir, map[string]string{"manifest.json": goodManifest, "run.py": "print(1)"}) + if err := os.Symlink(filepath.Join(dir, "run.py"), filepath.Join(dir, "alias.py")); err != nil { + t.Fatal(err) + } + if _, err := Collect(dir); err == nil || !strings.Contains(err.Error(), "symbolic link") { + t.Errorf("err = %v, want a symbolic-link refusal", err) + } +} + +func TestZipIsReproducible(t *testing.T) { + dir := t.TempDir() + writeTree(t, dir, map[string]string{"manifest.json": goodManifest, "run.py": "print(1)"}) + build := func() string { + files, err := Collect(dir) + if err != nil { + t.Fatal(err) + } + b, err := Zip(files) + if err != nil { + t.Fatal(err) + } + return Checksum(b) + } + first := build() + later := time.Now().Add(time.Hour) + if err := os.Chtimes(filepath.Join(dir, "run.py"), later, later); err != nil { + t.Fatal(err) + } + if second := build(); second != first { + t.Errorf("checksum changed with an mtime: %s then %s", first, second) + } +} + +func TestModesSurviveTheZip(t *testing.T) { + skipOnWindows(t, "Windows records no execute bit") + dir := t.TempDir() + writeTree(t, dir, map[string]string{"manifest.json": goodManifest, "run.py": "print(1)", "run.sh": "#!/bin/sh\n"}) + if err := os.Chmod(filepath.Join(dir, "run.py"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Chmod(filepath.Join(dir, "manifest.json"), 0o600); err != nil { + t.Fatal(err) + } + files, err := Collect(dir) + if err != nil { + t.Fatal(err) + } + b, err := Zip(files) + if err != nil { + t.Fatal(err) + } + r, err := zip.NewReader(bytes.NewReader(b), int64(len(b))) + if err != nil { + t.Fatal(err) + } + modes := map[string]os.FileMode{} + for _, f := range r.File { + modes[f.Name] = f.Mode().Perm() + } + if modes["run.py"] != 0o755 || modes["manifest.json"] != 0o644 { + t.Errorf("zipped modes = %v, want run.py 0755 and manifest.json 0644", modes) + } + if err := files.CheckEntrypointMode("run.sh"); err == nil { + t.Error("a 0644 run.sh entrypoint passed") + } + if err := os.Chmod(filepath.Join(dir, "run.py"), 0o644); err != nil { + t.Fatal(err) + } + files, err = Collect(dir) + if err != nil { + t.Fatal(err) + } + if err := files.CheckEntrypointMode("run.py"); err != nil { + t.Errorf("a 0644 run.py was refused: %v", err) + } +} + +func TestDisplayCapEdges(t *testing.T) { + if DisplayLimitBytes != 65536 { + t.Fatalf("DisplayLimitBytes = %d, want 65536", DisplayLimitBytes) + } + out := t.TempDir() + write := func(n int) { + if err := os.WriteFile(filepath.Join(out, "display.md"), bytes.Repeat([]byte("x"), n), 0o644); err != nil { + t.Fatal(err) + } + } + write(DisplayLimitBytes) + if res, err := ReadResult(out); err != nil || res.DisplayBytes != DisplayLimitBytes { + t.Errorf("at the cap: %+v, %v", res, err) + } + write(DisplayLimitBytes + 1) + var refused *OutputRefused + if _, err := ReadResult(out); !errors.As(err, &refused) { + t.Errorf("one byte over the cap: err = %v, want OutputRefused", err) + } +} + +func TestResultReadsOnlyTheRunnersKeys(t *testing.T) { + out := t.TempDir() + writeTree(t, out, map[string]string{ + "display.md": "# Class\n", + "summary.txt": " 3 answers from 2 learners \r\nsecond line\n", + "counts.json": `{"answers": 3, "logs": 7, "log_freshness_at": "2026-10-08T00:00:00Z", "learners": 2}`, + }) + res, err := ReadResult(out) + if err != nil { + t.Fatal(err) + } + if res.Summary == nil || *res.Summary != "3 answers from 2 learners" { + t.Errorf("summary = %v", res.Summary) + } + if len(res.Counts) != 3 || res.Counts["answers"] != float64(3) || res.Counts["learners"] != nil { + t.Errorf("counts = %v", res.Counts) + } +} + +func TestResultRefusesALinkedDisplay(t *testing.T) { + skipOnWindows(t, "links need privileges on Windows") + out := t.TempDir() + target := filepath.Join(t.TempDir(), "elsewhere.md") + if err := os.WriteFile(target, []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.Symlink(target, filepath.Join(out, "display.md")); err != nil { + t.Fatal(err) + } + var refused *OutputRefused + if _, err := ReadResult(out); !errors.As(err, &refused) { + t.Errorf("err = %v, want OutputRefused", err) + } +} diff --git a/internal/packages/manifest.go b/internal/packages/manifest.go new file mode 100644 index 0000000..6309f2d --- /dev/null +++ b/internal/packages/manifest.go @@ -0,0 +1,77 @@ +// Package packages is the local side of a Researcher Dashboard package: reading the manifest +// fields the runner acts on, collecting and zipping the package, staging and running it the +// way the runner does, and reading the result files. The package rules themselves, and +// applicability, are report-server's, which cc-data asks rather than copies. +package packages + +import ( + "encoding/json" + "fmt" + "regexp" + "strings" + "unicode" +) + +var nameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,62}$`) + +// ValidName reports whether name fits the catalog's name grammar, which init needs before +// report-server has seen anything. +func ValidName(name string) bool { return nameRe.MatchString(name) } + +// URLs is a manifest's applicability patterns, passed to report-server as written. +type URLs struct { + All []string `json:"all"` + Any []string `json:"any"` + None []string `json:"none"` +} + +// DeclaresPatterns reports whether any sense holds a pattern; a package that declares none +// applies to every scope. +func (u URLs) DeclaresPatterns() bool { + return len(u.All)+len(u.Any)+len(u.None) > 0 +} + +// Manifest is the part of manifest.json the runner acts on. Everything else, and every limit, +// is report-server's to check (cmd's validate call). +type Manifest struct { + Name string `json:"name"` + Title string `json:"title"` + Version string `json:"version"` + Description string `json:"description,omitempty"` + URLs URLs `json:"urls"` + CluePrepull bool `json:"clue_prepull"` + Entrypoint string `json:"entrypoint"` + ExpectedDurationSeconds int `json:"expected_duration_seconds"` +} + +// ReadManifest reads the fields a run acts on, refusing an entrypoint outside files and a +// duration that is not a positive integer; every other rule is report-server's. +func ReadManifest(data []byte, files map[string]bool) (Manifest, error) { + var m Manifest + if err := json.Unmarshal(data, &m); err != nil { + return m, fmt.Errorf("manifest.json: %v", err) + } + if !files[m.Entrypoint] || unsafePath(m.Entrypoint) { + return m, fmt.Errorf("manifest.json: entrypoint %q is not a file in the package", m.Entrypoint) + } + if m.ExpectedDurationSeconds < 1 { + return m, fmt.Errorf("manifest.json: expected_duration_seconds must be a positive integer") + } + return m, nil +} + +// unsafePath is report-server's Archive.unsafe_path?: absolute, a drive letter, or a ".." segment. +func unsafePath(name string) bool { + if strings.HasPrefix(name, "/") || strings.HasPrefix(name, `\`) { + return true + } + if len(name) >= 2 && name[1] == ':' && unicode.IsLetter(rune(name[0])) { + return true + } + for _, seg := range strings.FieldsFunc(name, func(r rune) bool { return r == '/' || r == '\\' }) { + if seg == ".." { + return true + } + } + return false +} diff --git a/internal/packages/result.go b/internal/packages/result.go new file mode 100644 index 0000000..fd24440 --- /dev/null +++ b/internal/packages/result.go @@ -0,0 +1,113 @@ +package packages + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "io/fs" + "os" + "path/filepath" + "strings" +) + +// DisplayLimitBytes is the runner's cap on display.md in UTF-8 bytes, decided only in +// final-design.md section 10. +const DisplayLimitBytes = 64 * 1024 + +// countKeys are the counts.json keys the runner reads; anything else a package writes there +// is ignored, as it is on the VM. +var countKeys = []string{"answers", "logs", "log_freshness_at"} + +// Result is what the runner keeps of a run: the display, the summary line and the counts. +type Result struct { + DisplayPath string `json:"display"` + DisplayBytes int64 `json:"display_bytes"` + Summary *string `json:"summary"` + Counts map[string]any `json:"counts"` +} + +// OutputRefused is a result the runner would refuse: no display.md, one that is a link or +// not a file, or one over the cap. +type OutputRefused struct{ Reason string } + +func (e *OutputRefused) Error() string { return e.Reason } + +// ReadResult reads the output directory as the runner does, following no link: the +// directory must be itself, and each file must be a regular file. +func ReadResult(outDir string) (Result, error) { + var res Result + if err := RealDir(outDir); err != nil { + return res, &OutputRefused{Reason: "the package replaced its output directory with a link"} + } + _, size, err := readOwnFile(outDir, "display.md", DisplayLimitBytes) + switch { + case errors.Is(err, fs.ErrNotExist): + return res, &OutputRefused{Reason: "the package wrote no display.md"} + case err != nil: + return res, err + case size > DisplayLimitBytes: + return res, &OutputRefused{Reason: fmt.Sprintf("the package's display.md is %d bytes, over the %d-byte limit", size, DisplayLimitBytes)} + } + res.DisplayPath = filepath.Join(outDir, "display.md") + res.DisplayBytes = size + if text, _, err := readOwnFile(outDir, "summary.txt", DisplayLimitBytes); err == nil { + line := strings.TrimSpace(strings.SplitN(strings.ReplaceAll(text, "\r\n", "\n"), "\n", 2)[0]) + if line != "" { + res.Summary = &line + } + } else if !errors.Is(err, fs.ErrNotExist) { + return res, err + } + res.Counts = map[string]any{} + if text, _, err := readOwnFile(outDir, "counts.json", DisplayLimitBytes); err == nil { + var parsed map[string]any + if json.Unmarshal([]byte(text), &parsed) == nil { + for _, k := range countKeys { + if v, ok := parsed[k]; ok { + res.Counts[k] = v + } + } + } + } + return res, nil +} + +// readOwnFile returns up to limit+1 bytes of a regular, unlinked file and its full size. The +// Lstat-then-SameFile pair stands in for O_NOFOLLOW, which Windows lacks. +func readOwnFile(dir, name string, limit int64) (string, int64, error) { + p := filepath.Join(dir, name) + before, err := os.Lstat(p) + if err != nil { + return "", 0, err + } + if before.Mode()&fs.ModeSymlink != 0 { + return "", 0, &OutputRefused{Reason: fmt.Sprintf("the package's %s is a link, not a file", name)} + } + if !before.Mode().IsRegular() { + return "", 0, &OutputRefused{Reason: fmt.Sprintf("the package's %s is not a file", name)} + } + f, err := os.Open(p) + if err != nil { + return "", 0, err + } + defer f.Close() + after, err := f.Stat() + if err != nil || !os.SameFile(before, after) { + return "", 0, &OutputRefused{Reason: fmt.Sprintf("the package's %s changed while it was read", name)} + } + b, err := io.ReadAll(io.LimitReader(f, limit+1)) + return string(b), after.Size(), err +} + +// RealDir reports an error unless p is a directory reached without following a link. +func RealDir(p string) error { + info, err := os.Lstat(p) + if err != nil { + return err + } + if info.Mode()&fs.ModeSymlink != 0 || !info.IsDir() { + return fmt.Errorf("%s is a link or not a directory", p) + } + return nil +} diff --git a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md index a99fa5c..61f5cc0 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md +++ b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md @@ -203,7 +203,6 @@ Links are checked with `Lstat` then `os.SameFile` rather than `O_NOFOLLOW`, whic package packages import ( - "bytes" "encoding/json" "fmt" "regexp" @@ -243,28 +242,24 @@ type Manifest struct { ExpectedDurationSeconds int `json:"expected_duration_seconds"` } -// ReadManifest reads the fields package run needs to stage and start a package: the -// entrypoint, which must be one of files, the duration that sets the time bound, the patterns -// and clue_prepull. A wrongly typed field is refused here because the run cannot proceed -// without it; the full rules are report-server's. +// ReadManifest reads the fields a run acts on, refusing an entrypoint outside files and a +// duration that is not a positive integer; every other rule is report-server's. func ReadManifest(data []byte, files map[string]bool) (Manifest, error) { var m Manifest if err := json.Unmarshal(data, &m); err != nil { return m, fmt.Errorf("manifest.json: %v", err) } - var raw map[string]json.RawMessage - json.Unmarshal(data, &raw) - if !files[m.Entrypoint] || UnsafePath(m.Entrypoint) { + if !files[m.Entrypoint] || unsafePath(m.Entrypoint) { return m, fmt.Errorf("manifest.json: entrypoint %q is not a file in the package", m.Entrypoint) } - if v := raw["expected_duration_seconds"]; len(v) == 0 || v[0] == '"' || bytes.ContainsAny(v, ".eE") || m.ExpectedDurationSeconds < 1 { + if m.ExpectedDurationSeconds < 1 { return m, fmt.Errorf("manifest.json: expected_duration_seconds must be a positive integer") } return m, nil } -// UnsafePath is report-server's Archive.unsafe_path?: absolute, a drive letter, or a ".." segment. -func UnsafePath(name string) bool { +// unsafePath is report-server's Archive.unsafe_path?: absolute, a drive letter, or a ".." segment. +func unsafePath(name string) bool { if strings.HasPrefix(name, "/") || strings.HasPrefix(name, `\`) { return true } @@ -330,9 +325,8 @@ func (f Files) Set() map[string]bool { return set } -// excluded is what a build leaves behind: anything under a segment starting with ".", -// which covers this tool's own .cc-data-run and .cc-data-build as well as .git, and the -// three things cc-data-studies ignores. +// excluded is what a build leaves out: any dot path (.git, .cc-data-run, .cc-data-build), +// __pycache__, *.pyc and local-data. func excluded(rel string, isDir bool) bool { base := path.Base(rel) if strings.HasPrefix(base, ".") || base == "__pycache__" || (isDir && base == "local-data") { @@ -341,9 +335,8 @@ func excluded(rel string, isDir bool) bool { return !isDir && strings.HasSuffix(base, ".pyc") } -// Collect walks dir for the files a package is made of. A symbolic link or any other -// non-regular file is refused rather than skipped, since the runner refuses an archive -// holding one and a link silently dropped would be a package that differs from its tree. +// Collect walks dir for the files a package is made of. A link or other non-regular file is +// refused rather than skipped, since the runner refuses an archive holding one. func Collect(dir string) (Files, error) { files := Files{Dir: dir, Modes: map[string]os.FileMode{}} err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error { @@ -385,9 +378,8 @@ func Collect(dir string) (Files, error) { return files, err } -// Zip writes the files as a reproducible archive: sorted entries, Deflate, a fixed time, and -// each file's ship mode. archive/zip also writes that time as an extended-timestamp extra -// field, which is as fixed as the time itself. The size limits are report-server's to apply. +// Zip writes a reproducible archive: sorted entries, Deflate, a fixed time and each file's ship +// mode. The size limits are report-server's. func Zip(files Files) ([]byte, error) { var buf bytes.Buffer w := zip.NewWriter(&buf) @@ -415,8 +407,7 @@ func Zip(files Files) ([]byte, error) { } // CheckEntrypointMode refuses a non-.py entrypoint that would not be executable once unzipped, -// since the runner executes such an entrypoint directly. Windows records no execute bit, so -// there the check cannot be made. +// since the runner executes it directly. Windows records no execute bit to check. func (f Files) CheckEntrypointMode(entrypoint string) error { if runtime.GOOS == "windows" || strings.HasSuffix(entrypoint, ".py") || f.Modes[entrypoint]&0o111 != 0 { return nil @@ -444,10 +435,8 @@ import ( "strings" ) -// DisplayLimitBytes is the cap on display.md, counted in UTF-8 bytes. final-design.md -// section 10 is the only place this number is decided; the runner and this constant both -// cite it, because a local cap looser than the runner's is a check that passes here and a -// VM that then refuses. +// DisplayLimitBytes is the runner's cap on display.md in UTF-8 bytes, decided only in +// final-design.md section 10. const DisplayLimitBytes = 64 * 1024 // countKeys are the counts.json keys the runner reads; anything else a package writes there @@ -553,7 +542,7 @@ Tests (the link and mode tests skip on Windows, where links need privileges and - **`TestCollectLeavesOutWhatBuildMustNotShip`**: the tree holds `.git/`, `.cc-data-run/`, `.cc-data-build/`, `__pycache__/cache.txt`, `lib/y.pyc`, `local-data/` and `.DS_Store`. Exactly `lib/q.sql,manifest.json,run.py` is collected. `__pycache__` holds a non-`.pyc` file so the directory rule is tested on its own. - **`TestCollectRefusesALink`**. - **`TestZipIsReproducible`**: a touched mtime leaves the checksum unchanged. -- **`TestModesSurviveBuildAndStaging`**: `run.py` (0755 on disk) zips as 0755 and a 0600 `manifest.json` as 0644. A 0644 `run.sh` entrypoint fails `CheckEntrypointMode`, and a 0644 `run.py` passes it. +- **`TestModesSurviveTheZip`**: `run.py` (0755 on disk) zips as 0755 and a 0600 `manifest.json` as 0644. A 0644 `run.sh` entrypoint fails `CheckEntrypointMode`, and a 0644 `run.py` passes it. - **`TestDisplayCapEdges`**: 65,536 bytes pass and 65,537 are refused, and the constant equals 65,536. - **`TestResultReadsOnlyTheRunnersKeys`**: the summary is the first line, trimmed, and `learners` in `counts.json` is dropped. - **`TestResultRefusesALinkedDisplay`**. @@ -562,8 +551,8 @@ Tests (the link and mode tests skip on Windows, where links need privileges and |---|---| | `size >= DisplayLimitBytes` | `TestDisplayCapEdges` | | `__pycache__` not excluded | `TestCollectLeavesOutWhatBuildMustNotShip` | -| every entry zipped 0644 | `TestModesSurviveBuildAndStaging` | -| a quoted duration accepted | `TestReadManifestNeedsWhatTheRunActsOn` | +| every entry zipped 0644 | `TestModesSurviveTheZip` | +| a duration of 0 accepted | `TestReadManifestNeedsWhatTheRunActsOn` | --- From eb6db3091299545df56494d89c9740683d763bdd Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 15:59:36 -0400 Subject: [PATCH 05/17] feat: run a package under the runner's layout [REPORT-146] packages.Run is package run's engine: - It refuses a clue_prepull package, and asks an injected Applies for a package that declares patterns: a "does not apply" verdict is refused with the runner's prefix, and an unconfirmed one warns and runs. - It rebuilds .cc-data-run (a staged copy of what build ships, in/, out/, a persistent data/ and a .gitignore of *), refusing any linked directory before removing anything, and writes the runner's scope.json from the author's four-key scope file. - It runs the entrypoint from the staged copy with the runner's variables plus a short passthrough list (credential lookup, locale, proxies), with python3.11 or python3 for a .py entrypoint. - The entrypoint runs in its own process group, which is killed at the time bound, on an interrupt, and once the entrypoint exits. --- internal/packages/proc_unix.go | 23 ++ internal/packages/proc_windows.go | 10 + internal/packages/run.go | 277 ++++++++++++++++++ internal/packages/run_test.go | 265 +++++++++++++++++ internal/packages/scope.go | 99 +++++++ .../implementation.md | 9 +- 6 files changed, 678 insertions(+), 5 deletions(-) create mode 100644 internal/packages/proc_unix.go create mode 100644 internal/packages/proc_windows.go create mode 100644 internal/packages/run.go create mode 100644 internal/packages/run_test.go create mode 100644 internal/packages/scope.go diff --git a/internal/packages/proc_unix.go b/internal/packages/proc_unix.go new file mode 100644 index 0000000..0d9442b --- /dev/null +++ b/internal/packages/proc_unix.go @@ -0,0 +1,23 @@ +//go:build !windows + +package packages + +import ( + "os/exec" + "syscall" +) + +// killGroup runs the package in its own process group and kills the whole group on timeout, +// so a child the entrypoint started (cc-data, a shell) does not outlive the bound. +func killGroup(cmd *exec.Cmd) { + cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) } +} + +// reapGroup kills whatever is left of the package's process group once the entrypoint has +// exited. The group outlives its leader while any member does, so the kill still finds it. +func reapGroup(cmd *exec.Cmd) { + if cmd.Process != nil { + syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) + } +} diff --git a/internal/packages/proc_windows.go b/internal/packages/proc_windows.go new file mode 100644 index 0000000..3b2ad4c --- /dev/null +++ b/internal/packages/proc_windows.go @@ -0,0 +1,10 @@ +//go:build windows + +package packages + +import "os/exec" + +// killGroup and reapGroup kill only the entrypoint on Windows, which has no release. +func killGroup(cmd *exec.Cmd) {} + +func reapGroup(cmd *exec.Cmd) {} diff --git a/internal/packages/run.go b/internal/packages/run.go new file mode 100644 index 0000000..424251e --- /dev/null +++ b/internal/packages/run.go @@ -0,0 +1,277 @@ +package packages + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "sort" + "strings" + "time" +) + +// TimeoutMargin is what the runner adds to expected_duration_seconds before it kills a run. +const TimeoutMargin = 10 * time.Minute + +// RunDirName is the working tree package run keeps inside the package directory. It starts +// with "." so Collect, and so build, never ships it. +const RunDirName = ".cc-data-run" + +// passthrough is what a laptop adds to the runner's environment so the package's own cc-data +// calls find the researcher's stored credential (keychain or credentials file), reach the +// server, and run at all. The runner sets HTTPS_PROXY and HTTP_PROXY when it has a proxy, and +// NO_PROXY keeps a laptop's exemptions with them. Matching is case-insensitive, so the +// lower-case spellings pass too. +var passthrough = []string{ + "HOME", "PATH", "USER", "LOGNAME", "LANG", "TZ", "TMPDIR", + "DBUS_SESSION_BUS_ADDRESS", "XDG_RUNTIME_DIR", + "HTTPS_PROXY", "HTTP_PROXY", "NO_PROXY", + // Windows has no release, but CI builds and tests there. + "SYSTEMROOT", "USERPROFILE", "APPDATA", "LOCALAPPDATA", "PATHEXT", "COMSPEC", +} + +// Verdict is report-server's answer to whether a package's patterns match a scope. Unconfirmed +// means the report-server answering has no applicability route yet, so nothing was checked. +type Verdict struct { + Applies bool + Reason string + Unconfirmed bool +} + +// Refused is a package the runner would not start: it does not apply, or it needs the CLUE +// pre-pull, which cc-data cannot do. +type Refused struct{ Reason string } + +func (e *Refused) Error() string { return e.Reason } + +// Failed is a package that started and did not finish cleanly. +type Failed struct{ Reason string } + +func (e *Failed) Error() string { return e.Reason } + +// RunOptions is everything package run decides before the entrypoint starts. +type RunOptions struct { + Dir string // the package directory + Manifest Manifest // validated against Files + Files Files // what build would ship + Scope LocalScope // the author's scope file + Applies func(context.Context, URLs) (Verdict, error) // report-server's applicability answer + Dataset string // the full ref, / + Name string // the ref's bare name + Portal string // the ref's portal host + DataRoot string // CC_DATA_ROOT as package run resolved it + ReportSrv string // the stored credential's server, or "" + Stderr io.Writer // where the package's own output goes + Environ func() []string + LookPath func(string) (string, error) + Timeout time.Duration // zero means expected_duration_seconds + TimeoutMargin +} + +// Run prepares the runner's layout, runs the entrypoint and reads its result. +func Run(ctx context.Context, o RunOptions) (Result, error) { + if o.Manifest.CluePrepull { + return Result{}, &Refused{Reason: "the package sets clue_prepull, and cc-data cannot fetch CLUE data yet, so it runs only on the VM"} + } + if o.Manifest.URLs.DeclaresPatterns() { + v, err := o.Applies(ctx, o.Manifest.URLs) + switch { + case err != nil: + return Result{}, err + case v.Unconfirmed: + fmt.Fprintln(o.Stderr, "warning: report-server cannot check applicability yet; the package runs, and the VM's runner is the check") + case !v.Applies: + return Result{}, &Refused{Reason: "the package does not apply to this class: " + v.Reason} + } + } + run := filepath.Join(o.Dir, RunDirName) + paths, err := prepare(run, o.Files) + if err != nil { + return Result{}, err + } + scope := ScopeFile{LocalScope: o.Scope, ClueSource: "firebase", Dataset: o.Name, OutputDir: paths.out} + b, err := json.MarshalIndent(scope, "", " ") + if err != nil { + return Result{}, err + } + scopePath := filepath.Join(paths.in, "scope.json") + if err := os.WriteFile(scopePath, b, 0o644); err != nil { + return Result{}, err + } + + env := map[string]string{ + "CC_DATA_ROOT": o.DataRoot, + "CC_DATA_PORTAL": o.Portal, + "CC_DATA_LOCAL": paths.data, + "RD_DATASET": o.Dataset, + "RD_SCOPE_FILE": scopePath, + "RD_DATA_DIR": paths.data, + "RD_OUTPUT_DIR": paths.out, + } + if o.ReportSrv != "" { + env["RD_REPORT_SERVER_URL"] = o.ReportSrv + } + command, args, note, err := entrypointCommand(paths.pkg, o.Manifest.Entrypoint, o.LookPath) + if err != nil { + return Result{}, err + } + if note != "" { + fmt.Fprintln(o.Stderr, note) + } + + timeout := o.Timeout + if timeout == 0 { + timeout = time.Duration(o.Manifest.ExpectedDurationSeconds)*time.Second + TimeoutMargin + } + runCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + cmd := exec.CommandContext(runCtx, command, args...) + cmd.Dir = paths.pkg + cmd.Env = environment(o.Environ(), env) + cmd.Stdout = o.Stderr + cmd.Stderr = o.Stderr + killGroup(cmd) + err = cmd.Run() + // As on the VM, nothing the package started outlives it or writes after it. + reapGroup(cmd) + if err != nil { + switch { + case errors.Is(runCtx.Err(), context.DeadlineExceeded): + return Result{}, &Failed{Reason: fmt.Sprintf("the package ran past its %ds bound", int(timeout.Seconds()))} + case ctx.Err() != nil: + return Result{}, &Failed{Reason: "the package was interrupted"} + } + return Result{}, &Failed{Reason: fmt.Sprintf("the package exited unsuccessfully: %v", err)} + } + return ReadResult(paths.out) +} + +type runPaths struct{ pkg, in, out, data string } + +// prepare empties and rebuilds the run tree. Every directory it removes or creates under the +// package must be a real directory, so a link left there can never aim the removal outside. +func prepare(run string, files Files) (runPaths, error) { + p := runPaths{ + pkg: filepath.Join(run, "pkg"), + in: filepath.Join(run, "in"), + out: filepath.Join(run, "out"), + data: filepath.Join(run, "data"), + } + if err := ensureRealDir(run); err != nil { + return p, err + } + if err := os.WriteFile(filepath.Join(run, ".gitignore"), []byte("*\n"), 0o644); err != nil { + return p, err + } + for _, d := range []string{p.pkg, p.in, p.out} { + if _, err := os.Lstat(d); err == nil { + if err := RealDir(d); err != nil { + return p, fmt.Errorf("refusing to empty %s: it is a link or not a directory", d) + } + if err := os.RemoveAll(d); err != nil { + return p, err + } + } + if err := os.Mkdir(d, 0o755); err != nil { + return p, err + } + } + if err := ensureRealDir(p.data); err != nil { + return p, err + } + abs, err := filepath.Abs(run) + if err != nil { + return p, err + } + p = runPaths{filepath.Join(abs, "pkg"), filepath.Join(abs, "in"), filepath.Join(abs, "out"), filepath.Join(abs, "data")} + for _, rel := range files.Paths { + if err := copyFile(filepath.Join(files.Dir, filepath.FromSlash(rel)), filepath.Join(p.pkg, filepath.FromSlash(rel)), files.Modes[rel]); err != nil { + return p, err + } + } + return p, nil +} + +func ensureRealDir(d string) error { + if _, err := os.Lstat(d); errors.Is(err, os.ErrNotExist) { + return os.Mkdir(d, 0o755) + } + if err := RealDir(d); err != nil { + return fmt.Errorf("refusing to use %s: it is a link or not a directory", d) + } + return nil +} + +// copyFile stages one file with the mode build gives it, so the staged package is the one +// unzip produces on the VM. +func copyFile(src, dst string, mode os.FileMode) error { + if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { + return err + } + in, err := os.Open(src) + if err != nil { + return err + } + defer in.Close() + out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_EXCL, mode) + if err != nil { + return err + } + if _, err := io.Copy(out, in); err != nil { + out.Close() + return err + } + return out.Close() +} + +// entrypointCommand is the runner's choice of interpreter: python3.11 for a .py entrypoint, +// otherwise the entrypoint itself. A laptop without python3.11 falls back to python3, saying so. +func entrypointCommand(pkgDir, entrypoint string, lookPath func(string) (string, error)) (string, []string, string, error) { + path := filepath.Join(pkgDir, filepath.FromSlash(entrypoint)) + if !strings.HasSuffix(entrypoint, ".py") { + return path, nil, "", nil + } + if p, err := lookPath("python3.11"); err == nil { + return p, []string{path}, "", nil + } + if p, err := lookPath("python3"); err == nil { + return p, []string{path}, "note: python3.11, the VM's interpreter, is not on PATH; running with python3", nil + } + return "", nil, "", fmt.Errorf("neither python3.11 nor python3 is on PATH") +} + +// environment is the passthrough taken from the caller's environment (LC_* included), then +// the runner's variables over it; nothing else is inherited. +func environment(from []string, set map[string]string) []string { + keep := map[string]string{} + for _, kv := range from { + k, v, ok := strings.Cut(kv, "=") + if !ok { + continue + } + if strings.HasPrefix(k, "LC_") || contains(passthrough, strings.ToUpper(k)) { + keep[k] = v + } + } + for k, v := range set { + keep[k] = v + } + out := make([]string, 0, len(keep)) + for k, v := range keep { + out = append(out, k+"="+v) + } + sort.Strings(out) + return out +} + +func contains(list []string, s string) bool { + for _, x := range list { + if x == s { + return true + } + } + return false +} diff --git a/internal/packages/run_test.go b/internal/packages/run_test.go new file mode 100644 index 0000000..71e55a6 --- /dev/null +++ b/internal/packages/run_test.go @@ -0,0 +1,265 @@ +package packages + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" +) + +const scopeJSON = `{ + "kind": "class", + "id": "0123456789abcdef0123456789abcdef0123456789abcdef", + "classes": [{"class_hash": "0123456789abcdef0123456789abcdef0123456789abcdef", "class_id": 6}], + "assignments": [{"offering_id": 9, "runnable_id": 1234, "name": "Moth 1.2", "url": "https://activity-player.concord.org/?activity=1"}] +}` + +// shellPackage writes a package whose entrypoint is run.sh with the given body. +func shellPackage(t *testing.T, body string, files map[string]string) (string, Files) { + t.Helper() + dir := t.TempDir() + tree := map[string]string{"manifest.json": goodManifest, "run.sh": "#!/bin/sh\n" + body} + for k, v := range files { + tree[k] = v + } + writeTree(t, dir, tree) + if err := os.Chmod(filepath.Join(dir, "run.sh"), 0o755); err != nil { + t.Fatal(err) + } + collected, err := Collect(dir) + if err != nil { + t.Fatal(err) + } + return dir, collected +} + +// runOpts is a run of the shell package against a fixed ref, with the given environment and an +// Applies that must not be called unless the test sets one. +func runOpts(t *testing.T, dir string, files Files, environ []string, stderr *bytes.Buffer) RunOptions { + t.Helper() + scope, err := ParseLocalScope([]byte(scopeJSON)) + if err != nil { + t.Fatal(err) + } + return RunOptions{ + Dir: dir, + Manifest: Manifest{Entrypoint: "run.sh", ExpectedDurationSeconds: 60}, + Files: files, + Scope: scope, + Applies: func(context.Context, URLs) (Verdict, error) { + t.Error("Applies called for a package that declares no patterns") + return Verdict{Applies: true}, nil + }, + Dataset: "learn.concord.org/wildfire", + Name: "wildfire", + Portal: "learn.concord.org", + DataRoot: "/data/root", + ReportSrv: "https://report.example", + Stderr: stderr, + Environ: func() []string { return environ }, + LookPath: exec.LookPath, + } +} + +func readEnv(t *testing.T, path string) map[string]string { + t.Helper() + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + env := map[string]string{} + for _, line := range strings.Split(string(raw), "\n") { + if k, v, ok := strings.Cut(line, "="); ok { + env[k] = v + } + } + return env +} + +func TestLocalScopeRefusesTheRunnersKeys(t *testing.T) { + withDataset := strings.Replace(scopeJSON, `"kind": "class",`, `"kind": "class", "dataset": "pkg-1",`, 1) + if _, err := ParseLocalScope([]byte(withDataset)); err == nil { + t.Error("a scope file setting dataset was accepted") + } + if _, err := ParseLocalScope([]byte(scopeJSON)); err != nil { + t.Errorf("a good scope file was refused: %v", err) + } +} + +func TestRunGivesThePackageTheRunnersLayout(t *testing.T) { + skipOnWindows(t, "the entrypoint is a shell script") + dir, files := shellPackage(t, `env > "$RD_OUTPUT_DIR/../env.txt" +cp "$RD_SCOPE_FILE" "$RD_OUTPUT_DIR/../scope.json" +ls -A > "$RD_OUTPUT_DIR/../ls.txt" +ls -l run.sh > "$RD_OUTPUT_DIR/../mode.txt" +echo ok > "$RD_OUTPUT_DIR/display.md" +`, map[string]string{"local-data/x.csv": "secret"}) + var stderr bytes.Buffer + environ := []string{"PATH=" + os.Getenv("PATH"), "HOME=/home/r", "LC_ALL=C", "https_proxy=http://proxy:3128", "AWS_SECRET_ACCESS_KEY=leak"} + res, err := Run(context.Background(), runOpts(t, dir, files, environ, &stderr)) + if err != nil { + t.Fatalf("%v\n%s", err, stderr.String()) + } + run := filepath.Join(dir, RunDirName) + if res.DisplayPath != filepath.Join(run, "out", "display.md") { + t.Errorf("display path %s", res.DisplayPath) + } + + env := readEnv(t, filepath.Join(run, "env.txt")) + for k, want := range map[string]string{ + "RD_DATASET": "learn.concord.org/wildfire", + "CC_DATA_PORTAL": "learn.concord.org", + "CC_DATA_ROOT": "/data/root", + "CC_DATA_LOCAL": filepath.Join(run, "data"), + "RD_DATA_DIR": filepath.Join(run, "data"), + "RD_SCOPE_FILE": filepath.Join(run, "in", "scope.json"), + "RD_OUTPUT_DIR": filepath.Join(run, "out"), + "RD_REPORT_SERVER_URL": "https://report.example", + "LC_ALL": "C", + "https_proxy": "http://proxy:3128", + } { + if env[k] != want { + t.Errorf("%s = %q, want %q", k, env[k], want) + } + } + if _, ok := env["AWS_SECRET_ACCESS_KEY"]; ok { + t.Error("a variable outside the passthrough reached the package") + } + + var scope map[string]any + raw, err := os.ReadFile(filepath.Join(run, "scope.json")) + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(raw, &scope); err != nil { + t.Fatal(err) + } + if scope["dataset"] != "wildfire" || scope["clue_source"] != "firebase" || scope["output_dir"] != filepath.Join(run, "out") { + t.Errorf("scope.json = %v", scope) + } + + ls, err := os.ReadFile(filepath.Join(run, "ls.txt")) + if err != nil { + t.Fatal(err) + } + if got := strings.Fields(string(ls)); strings.Join(got, ",") != "manifest.json,run.sh" { + t.Errorf("working directory holds %v, want only what build ships", got) + } + mode, err := os.ReadFile(filepath.Join(run, "mode.txt")) + if err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(string(mode), "-rwxr-xr-x") { + t.Errorf("staged run.sh mode: %s", mode) + } + if gi, err := os.ReadFile(filepath.Join(run, ".gitignore")); err != nil || string(gi) != "*\n" { + t.Errorf(".gitignore = %q, %v", gi, err) + } +} + +func TestRunRefusesBeforeStarting(t *testing.T) { + skipOnWindows(t, "the entrypoint is a shell script") + dir, files := shellPackage(t, `touch "$RD_OUTPUT_DIR/../started"`+"\n", nil) + var stderr bytes.Buffer + o := runOpts(t, dir, files, nil, &stderr) + o.Manifest.URLs = URLs{All: []string{"*//wildfire.concord.org/*"}} + o.Applies = func(context.Context, URLs) (Verdict, error) { + return Verdict{Reason: "no URL in this class matches the required pattern *//wildfire.concord.org/*"}, nil + } + _, err := Run(context.Background(), o) + var refused *Refused + if !errors.As(err, &refused) || err.Error() != "the package does not apply to this class: no URL in this class matches the required pattern *//wildfire.concord.org/*" { + t.Errorf("err = %v", err) + } + if _, err := os.Stat(filepath.Join(dir, RunDirName, "started")); err == nil { + t.Error("the entrypoint started for a package that does not apply") + } + + o = runOpts(t, dir, files, nil, &stderr) + o.Manifest.CluePrepull = true + if _, err := Run(context.Background(), o); !errors.As(err, &refused) { + t.Errorf("clue_prepull: err = %v, want Refused", err) + } +} + +func TestRunWarnsAndRunsWhenApplicabilityIsUnconfirmed(t *testing.T) { + skipOnWindows(t, "the entrypoint is a shell script") + dir, files := shellPackage(t, `echo ok > "$RD_OUTPUT_DIR/display.md"`+"\n", nil) + var stderr bytes.Buffer + o := runOpts(t, dir, files, []string{"PATH=" + os.Getenv("PATH")}, &stderr) + o.Manifest.URLs = URLs{Any: []string{"*question-interactives/*"}} + o.Applies = func(context.Context, URLs) (Verdict, error) { return Verdict{Unconfirmed: true}, nil } + if _, err := Run(context.Background(), o); err != nil { + t.Fatal(err) + } + if !strings.Contains(stderr.String(), "cannot check applicability yet") { + t.Errorf("stderr = %q, want the unconfirmed warning", stderr.String()) + } +} + +func TestRunKillsAtTheBound(t *testing.T) { + skipOnWindows(t, "the entrypoint is a shell script") + dir, files := shellPackage(t, "sleep 30 &\nwait\n", nil) + var stderr bytes.Buffer + o := runOpts(t, dir, files, []string{"PATH=" + os.Getenv("PATH")}, &stderr) + o.Timeout = 300 * time.Millisecond + start := time.Now() + _, err := Run(context.Background(), o) + var failed *Failed + if !errors.As(err, &failed) || !strings.Contains(err.Error(), "past its") || time.Since(start) > 5*time.Second { + t.Errorf("err = %v after %v", err, time.Since(start)) + } +} + +func TestRunKillsTheGroupWhenInterrupted(t *testing.T) { + skipOnWindows(t, "the entrypoint is a shell script") + dir, files := shellPackage(t, "sleep 30 &\nwait\n", nil) + var stderr bytes.Buffer + ctx, cancel := context.WithCancel(context.Background()) + time.AfterFunc(300*time.Millisecond, cancel) + start := time.Now() + _, err := Run(ctx, runOpts(t, dir, files, []string{"PATH=" + os.Getenv("PATH")}, &stderr)) + var failed *Failed + if !errors.As(err, &failed) || !strings.Contains(err.Error(), "interrupted") || time.Since(start) > 5*time.Second { + t.Errorf("err = %v after %v", err, time.Since(start)) + } +} + +func TestRunLeavesNothingBehind(t *testing.T) { + skipOnWindows(t, "the entrypoint is a shell script") + dir, files := shellPackage(t, `( sleep 1; echo late > "$RD_OUTPUT_DIR/late.txt" ) >/dev/null 2>&1 & +echo ok > "$RD_OUTPUT_DIR/display.md" +`, nil) + var stderr bytes.Buffer + if _, err := Run(context.Background(), runOpts(t, dir, files, []string{"PATH=" + os.Getenv("PATH")}, &stderr)); err != nil { + t.Fatal(err) + } + time.Sleep(1500 * time.Millisecond) + if _, err := os.Stat(filepath.Join(dir, RunDirName, "out", "late.txt")); err == nil { + t.Error("a background process the package left behind wrote after the run") + } +} + +func TestRunRefusesALinkedRunDirectory(t *testing.T) { + skipOnWindows(t, "links need privileges on Windows") + dir, files := shellPackage(t, `echo ok > "$RD_OUTPUT_DIR/display.md"`+"\n", nil) + elsewhere := t.TempDir() + keep := filepath.Join(elsewhere, "out", "keep.txt") + writeTree(t, elsewhere, map[string]string{"out/keep.txt": "keep"}) + if err := os.Symlink(elsewhere, filepath.Join(dir, RunDirName)); err != nil { + t.Fatal(err) + } + var stderr bytes.Buffer + if _, err := Run(context.Background(), runOpts(t, dir, files, nil, &stderr)); err == nil { + t.Error("a linked run directory was used") + } + if _, err := os.Stat(keep); err != nil { + t.Errorf("a file behind the link was removed: %v", err) + } +} diff --git a/internal/packages/scope.go b/internal/packages/scope.go new file mode 100644 index 0000000..aa26153 --- /dev/null +++ b/internal/packages/scope.go @@ -0,0 +1,99 @@ +package packages + +import ( + "bytes" + "encoding/json" + "fmt" + "regexp" +) + +var classHashRe = regexp.MustCompile(`^[0-9a-f]{48}$`) + +// Class and Assignment are scope.json's entries. +type Class struct { + ClassHash string `json:"class_hash"` + ClassID int64 `json:"class_id"` +} + +type Assignment struct { + OfferingID int64 `json:"offering_id"` + RunnableID int64 `json:"runnable_id"` + Name *string `json:"name"` + URL string `json:"url"` +} + +// LocalScope is the four keys rigse supplies on the VM, which an author writes for a laptop. +type LocalScope struct { + Kind string `json:"kind"` + ID string `json:"id"` + Classes []Class `json:"classes"` + Assignments []Assignment `json:"assignments"` +} + +// ScopeFile is scope.json as the runner writes it. ClueSource is always "firebase" today, the +// runner's constant; Dataset is the bare dataset name, as the runner writes pkg-, while +// RD_DATASET carries the full ref. +type ScopeFile struct { + LocalScope + ClueSource string `json:"clue_source"` + Dataset string `json:"dataset"` + OutputDir string `json:"output_dir"` +} + +// MarshalJSON keeps the runner's key order, so a diff between a local and a VM scope.json is +// only the values. +func (s ScopeFile) MarshalJSON() ([]byte, error) { + type ordered struct { + Kind string `json:"kind"` + ID string `json:"id"` + Classes []Class `json:"classes"` + Assignments []Assignment `json:"assignments"` + ClueSource string `json:"clue_source"` + Dataset string `json:"dataset"` + OutputDir string `json:"output_dir"` + } + return json.Marshal(ordered{s.Kind, s.ID, s.Classes, s.Assignments, s.ClueSource, s.Dataset, s.OutputDir}) +} + +// ParseLocalScope reads an author's scope file, refusing any key the four do not name (the +// other three are the runner's to set) and any shape the runner would never send. +func ParseLocalScope(data []byte) (LocalScope, error) { + var s LocalScope + dec := json.NewDecoder(bytes.NewReader(data)) + dec.DisallowUnknownFields() + if err := dec.Decode(&s); err != nil { + return s, fmt.Errorf("scope file: %v (it holds exactly kind, id, classes and assignments)", err) + } + switch { + case s.Kind == "": + return s, fmt.Errorf("scope file: kind is required") + case s.ID == "": + return s, fmt.Errorf("scope file: id is required") + case len(s.Classes) == 0: + return s, fmt.Errorf("scope file: classes must name at least one class") + case s.Assignments == nil: + return s, fmt.Errorf("scope file: assignments is required (an empty list is allowed)") + } + for i, c := range s.Classes { + if !classHashRe.MatchString(c.ClassHash) || c.ClassID < 1 { + return s, fmt.Errorf("scope file: classes[%d] needs a 48-hex class_hash and a positive class_id", i) + } + } + for i, a := range s.Assignments { + if a.OfferingID < 1 || a.RunnableID < 1 { + return s, fmt.Errorf("scope file: assignments[%d] needs a positive offering_id and runnable_id", i) + } + } + return s, nil +} + +// AssignmentURLs is the scope's half of the URLs a package's patterns are matched against. +func (s LocalScope) AssignmentURLs() []string { + var urls []string + for _, a := range s.Assignments { + if a.URL != "" { + urls = append(urls, a.URL) + } + } + return urls +} diff --git a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md index 61f5cc0..124c529 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md +++ b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md @@ -586,7 +586,7 @@ import ( var classHashRe = regexp.MustCompile(`^[0-9a-f]{48}$`) -// Class and Assignment are scope.json's entries (final-design.md section 10). +// Class and Assignment are scope.json's entries. type Class struct { ClassHash string `json:"class_hash"` ClassID int64 `json:"class_id"` @@ -752,7 +752,7 @@ type RunOptions struct { // Run prepares the runner's layout, runs the entrypoint and reads its result. func Run(ctx context.Context, o RunOptions) (Result, error) { if o.Manifest.CluePrepull { - return Result{}, &Refused{Reason: "the package sets clue_prepull, and cc-data cannot fetch CLUE data yet (final-design.md section 13), so it runs only on the VM"} + return Result{}, &Refused{Reason: "the package sets clue_prepull, and cc-data cannot fetch CLUE data yet, so it runs only on the VM"} } if o.Manifest.URLs.DeclaresPatterns() { v, err := o.Applies(ctx, o.Manifest.URLs) @@ -813,8 +813,7 @@ func Run(ctx context.Context, o RunOptions) (Result, error) { cmd.Stderr = o.Stderr killGroup(cmd) err = cmd.Run() - // The runner kills everything the package left behind before it reads the output, so a - // background process can neither outlive the run nor write a later run's files. + // As on the VM, nothing the package started outlives it or writes after it. reapGroup(cmd) if err != nil { switch { @@ -999,7 +998,7 @@ Tests (the entrypoint is a `/bin/sh` script, so these skip on Windows): - The package dumps its environment, copies `scope.json` and lists its working directory. - The environment carries `RD_DATASET` (the full ref), `CC_DATA_PORTAL`, `RD_SCOPE_FILE` a passed-through `LC_ALL` and lower-case `https_proxy`, and not an injected `AWS_SECRET_ACCESS_KEY`. - `scope.json` has the bare `dataset`, `clue_source: "firebase"` and the absolute `output_dir`. - - The working directory has no `local-data/`, and `.gitignore` is `*`. + - The working directory holds exactly `manifest.json` and `run.sh` (no `local-data/`), the staged `run.sh` keeps mode 0755, and `.gitignore` is `*`. - **`TestRunRefusesBeforeStarting`**: when `Applies` says no, the error is the runner's prefix plus the reason, and the entrypoint never starts. A `clue_prepull` package is refused. - **`TestRunWarnsAndRunsWhenApplicabilityIsUnconfirmed`**: an unconfirmed verdict prints the warning and runs. - **`TestRunKillsAtTheBound`**: `sleep 30 & wait` under a 300 ms bound fails as "past its bound" within 5 seconds. From a3fc7c5d3312dcc9cff08a5bbbce5c7ff8117a16 Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 16:00:26 -0400 Subject: [PATCH 06/17] feat: add the package publish, validate and applies calls [REPORT-146] Client.do delegates to a new send that names the body's media type, so the zip routes share the bearer, the cross-origin redirect refusal, the no-retry rule for POST and the error-envelope decoding with every other call. PublishPackage and ValidatePackage post the raw zip as application/zip, with origin and official as query parameters only when given. PackageApplies posts the patterns and the scope's assignment URLs. RouteMissing recognizes the 404 a report-server without the validate and applies routes answers. The command tests exercise the requests. --- internal/api/client.go | 11 +- internal/api/packages.go | 107 ++++++++++++++++++ .../implementation.md | 52 ++++----- 3 files changed, 138 insertions(+), 32 deletions(-) create mode 100644 internal/api/packages.go diff --git a/internal/api/client.go b/internal/api/client.go index b48db69..e61c75e 100644 --- a/internal/api/client.go +++ b/internal/api/client.go @@ -95,6 +95,11 @@ const maxBodyBytes = 64 << 20 // do runs a request under the retry policy and returns the 2xx body, or a typed // *APIError (contract) / *TransientError (budget exhausted). func (c *Client) do(ctx context.Context, method, path string, query url.Values, body []byte) ([]byte, error) { + return c.send(ctx, method, path, query, body, "application/json") +} + +// send is do with the body's media type named, for the routes that take raw bytes. +func (c *Client) send(ctx context.Context, method, path string, query url.Values, body []byte, contentType string) ([]byte, error) { u := c.BaseURL + path if len(query) > 0 { u += "?" + query.Encode() @@ -110,7 +115,7 @@ func (c *Client) do(ctx context.Context, method, path string, query url.Values, return nil, err } - data, status, err := c.attempt(ctx, method, u, body) + data, status, err := c.attempt(ctx, method, u, body, contentType) if err != nil { last = err // A non-idempotent request (POST) may have reached the server, so @@ -139,7 +144,7 @@ func (c *Client) do(ctx context.Context, method, path string, query url.Values, // attempt performs one HTTP request under a per-attempt deadline and returns the // body and status; a transport error (including deadline) is returned as err. -func (c *Client) attempt(ctx context.Context, method, u string, body []byte) ([]byte, int, error) { +func (c *Client) attempt(ctx context.Context, method, u string, body []byte, contentType string) ([]byte, int, error) { reqCtx := ctx var cancel context.CancelFunc if c.RequestTimeout > 0 { @@ -160,7 +165,7 @@ func (c *Client) attempt(ctx context.Context, method, u string, body []byte) ([] } req.Header.Set("Accept", "application/json") if body != nil { - req.Header.Set("Content-Type", "application/json") + req.Header.Set("Content-Type", contentType) } resp, err := c.HTTP.Do(req) diff --git a/internal/api/packages.go b/internal/api/packages.go new file mode 100644 index 0000000..24c6caa --- /dev/null +++ b/internal/api/packages.go @@ -0,0 +1,107 @@ +package api + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/url" +) + +// PublishedPackage is report-server's answer to a publish. +type PublishedPackage struct { + CatalogID int64 `json:"catalog_id"` + Identity string `json:"identity"` + Version string `json:"version"` + Checksum string `json:"checksum"` + Visibility string `json:"visibility"` + Official bool `json:"official"` + CurrentVersion string `json:"current_version"` +} + +// PublishPackage posts a package archive as the raw request body. origin is "" for the +// caller's own users/, or projects/; official asks the server to mark it official, +// which only a publisher may. Like every POST it is never retried. +func (c *Client) PublishPackage(ctx context.Context, archive []byte, origin string, official bool) (*PublishedPackage, error) { + data, err := c.send(ctx, http.MethodPost, "/api/v1/packages", packageQuery(origin, official), archive, "application/zip") + if err != nil { + return nil, err + } + var out PublishedPackage + if err := json.Unmarshal(data, &out); err != nil { + return nil, err + } + return &out, nil +} + +// ValidatedPackage is what a publish of the archive would record. The last two are not +// refusals: each says only a publish would be refused now, with publish's own message for the +// second. +type ValidatedPackage struct { + Identity string `json:"identity"` + Version string `json:"version"` + Checksum string `json:"checksum"` + Visibility string `json:"visibility"` + AlreadyPublished bool `json:"already_published"` + PublishingUnavailable *string `json:"publishing_unavailable"` +} + +// ValidatePackage asks report-server to apply every publish check to the archive and stop +// before storing it, with the same origin and official a publish would send. A refusal is the +// same coded error a publish would answer. +func (c *Client) ValidatePackage(ctx context.Context, archive []byte, origin string, official bool) (*ValidatedPackage, error) { + data, err := c.send(ctx, http.MethodPost, "/api/v1/packages/validate", packageQuery(origin, official), archive, "application/zip") + if err != nil { + return nil, err + } + var out ValidatedPackage + if err := json.Unmarshal(data, &out); err != nil { + return nil, err + } + return &out, nil +} + +// AppliesRequest names the patterns and the scope's assignment URLs, which report-service's +// deriver follows into their activities, so the URLs matched are the runner's. +type AppliesRequest struct { + URLs map[string][]string `json:"urls"` + AssignmentURLs []string `json:"assignment_urls,omitempty"` +} + +// AppliesAnswer is report-server's verdict, with what the deriver read and could not. +type AppliesAnswer struct { + Applies bool `json:"applies"` + Reason string `json:"reason"` + Unread []struct { + URL string `json:"url"` + Reason string `json:"reason"` + } `json:"unread"` + Truncated bool `json:"truncated"` +} + +// PackageApplies asks report-server's one matcher whether the patterns match the scope. +func (c *Client) PackageApplies(ctx context.Context, req AppliesRequest) (*AppliesAnswer, error) { + var out AppliesAnswer + if err := c.postJSON(ctx, "/api/v1/packages/applies", req, &out); err != nil { + return nil, err + } + return &out, nil +} + +func packageQuery(origin string, official bool) url.Values { + q := url.Values{} + if origin != "" { + q.Set("origin", origin) + } + if official { + q.Set("official", "true") + } + return q +} + +// RouteMissing reports a 404 from a route this cc-data expects and the server does not have +// yet: a report-server older than the validate and applies routes. +func RouteMissing(err error) bool { + var apiErr *APIError + return errors.As(err, &apiErr) && apiErr.Status == http.StatusNotFound +} diff --git a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md index 124c529..1cafed6 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md +++ b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md @@ -1043,7 +1043,7 @@ Tests (the entrypoint is a `/bin/sh` script, so these skip on Windows): + return c.send(ctx, method, path, query, body, "application/json") +} + -+// send is do with the body's media type named, for the one route that takes raw bytes. ++// send is do with the body's media type named, for the routes that take raw bytes. +func (c *Client) send(ctx context.Context, method, path string, query url.Values, body []byte, contentType string) ([]byte, error) { u := c.BaseURL + path if len(query) > 0 { @@ -1087,7 +1087,7 @@ import ( "net/url" ) -// PublishedPackage is report-server's answer to a publish (REPORT-142). +// PublishedPackage is report-server's answer to a publish. type PublishedPackage struct { CatalogID int64 `json:"catalog_id"` Identity string `json:"identity"` @@ -1102,14 +1102,7 @@ type PublishedPackage struct { // caller's own users/, or projects/; official asks the server to mark it official, // which only a publisher may. Like every POST it is never retried. func (c *Client) PublishPackage(ctx context.Context, archive []byte, origin string, official bool) (*PublishedPackage, error) { - q := url.Values{} - if origin != "" { - q.Set("origin", origin) - } - if official { - q.Set("official", "true") - } - data, err := c.send(ctx, http.MethodPost, "/api/v1/packages", q, archive, "application/zip") + data, err := c.send(ctx, http.MethodPost, "/api/v1/packages", packageQuery(origin, official), archive, "application/zip") if err != nil { return nil, err } @@ -1120,10 +1113,9 @@ func (c *Client) PublishPackage(ctx context.Context, archive []byte, origin stri return &out, nil } -// ValidatedPackage is what a publish of the archive would record, as report-server's validate -// route answers it without recording anything. Neither of the last two is a refusal: both say -// a publish of this zip would be refused right now (the version exists, or the portal cannot -// store packages yet), while running it is fine. PublishingUnavailable is publish's own message. +// ValidatedPackage is what a publish of the archive would record. The last two are not +// refusals: each says only a publish would be refused now, with publish's own message for the +// second. type ValidatedPackage struct { Identity string `json:"identity"` Version string `json:"version"` @@ -1137,14 +1129,7 @@ type ValidatedPackage struct { // before storing it, with the same origin and official a publish would send. A refusal is the // same coded error a publish would answer. func (c *Client) ValidatePackage(ctx context.Context, archive []byte, origin string, official bool) (*ValidatedPackage, error) { - q := url.Values{} - if origin != "" { - q.Set("origin", origin) - } - if official { - q.Set("official", "true") - } - data, err := c.send(ctx, http.MethodPost, "/api/v1/packages/validate", q, archive, "application/zip") + data, err := c.send(ctx, http.MethodPost, "/api/v1/packages/validate", packageQuery(origin, official), archive, "application/zip") if err != nil { return nil, err } @@ -1155,20 +1140,18 @@ func (c *Client) ValidatePackage(ctx context.Context, archive []byte, origin str return &out, nil } -// AppliesRequest names the patterns and the scope's URLs. AssignmentURLs are followed into -// their activities by report-service's deriver, so the URLs matched are the runner's. +// AppliesRequest names the patterns and the scope's assignment URLs, which report-service's +// deriver follows into their activities, so the URLs matched are the runner's. type AppliesRequest struct { URLs map[string][]string `json:"urls"` AssignmentURLs []string `json:"assignment_urls,omitempty"` - ScopeURLs []string `json:"scope_urls,omitempty"` } // AppliesAnswer is report-server's verdict, with what the deriver read and could not. type AppliesAnswer struct { - Applies bool `json:"applies"` - Reason string `json:"reason"` - InteractiveURLs []string `json:"interactive_urls"` - Unread []struct { + Applies bool `json:"applies"` + Reason string `json:"reason"` + Unread []struct { URL string `json:"url"` Reason string `json:"reason"` } `json:"unread"` @@ -1184,6 +1167,17 @@ func (c *Client) PackageApplies(ctx context.Context, req AppliesRequest) (*Appli return &out, nil } +func packageQuery(origin string, official bool) url.Values { + q := url.Values{} + if origin != "" { + q.Set("origin", origin) + } + if official { + q.Set("official", "true") + } + return q +} + // RouteMissing reports a 404 from a route this cc-data expects and the server does not have // yet: a report-server older than the validate and applies routes. func RouteMissing(err error) bool { From 8de65a9afff32cdc7db2d8def2ef776309bd9596 Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 16:04:34 -0400 Subject: [PATCH 07/17] feat: add the cc-data package commands [REPORT-146] - init writes a manifest.json skeleton and an embedded, stdlib-only run.py stub. The stub reuses or creates a Student ID Mapping run filtered to the scope's class, re-reads it live, pulls its answers and counts through run_answers and student_id_mapping. - run checks the package with report-server's validate route, asks its applies route through a 5-minute call for the deriver, and runs the package under the runner's layout. An interrupt cancels the run so the package's process group dies with it, and errors keep the server's code and exit class. - build zips reproducibly, validates with the --origin and --official the publish will use, warns on an already published version or a portal that cannot store packages yet, and writes to .cc-data-build/-.zip unless --out says otherwise. A name or version that would make that a path is refused. - publish posts the zip with the cc-data token, never retries, and checks the checksum report-server records. A report-server without the validate or applies route answers 404, which build and run report as a warning and continue past. --- cmd/package.go | 338 +++++++++++++ cmd/package_test.go | 475 ++++++++++++++++++ cmd/root.go | 1 + internal/packages/build.go | 57 +++ internal/packages/init.go | 55 ++ internal/packages/run.go | 8 +- internal/packages/template/run.py | 87 ++++ .../implementation.md | 45 +- 8 files changed, 1050 insertions(+), 16 deletions(-) create mode 100644 cmd/package.go create mode 100644 cmd/package_test.go create mode 100644 internal/packages/build.go create mode 100644 internal/packages/init.go create mode 100644 internal/packages/template/run.py diff --git a/cmd/package.go b/cmd/package.go new file mode 100644 index 0000000..86f5606 --- /dev/null +++ b/cmd/package.go @@ -0,0 +1,338 @@ +package cmd + +import ( + "context" + "errors" + "fmt" + "os" + "os/exec" + "os/signal" + "path/filepath" + "strings" + "time" + + "github.com/concord-consortium/cc-data-cli/internal/api" + "github.com/concord-consortium/cc-data-cli/internal/output" + "github.com/concord-consortium/cc-data-cli/internal/packages" + "github.com/spf13/cobra" +) + +const packageLong = `Develop, test and publish a Researcher Dashboard package. + +A package is a directory holding manifest.json and an entrypoint. init writes both; run +executes the package against your own dataset under the runner's rules; build zips it; +publish registers the zip in the catalog with your cc-data token. + +What run cannot reproduce from the VM: the package runs as you, with no network sandbox, +and with HOME, PATH, the locale and any proxy settings passed through so its own cc-data +calls find your login and reach the server. A .py entrypoint runs with python3.11 when it is on PATH, as on the VM, else +python3.` + +func newPackageCmd() *cobra.Command { + cmd := &cobra.Command{ + Use: "package", + Short: "Develop, test and publish a Researcher Dashboard package", + Long: packageLong, + } + cmd.AddCommand(newPackageInitCmd(), newPackageRunCmd(), newPackageBuildCmd(), newPackagePublishCmd()) + return cmd +} + +// packageDir is the optional [dir] argument, defaulting to the current directory. +func packageDir(args []string) string { + if len(args) == 1 { + return args[0] + } + return "." +} + +// loadPackage collects a package directory and validates its manifest against what it holds. +func loadPackage(dir string) (packages.Manifest, packages.Files, error) { + files, err := packages.Collect(dir) + if err != nil { + return packages.Manifest{}, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_PACKAGE", Message: err.Error()} + } + data, err := os.ReadFile(filepath.Join(dir, "manifest.json")) + if err != nil { + return packages.Manifest{}, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_MANIFEST", Message: fmt.Sprintf("no manifest.json in %s", dir)} + } + m, err := packages.ReadManifest(data, files.Set()) + if err != nil { + return m, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_MANIFEST", Message: err.Error()} + } + if err := files.CheckEntrypointMode(m.Entrypoint); err != nil { + return m, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_PACKAGE", Message: err.Error()} + } + return m, files, nil +} + +func newPackageInitCmd() *cobra.Command { + var name string + cmd := &cobra.Command{ + Use: "init [dir]", + Short: "Write a manifest.json skeleton and a run.py stub", + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + dir := packageDir(args) + written, err := packages.Init(dir, name) + if err != nil { + return output.Usagef("%v", err) + } + return output.ResultLine(map[string]any{"dir": dir, "written": written}) + }, + } + cmd.Flags().StringVar(&name, "name", "", "the package name (default: the directory's name, when it fits ^[a-z0-9][a-z0-9-]{0,62}$)") + return cmd +} + +func newPackageRunCmd() *cobra.Command { + var datasetRef, scopePath string + cmd := &cobra.Command{ + Use: "run [dir] --dataset --scope ", + Short: "Run a package locally exactly as the dashboard's runner will", + Long: "Run a package against your own dataset under the runner's rules: the same scope.json,\n" + + "environment variables, output files, display.md cap and applicability check.\n\n" + + "--scope names a file holding the scope's kind, id, classes and assignments.\n" + + "Results are written under /.cc-data-run/, which build never ships.", + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + if datasetRef == "" || scopePath == "" { + return output.Usagef("--dataset and --scope are required") + } + dir := packageDir(args) + m, files, err := loadPackage(dir) + if err != nil { + return err + } + raw, err := os.ReadFile(scopePath) + if err != nil { + return output.Usagef("reading the scope file: %v", err) + } + scope, err := packages.ParseLocalScope(raw) + if err != nil { + return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: err.Error()} + } + cfg, dataRoot, err := loadRuntime() + if err != nil { + return err + } + ref, err := resolveRef(cfg, datasetRef) + if err != nil { + return err + } + client, err := api.ForPortal(ref.Portal) + if err != nil { + return err + } + // The terminal's Ctrl-C cannot reach the package's own process group, so it cancels the run. + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) + defer stop() + archive, err := packages.Zip(files) + if err != nil { + return output.Internalf("%v", err) + } + // A version that is already published, or a portal that cannot store packages yet, + // still runs; only a publish is refused. + if _, err := validate(ctx, client, archive, "", false); err != nil { + return err + } + start := time.Now() + res, err := packages.Run(ctx, packages.RunOptions{ + Dir: dir, Manifest: m, Files: files, Scope: scope, Applies: appliesFor(client, scope), + Dataset: ref.String(), Name: ref.Name, Portal: ref.Portal.Host(), DataRoot: dataRoot, + ReportSrv: client.BaseURL, Stderr: output.Stderr(), Environ: os.Environ, LookPath: exec.LookPath, + }) + if err != nil { + return packageRunError(err) + } + return output.ResultLine(map[string]any{ + "display": res.DisplayPath, "display_bytes": res.DisplayBytes, "summary": res.Summary, + "counts": res.Counts, "elapsed_seconds": int(time.Since(start).Seconds()), + }) + }, + } + cmd.Flags().StringVar(&datasetRef, "dataset", "", "the dataset the package pulls into and reads: /") + cmd.Flags().StringVar(&scopePath, "scope", "", "a JSON file holding the scope's kind, id, classes and assignments") + return cmd +} + +// derivationTimeout covers report-service's deriver, which may spend up to 240 seconds +// fetching a scope's activities; the client's 60-second default would cut it off. +const derivationTimeout = 5 * time.Minute + +// validate asks report-server whether a publish of the archive would be accepted, and returns +// its answer. A report-server without the route is not a refusal: the archive is unchecked, +// the answer is nil, and saying so is the whole result. +func validate(ctx context.Context, client *api.Client, archive []byte, origin string, official bool) (*api.ValidatedPackage, error) { + ans, err := client.ValidatePackage(ctx, archive, origin, official) + switch { + case api.RouteMissing(err): + output.Warnf("report-server has no validate route yet, so this package is not checked against the catalog's rules until publish") + return nil, nil + case err != nil: + return nil, api.AsCLIError(err) + } + return ans, nil +} + +// appliesFor is package run's applicability check: report-server's one matcher, against the +// scope's assignment URLs and the interactive URLs report-service's deriver finds in them. +func appliesFor(client *api.Client, scope packages.LocalScope) func(context.Context, packages.URLs) (packages.Verdict, error) { + return func(ctx context.Context, urls packages.URLs) (packages.Verdict, error) { + long := *client + long.RequestTimeout = derivationTimeout + ans, err := long.PackageApplies(ctx, api.AppliesRequest{ + URLs: map[string][]string{"all": urls.All, "any": urls.Any, "none": urls.None}, + AssignmentURLs: scope.AssignmentURLs(), + }) + switch { + case api.RouteMissing(err): + return packages.Verdict{Unconfirmed: true}, nil + case err != nil: + return packages.Verdict{}, api.AsCLIError(err) + } + for _, u := range ans.Unread { + output.Warnf("the class profile could not read %s (%s), so its interactives are not matched", u.URL, u.Reason) + } + if ans.Truncated { + output.Warnf("the class profile was truncated, so not every interactive URL was matched") + } + return packages.Verdict{Applies: ans.Applies, Reason: ans.Reason}, nil + } +} + +func packageRunError(err error) error { + var refused *packages.Refused + var failed *packages.Failed + var out *packages.OutputRefused + var cliErr *output.CLIError + switch { + case errors.As(err, &cliErr): + return cliErr + case errors.As(err, &refused): + return &output.CLIError{ExitCode: output.ExitInternal, Code: "PACKAGE_REFUSED", Message: err.Error()} + case errors.As(err, &failed): + return &output.CLIError{ExitCode: output.ExitInternal, Code: "PACKAGE_FAILED", Message: err.Error()} + case errors.As(err, &out): + return &output.CLIError{ExitCode: output.ExitInternal, Code: "PACKAGE_OUTPUT_REFUSED", Message: err.Error()} + } + return output.Internalf("%v", err) +} + +type packageBuildFlags struct { + out, portal, origin string + official bool +} + +// run builds, validates and writes; it takes the client so it can be exercised against a test +// server without a stored credential. +func (f packageBuildFlags) run(ctx context.Context, client *api.Client, dir string) error { + m, files, err := loadPackage(dir) + if err != nil { + return err + } + out := f.out + if out == "" { + // The manifest is unchecked until validate answers, and unchecked for good on a server + // without the route, so its name and version must not be able to steer the path. + name := fmt.Sprintf("%s-%s.zip", m.Name, m.Version) + if strings.ContainsAny(name, `/\`) { + return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_MANIFEST", Message: fmt.Sprintf("the manifest's name and version make %q, which is not a file name; fix them or pass --out", name)} + } + out = filepath.Join(dir, packages.BuildDirName, name) + } else if err := packages.CheckOutsidePackage(dir, out); err != nil { + return output.Usagef("%v", err) + } + archive, err := packages.Zip(files) + if err != nil { + return output.Internalf("%v", err) + } + ans, err := validate(ctx, client, archive, f.origin, f.official) + if err != nil { + return err + } + if ans != nil && ans.AlreadyPublished { + output.Warnf("%s %s is already published; publish will refuse it until the version changes", ans.Identity, ans.Version) + } + if ans != nil && ans.PublishingUnavailable != nil { + output.Warnf("this package cannot be published yet: %s", *ans.PublishingUnavailable) + } + if err := packages.WriteBuild(out, archive); err != nil { + return output.Internalf("%v", err) + } + return output.ResultLine(map[string]any{ + "path": out, "checksum": packages.Checksum(archive), "bytes": len(archive), "files": len(files.Paths), + }) +} + +func newPackageBuildCmd() *cobra.Command { + var f packageBuildFlags + cmd := &cobra.Command{ + Use: "build [dir]", + Short: "Zip a package reproducibly, check it against the catalog's rules and print its checksum", + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + client, err := reportsClient(f.portal) + if err != nil { + return err + } + return f.run(context.Background(), client, packageDir(args)) + }, + } + cmd.Flags().StringVar(&f.out, "out", "", "where to write the zip (default: /.cc-data-build/-.zip)") + cmd.Flags().StringVar(&f.portal, "portal", "", "portal whose catalog rules to check against: an environment alias or a hostname") + cmd.Flags().StringVar(&f.origin, "origin", "", "projects/, when the package will be published as a project's") + cmd.Flags().BoolVar(&f.official, "official", false, "check it as an official publish will be checked (publisher role only)") + return cmd +} + +type packagePublishFlags struct { + portal, origin string + official, asJSON bool +} + +// run publishes the archive; it takes the client so it can be exercised against a test server +// without a stored credential. +func (f packagePublishFlags) run(ctx context.Context, client *api.Client, archive []byte) error { + got, err := client.PublishPackage(ctx, archive, f.origin, f.official) + if err != nil { + return api.AsWriteCLIError(err, "Publishing again is safe: ALREADY_EXISTS then means this attempt landed") + } + if want := packages.Checksum(archive); got.Checksum != want { + return output.Internalf("report-server recorded checksum %s, but the archive sent is %s", got.Checksum, want) + } + if f.asJSON { + return output.ResultLine(got) + } + fmt.Fprintf(output.Stdout(), "identity %s\nversion %s\nvisibility %s\nofficial %t\ncurrent version %s\nchecksum %s\n", + got.Identity, got.Version, got.Visibility, got.Official, got.CurrentVersion, got.Checksum) + return nil +} + +func newPackagePublishCmd() *cobra.Command { + var f packagePublishFlags + cmd := &cobra.Command{ + Use: "publish [--portal ]", + Short: "Publish a built package to the catalog with your cc-data token", + Long: "POST the zip to report-server's catalog, which creates the package private on its first\n" + + "publish and records the version. --origin projects/ publishes a project package;\n" + + "--official needs the publisher role. The request is never retried.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + archive, err := os.ReadFile(args[0]) + if err != nil { + return output.Usagef("reading %s: %v", args[0], err) + } + client, err := reportsClient(f.portal) + if err != nil { + return err + } + return f.run(context.Background(), client, archive) + }, + } + cmd.Flags().StringVar(&f.portal, "portal", "", "portal whose catalog to publish to: an environment alias or a hostname") + cmd.Flags().StringVar(&f.origin, "origin", "", "projects/ to publish a project's package") + cmd.Flags().BoolVar(&f.official, "official", false, "mark the package official (publisher role only)") + cmd.Flags().BoolVar(&f.asJSON, "json", false, "emit the catalog's answer as JSON") + return cmd +} diff --git a/cmd/package_test.go b/cmd/package_test.go new file mode 100644 index 0000000..ed15a94 --- /dev/null +++ b/cmd/package_test.go @@ -0,0 +1,475 @@ +package cmd + +import ( + "archive/zip" + "bytes" + "context" + "encoding/json" + "errors" + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync" + "testing" + + "github.com/concord-consortium/cc-data-cli/internal/api" + "github.com/concord-consortium/cc-data-cli/internal/output" + "github.com/concord-consortium/cc-data-cli/internal/packages" +) + +const testManifest = `{"name": "wildfire-responses", "title": "Wildfire responses", "version": "0.1.0", + "urls": {"all": [], "any": [], "none": []}, "clue_prepull": false, + "entrypoint": "run.py", "expected_duration_seconds": 300}` + +type seenRequest struct { + method, path, contentType, auth string + query map[string][]string + contentLength int64 + body []byte +} + +// packageServer answers the routes it is given and 404s every other path, as a report-server +// without the validate and applies routes does. +type packageServer struct { + *httptest.Server + mu sync.Mutex + seen []seenRequest +} + +func newPackageServer(t *testing.T, routes map[string]func(http.ResponseWriter, []byte)) *packageServer { + t.Helper() + s := &packageServer{} + s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + s.mu.Lock() + s.seen = append(s.seen, seenRequest{r.Method, r.URL.Path, r.Header.Get("Content-Type"), r.Header.Get("Authorization"), r.URL.Query(), r.ContentLength, body}) + s.mu.Unlock() + if handle, ok := routes[r.URL.Path]; ok { + handle(w, body) + return + } + w.WriteHeader(http.StatusNotFound) + w.Write([]byte(`{"error":"NOT_FOUND","message":"Not found."}`)) + })) + t.Cleanup(s.Close) + return s +} + +func (s *packageServer) requests(path string) []seenRequest { + s.mu.Lock() + defer s.mu.Unlock() + var out []seenRequest + for _, r := range s.seen { + if r.path == path { + out = append(out, r) + } + } + return out +} + +func answer(status int, body string) func(http.ResponseWriter, []byte) { + return func(w http.ResponseWriter, _ []byte) { + w.WriteHeader(status) + w.Write([]byte(body)) + } +} + +const validated = `{"identity": "users/7/p", "version": "0.1.0", "checksum": "sha256:x", "visibility": "private", "already_published": false, "publishing_unavailable": null}` + +func validateAnswering(body string) map[string]func(http.ResponseWriter, []byte) { + return map[string]func(http.ResponseWriter, []byte){"/api/v1/packages/validate": answer(http.StatusOK, body)} +} + +func writePackage(t *testing.T) string { + t.Helper() + dir := t.TempDir() + for name, content := range map[string]string{"manifest.json": testManifest, "run.py": "print(1)\n"} { + if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + return dir +} + +func capture(t *testing.T) (*bytes.Buffer, *bytes.Buffer) { + t.Helper() + var out, errb bytes.Buffer + restore := output.SetStreams(&out, &errb) + t.Cleanup(restore) + return &out, &errb +} + +func defaultZip(dir string) string { + return filepath.Join(dir, packages.BuildDirName, "wildfire-responses-0.1.0.zip") +} + +func asCLIError(t *testing.T, err error) *output.CLIError { + t.Helper() + var cliErr *output.CLIError + if !errors.As(err, &cliErr) { + t.Fatalf("err = %v (%T), want a *output.CLIError", err, err) + } + return cliErr +} + +func TestPackageInitRefusesASecondInit(t *testing.T) { + dir := filepath.Join(t.TempDir(), "my-package") + written, err := packages.Init(dir, "") + if err != nil || strings.Join(written, ",") != "manifest.json,run.py" { + t.Fatalf("first init: %v, %v", written, err) + } + manifest, err := os.ReadFile(filepath.Join(dir, "manifest.json")) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(manifest), `"name": "my-package"`) { + t.Errorf("manifest = %s", manifest) + } + if err := os.WriteFile(filepath.Join(dir, "run.py"), []byte("mine"), 0o644); err != nil { + t.Fatal(err) + } + if _, err := packages.Init(dir, ""); err == nil { + t.Error("a second init succeeded") + } + if got, _ := os.ReadFile(filepath.Join(dir, "run.py")); string(got) != "mine" { + t.Errorf("the second init overwrote run.py: %q", got) + } + if _, err := packages.Init(filepath.Join(t.TempDir(), "Not_A_Name"), ""); err == nil { + t.Error("init accepted a directory name outside the grammar without --name") + } +} + +func TestPackageBuildValidatesThenWrites(t *testing.T) { + dir := writePackage(t) + srv := newPackageServer(t, validateAnswering(validated)) + out, _ := capture(t) + if err := (packageBuildFlags{}).run(context.Background(), api.New(srv.URL, "tok"), dir); err != nil { + t.Fatal(err) + } + reqs := srv.requests("/api/v1/packages/validate") + if len(reqs) != 1 { + t.Fatalf("%d validate requests, want 1", len(reqs)) + } + written, err := os.ReadFile(defaultZip(dir)) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(reqs[0].body, written) || reqs[0].contentType != "application/zip" { + t.Errorf("validate got %d bytes as %q; the zip written is %d bytes", len(reqs[0].body), reqs[0].contentType, len(written)) + } + r, err := zip.NewReader(bytes.NewReader(written), int64(len(written))) + if err != nil || len(r.File) != 2 { + t.Fatalf("zip: %v, %v", r, err) + } + var line map[string]any + if err := json.Unmarshal(out.Bytes(), &line); err != nil { + t.Fatal(err) + } + if line["checksum"] != packages.Checksum(written) { + t.Errorf("printed checksum %v, archive %s", line["checksum"], packages.Checksum(written)) + } + + t.Chdir(dir) + out.Reset() + if err := (packageBuildFlags{}).run(context.Background(), api.New(srv.URL, "tok"), "."); err != nil { + t.Fatal(err) + } + var again map[string]any + if err := json.Unmarshal(out.Bytes(), &again); err != nil { + t.Fatal(err) + } + if again["checksum"] != line["checksum"] { + t.Errorf("a second build from inside the package gave %v, want %v", again["checksum"], line["checksum"]) + } +} + +func TestPackageBuildWritesNothingTheServerRefuses(t *testing.T) { + dir := writePackage(t) + srv := newPackageServer(t, map[string]func(http.ResponseWriter, []byte){ + "/api/v1/packages/validate": answer(http.StatusUnprocessableEntity, `{"error":"UNPROCESSABLE","message":"version must be MAJOR.MINOR.PATCH"}`), + }) + capture(t) + err := (packageBuildFlags{}).run(context.Background(), api.New(srv.URL, "tok"), dir) + if cliErr := asCLIError(t, err); cliErr.Code != "UNPROCESSABLE" || cliErr.ExitCode != output.ExitContract { + t.Errorf("err = %+v", cliErr) + } + if _, err := os.Stat(defaultZip(dir)); err == nil { + t.Error("a zip was written for a refused archive") + } +} + +// buildWarnings builds against a validate route answering body and returns stderr. +func buildWarnings(t *testing.T, body string) string { + t.Helper() + dir := writePackage(t) + srv := newPackageServer(t, validateAnswering(body)) + _, errb := capture(t) + if err := (packageBuildFlags{}).run(context.Background(), api.New(srv.URL, "tok"), dir); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(defaultZip(dir)); err != nil { + t.Errorf("no zip written: %v", err) + } + return errb.String() +} + +func TestPackageBuildWarnsOnAPublishedVersionAndStillWrites(t *testing.T) { + got := buildWarnings(t, strings.Replace(validated, `"already_published": false`, `"already_published": true`, 1)) + if !strings.Contains(got, "users/7/p 0.1.0 is already published; publish will refuse it until the version changes") { + t.Errorf("stderr = %q", got) + } +} + +func TestPackageBuildIsQuietOnANewVersion(t *testing.T) { + if got := buildWarnings(t, validated); strings.Contains(got, "already published") { + t.Errorf("stderr = %q", got) + } +} + +func TestPackageBuildWarnsWhenThePortalCannotPublishAndStillWrites(t *testing.T) { + got := buildWarnings(t, strings.Replace(validated, `"publishing_unavailable": null`, `"publishing_unavailable": "publishing is not configured for x"`, 1)) + if !strings.Contains(got, "this package cannot be published yet: publishing is not configured for x") { + t.Errorf("stderr = %q", got) + } +} + +func TestPackageBuildIsQuietWhenPublishingIsAvailable(t *testing.T) { + if got := buildWarnings(t, validated); strings.Contains(got, "cannot be published yet") { + t.Errorf("stderr = %q", got) + } +} + +func TestPackageBuildValidatesAsThePublishWillBe(t *testing.T) { + dir := writePackage(t) + srv := newPackageServer(t, validateAnswering(validated)) + capture(t) + client := api.New(srv.URL, "tok") + if err := (packageBuildFlags{origin: "projects/20", official: true}).run(context.Background(), client, dir); err != nil { + t.Fatal(err) + } + if err := (packageBuildFlags{}).run(context.Background(), client, dir); err != nil { + t.Fatal(err) + } + reqs := srv.requests("/api/v1/packages/validate") + if len(reqs) != 2 { + t.Fatalf("%d validate requests, want 2", len(reqs)) + } + if q := reqs[0].query; q["origin"][0] != "projects/20" || q["official"][0] != "true" { + t.Errorf("flagged build's query = %v", q) + } + if q := reqs[1].query; len(q) != 0 { + t.Errorf("plain build's query = %v, want none", q) + } +} + +func TestValidateReturnsAPublishedVersionAsAnAnswerNotAnError(t *testing.T) { + srv := newPackageServer(t, validateAnswering(strings.Replace(validated, `"already_published": false`, `"already_published": true`, 1))) + capture(t) + ans, err := validate(context.Background(), api.New(srv.URL, "tok"), []byte("zip"), "", false) + if err != nil || ans == nil || !ans.AlreadyPublished { + t.Errorf("validate = %+v, %v", ans, err) + } +} + +func TestValidateReturnsAnUnavailablePortalAsAnAnswerNotAnError(t *testing.T) { + srv := newPackageServer(t, validateAnswering(strings.Replace(validated, `"publishing_unavailable": null`, `"publishing_unavailable": "publishing is not configured for x"`, 1))) + capture(t) + ans, err := validate(context.Background(), api.New(srv.URL, "tok"), []byte("zip"), "", false) + if err != nil || ans == nil || ans.PublishingUnavailable == nil || *ans.PublishingUnavailable != "publishing is not configured for x" { + t.Errorf("validate = %+v, %v", ans, err) + } +} + +func TestPackageBuildOnAServerWithoutTheRouteStillBuilds(t *testing.T) { + dir := writePackage(t) + srv := newPackageServer(t, nil) + _, errb := capture(t) + if err := (packageBuildFlags{}).run(context.Background(), api.New(srv.URL, "tok"), dir); err != nil { + t.Fatal(err) + } + if !strings.Contains(errb.String(), "no validate route yet") { + t.Errorf("stderr = %q", errb.String()) + } + if _, err := os.Stat(defaultZip(dir)); err != nil { + t.Errorf("no zip written: %v", err) + } +} + +func TestPackageBuildRefusesAnOutputItWouldCollect(t *testing.T) { + dir := writePackage(t) + srv := newPackageServer(t, validateAnswering(validated)) + capture(t) + err := (packageBuildFlags{out: filepath.Join(dir, "p.zip")}).run(context.Background(), api.New(srv.URL, "tok"), dir) + if cliErr := asCLIError(t, err); cliErr.ExitCode != output.ExitUsage { + t.Errorf("err = %+v", cliErr) + } + if err := (packageBuildFlags{out: filepath.Join(dir, ".builds", "p.zip")}).run(context.Background(), api.New(srv.URL, "tok"), dir); err != nil { + t.Errorf("an --out under a dot directory was refused: %v", err) + } +} + +func TestPackageBuildRefusesANameThatIsAPath(t *testing.T) { + root := t.TempDir() + dir := filepath.Join(root, "a", "pkg") + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + manifest := strings.Replace(testManifest, `"name": "wildfire-responses"`, `"name": "../../escaped"`, 1) + for name, content := range map[string]string{"manifest.json": manifest, "run.py": "print(1)\n"} { + if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + srv := newPackageServer(t, nil) + capture(t) + err := (packageBuildFlags{}).run(context.Background(), api.New(srv.URL, "tok"), dir) + if cliErr := asCLIError(t, err); cliErr.Code != "INVALID_MANIFEST" { + t.Errorf("err = %+v", cliErr) + } + var zips []string + filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error { + if err == nil && strings.HasSuffix(p, ".zip") { + zips = append(zips, p) + } + return nil + }) + if len(zips) != 0 { + t.Errorf("zips written: %v", zips) + } +} + +func testScope(t *testing.T) packages.LocalScope { + t.Helper() + scope, err := packages.ParseLocalScope([]byte(`{"kind": "class", "id": "h", + "classes": [{"class_hash": "0123456789abcdef0123456789abcdef0123456789abcdef", "class_id": 6}], + "assignments": [{"offering_id": 9, "runnable_id": 1234, "name": "Wildfire", "url": "https://activity-player.concord.org/?sequence=830"}]}`)) + if err != nil { + t.Fatal(err) + } + return scope +} + +func TestPackageRunAsksTheServersMatcher(t *testing.T) { + srv := newPackageServer(t, map[string]func(http.ResponseWriter, []byte){ + "/api/v1/packages/applies": answer(http.StatusOK, `{"applies": false, "reason": "no URL in this class matches the required pattern *//wildfire.concord.org/*", "interactive_urls": [], "unread": [], "truncated": false}`), + }) + capture(t) + verdict, err := appliesFor(api.New(srv.URL, "tok"), testScope(t))(context.Background(), packages.URLs{All: []string{"*//wildfire.concord.org/*"}}) + if err != nil { + t.Fatal(err) + } + if verdict.Applies || verdict.Unconfirmed || !strings.Contains(verdict.Reason, "required pattern") { + t.Errorf("verdict = %+v", verdict) + } + var sent struct { + URLs map[string][]string `json:"urls"` + AssignmentURLs []string `json:"assignment_urls"` + } + if err := json.Unmarshal(srv.requests("/api/v1/packages/applies")[0].body, &sent); err != nil { + t.Fatal(err) + } + if strings.Join(sent.URLs["all"], ",") != "*//wildfire.concord.org/*" || strings.Join(sent.AssignmentURLs, ",") != "https://activity-player.concord.org/?sequence=830" { + t.Errorf("sent %+v", sent) + } +} + +func TestPackageRunOnAServerWithoutTheRouteIsUnconfirmed(t *testing.T) { + srv := newPackageServer(t, nil) + capture(t) + verdict, err := appliesFor(api.New(srv.URL, "tok"), testScope(t))(context.Background(), packages.URLs{Any: []string{"*x*"}}) + if err != nil || !verdict.Unconfirmed { + t.Errorf("verdict = %+v, %v", verdict, err) + } +} + +func TestPackageRunAlwaysSendsURLs(t *testing.T) { + srv := newPackageServer(t, map[string]func(http.ResponseWriter, []byte){ + "/api/v1/packages/applies": answer(http.StatusOK, `{"applies": true, "reason": null}`), + }) + capture(t) + if _, err := appliesFor(api.New(srv.URL, "tok"), testScope(t))(context.Background(), packages.URLs{None: []string{"*x*"}}); err != nil { + t.Fatal(err) + } + var sent map[string]json.RawMessage + if err := json.Unmarshal(srv.requests("/api/v1/packages/applies")[0].body, &sent); err != nil { + t.Fatal(err) + } + if urls, ok := sent["urls"]; !ok || string(urls) == "null" { + t.Errorf("applies body = %s", srv.requests("/api/v1/packages/applies")[0].body) + } +} + +func TestPackageRunKeepsTheServersCode(t *testing.T) { + for _, tc := range []struct { + apiErr *api.APIError + exit int + }{ + {&api.APIError{Status: 503, Code: "SERVICE_UNAVAILABLE", Message: "deriver down"}, output.ExitContract}, + {&api.APIError{Status: 401, Code: "NOT_AUTHENTICATED"}, output.ExitNotAuth}, + } { + got := asCLIError(t, packageRunError(api.AsCLIError(tc.apiErr))) + if got.Code != tc.apiErr.Code || got.ExitCode != tc.exit { + t.Errorf("%s: got %s exit %d", tc.apiErr.Code, got.Code, got.ExitCode) + } + } +} + +func publishAnswering(checksum string) map[string]func(http.ResponseWriter, []byte) { + return map[string]func(http.ResponseWriter, []byte){"/api/v1/packages": func(w http.ResponseWriter, body []byte) { + if checksum == "" { + checksum = packages.Checksum(body) + } + w.WriteHeader(http.StatusCreated) + json.NewEncoder(w).Encode(map[string]any{"catalog_id": 3, "identity": "projects/20/p", "version": "0.1.0", + "checksum": checksum, "visibility": "private", "official": false, "current_version": "0.1.0"}) + }} +} + +func TestPackagePublishSendsTheZipAsTheBody(t *testing.T) { + srv := newPackageServer(t, publishAnswering("")) + out, _ := capture(t) + archive := []byte("PK zip bytes") + if err := (packagePublishFlags{origin: "projects/20"}).run(context.Background(), api.New(srv.URL, "tok"), archive); err != nil { + t.Fatal(err) + } + reqs := srv.requests("/api/v1/packages") + if len(reqs) != 1 { + t.Fatalf("%d publish requests, want 1", len(reqs)) + } + r := reqs[0] + if r.method != http.MethodPost || r.contentType != "application/zip" || r.contentLength != int64(len(archive)) || !bytes.Equal(r.body, archive) || r.auth != "Bearer tok" { + t.Errorf("request = %s %q length %d auth %q", r.method, r.contentType, r.contentLength, r.auth) + } + if r.query["origin"][0] != "projects/20" || r.query["official"] != nil { + t.Errorf("query = %v", r.query) + } + if !strings.Contains(out.String(), "identity projects/20/p") { + t.Errorf("stdout = %q", out.String()) + } +} + +func TestPackagePublishRefusesAChecksumItDidNotSend(t *testing.T) { + srv := newPackageServer(t, publishAnswering("sha256:other")) + capture(t) + err := (packagePublishFlags{}).run(context.Background(), api.New(srv.URL, "tok"), []byte("zip")) + if err == nil || !strings.Contains(err.Error(), "sha256:other") { + t.Errorf("err = %v", err) + } +} + +func TestPackagePublishPassesTheServersCodeThrough(t *testing.T) { + srv := newPackageServer(t, map[string]func(http.ResponseWriter, []byte){ + "/api/v1/packages": answer(http.StatusConflict, `{"error":"ALREADY_EXISTS","message":"version 0.1.0 exists"}`), + }) + capture(t) + err := (packagePublishFlags{}).run(context.Background(), api.New(srv.URL, "tok"), []byte("zip")) + if cliErr := asCLIError(t, err); cliErr.Code != "ALREADY_EXISTS" || cliErr.ExitCode != output.ExitContract { + t.Errorf("err = %+v", cliErr) + } + if n := len(srv.requests("/api/v1/packages")); n != 1 { + t.Errorf("%d publish requests, want 1", n) + } +} diff --git a/cmd/root.go b/cmd/root.go index 077176d..9353c1a 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -49,6 +49,7 @@ func newRootCmd() *cobra.Command { root.AddCommand(newReportsCmd()) root.AddCommand(newGetCmd()) root.AddCommand(newQueryCmd()) + root.AddCommand(newPackageCmd()) root.AddCommand(newReplCmd()) root.AddCommand(newInitCmd()) root.AddCommand(newUninstallCmd()) diff --git a/internal/packages/build.go b/internal/packages/build.go new file mode 100644 index 0000000..3629a9c --- /dev/null +++ b/internal/packages/build.go @@ -0,0 +1,57 @@ +package packages + +import ( + "fmt" + "os" + "path/filepath" + "strings" +) + +// BuildDirName is where build writes by default: inside the package, and skipped by +// Collect like every dot path, so one build's zip never reaches the next. +const BuildDirName = ".cc-data-build" + +// CheckOutsidePackage refuses an explicit build output that a later build would collect. +func CheckOutsidePackage(dir, out string) error { + absDir, err := filepath.Abs(dir) + if err != nil { + return err + } + absOut, err := filepath.Abs(out) + if err != nil { + return err + } + rel, err := filepath.Rel(absDir, absOut) + if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { + return nil + } + if excluded(filepath.ToSlash(rel), false) || excludedAncestor(filepath.ToSlash(rel)) { + return nil + } + return fmt.Errorf("--out %s is inside the package, where the next build would include it", out) +} + +func excludedAncestor(rel string) bool { + parts := strings.Split(rel, "/") + for i := 1; i < len(parts); i++ { + if excluded(strings.Join(parts[:i], "/"), true) { + return true + } + } + return false +} + +// WriteBuild writes the archive, creating its directory, and ignores the default build +// directory for git. +func WriteBuild(out string, archive []byte) error { + dir := filepath.Dir(out) + if err := os.MkdirAll(dir, 0o755); err != nil { + return err + } + if filepath.Base(dir) == BuildDirName { + if err := ignoreAll(dir); err != nil { + return err + } + } + return os.WriteFile(out, archive, 0o644) +} diff --git a/internal/packages/init.go b/internal/packages/init.go new file mode 100644 index 0000000..06dfea8 --- /dev/null +++ b/internal/packages/init.go @@ -0,0 +1,55 @@ +package packages + +import ( + _ "embed" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" +) + +//go:embed template/run.py +var runStub []byte + +// Init writes manifest.json and run.py into dir, creating it if needed. It reads nothing in +// dir except to refuse when either file already exists, in which case it writes neither. +func Init(dir, name string) ([]string, error) { + if name == "" { + abs, err := filepath.Abs(dir) + if err != nil { + return nil, err + } + name = filepath.Base(abs) + if !ValidName(name) { + return nil, fmt.Errorf("the directory name %q is not a package name; pass --name (^[a-z0-9][a-z0-9-]{0,62}$)", name) + } + } else if !ValidName(name) { + return nil, fmt.Errorf("--name %q must match ^[a-z0-9][a-z0-9-]{0,62}$", name) + } + for _, f := range []string{"manifest.json", "run.py"} { + if _, err := os.Lstat(filepath.Join(dir, f)); !errors.Is(err, os.ErrNotExist) { + return nil, fmt.Errorf("%s already exists in %s; init writes only into a package that has neither file", f, dir) + } + } + manifest := Manifest{ + Name: name, Title: name, Version: "0.1.0", + Description: "One line saying what this package shows.", + URLs: URLs{All: []string{}, Any: []string{}, None: []string{}}, + Entrypoint: "run.py", ExpectedDurationSeconds: 300, + } + b, err := json.MarshalIndent(manifest, "", " ") + if err != nil { + return nil, err + } + if err := os.MkdirAll(dir, 0o755); err != nil { + return nil, err + } + if err := os.WriteFile(filepath.Join(dir, "manifest.json"), append(b, '\n'), 0o644); err != nil { + return nil, err + } + if err := os.WriteFile(filepath.Join(dir, "run.py"), runStub, 0o755); err != nil { + return nil, err + } + return []string{"manifest.json", "run.py"}, nil +} diff --git a/internal/packages/run.go b/internal/packages/run.go index 424251e..f079d64 100644 --- a/internal/packages/run.go +++ b/internal/packages/run.go @@ -163,7 +163,7 @@ func prepare(run string, files Files) (runPaths, error) { if err := ensureRealDir(run); err != nil { return p, err } - if err := os.WriteFile(filepath.Join(run, ".gitignore"), []byte("*\n"), 0o644); err != nil { + if err := ignoreAll(run); err != nil { return p, err } for _, d := range []string{p.pkg, p.in, p.out} { @@ -195,6 +195,12 @@ func prepare(run string, files Files) (runPaths, error) { return p, nil } +// ignoreAll keeps git out of a directory this tool makes inside a package, whose files can +// hold student data. +func ignoreAll(dir string) error { + return os.WriteFile(filepath.Join(dir, ".gitignore"), []byte("*\n"), 0o644) +} + func ensureRealDir(d string) error { if _, err := os.Lstat(d); errors.Is(err, os.ErrNotExist) { return os.Mkdir(d, 0o755) diff --git a/internal/packages/template/run.py b/internal/packages/template/run.py new file mode 100644 index 0000000..f0ce06f --- /dev/null +++ b/internal/packages/template/run.py @@ -0,0 +1,87 @@ +#!/usr/bin/env python3 +"""A Researcher Dashboard package: counts a class's answers and the learners who wrote them. + +It runs the same way under `cc-data package run` on a laptop and on the dashboard's VM. It +reads the scope from RD_SCOPE_FILE, pulls its own data into RD_DATASET with cc-data, counts +through cc-data's views, and writes display.md, summary.txt and counts.json to the scope's +output_dir. Stdlib only: the VM installs no Python packages. + +It reads the class through a Student ID Mapping run, which the portal computes on every +download, so re-reading it picks up learners who joined since, and which has no limit on how +many assignments a class has. It reuses the researcher's own run for the class when there is +one and otherwise creates one, on the server the dataset's portal names. +""" +import json +import os +import subprocess +import sys + + +def cc_data(*args): + result = subprocess.run(["cc-data", *args], capture_output=True, text=True) + if result.returncode != 0: + detail = " ".join(p for p in (result.stderr.strip(), result.stdout.strip()) if p) + raise RuntimeError(f"cc-data {' '.join(args[:2])} exited {result.returncode}: {detail or '(no output)'}") + return result.stdout + + +def query(dataset, sql): + return json.loads(cc_data("query", "--dataset", dataset, sql, "--format", "json")) + + +def mapping_run(portal, class_id): + """The newest Student ID Mapping run filtered to exactly this class, else a new one.""" + runs = json.loads(cc_data("reports", "list", "--portal", portal, "--json"))["runs"] + mine = [r["run_id"] for r in runs + if r.get("slug") == "student-id-mapping" + and (r.get("report_filter") or {}).get("filters") == ["class"] + and (r.get("report_filter") or {}).get("class") == [class_id]] + if mine: + return max(int(run) for run in mine) + created = json.loads(cc_data( + "reports", "create", "--portal", portal, "--report-slug", "student-id-mapping", + "--report-filter", json.dumps({"class": [class_id]}), "--json")) + return int(created["run"]["run_id"]) + + +def main(): + with open(os.environ["RD_SCOPE_FILE"], encoding="utf-8") as handle: + scope = json.load(handle) + # RD_DATASET is the full / ref; scope["dataset"] is the bare name. + dataset = os.environ["RD_DATASET"] + class_id = int(scope["classes"][0]["class_id"]) + + try: + cc_data("dataset", "create", dataset) + except RuntimeError as err: + if "exists" not in str(err).lower(): + raise + run = mapping_run(os.environ["CC_DATA_PORTAL"], class_id) + cc_data("get", "report", str(run), "--dataset", dataset, "--refresh") + cc_data("get", "answers", str(run), "--dataset", dataset) + + rows = query(dataset, f"SELECT count(*) AS n FROM student_id_mapping WHERE run_id = {run}")[0]["n"] + if rows == 0: + raise RuntimeError("no report-service access to this class, or the class has no data") + answers = query(dataset, f"SELECT count(*) AS n FROM run_answers WHERE run_id = {run}")[0]["n"] + # By user_id, never by endpoint: a student has one endpoint per assignment. + learners = query(dataset, f""" + SELECT count(DISTINCT m.user_id) AS n FROM student_id_mapping m + JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint + WHERE a.run_id = {run} AND m.run_id = {run}""")[0]["n"] + + out = scope["output_dir"] + with open(os.path.join(out, "display.md"), "w", encoding="utf-8") as handle: + handle.write(f"# Class {class_id}\n\n- {answers} answers\n- {learners} learners with at least one answer\n") + with open(os.path.join(out, "summary.txt"), "w", encoding="utf-8") as handle: + handle.write(f"{answers} answers from {learners} learners\n") + with open(os.path.join(out, "counts.json"), "w", encoding="utf-8") as handle: + json.dump({"answers": answers}, handle) + + +if __name__ == "__main__": + try: + main() + except Exception as err: + print(str(err), file=sys.stderr) + sys.exit(1) diff --git a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md index 1cafed6..66e43df 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md +++ b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md @@ -841,7 +841,7 @@ func prepare(run string, files Files) (runPaths, error) { if err := ensureRealDir(run); err != nil { return p, err } - if err := os.WriteFile(filepath.Join(run, ".gitignore"), []byte("*\n"), 0o644); err != nil { + if err := ignoreAll(run); err != nil { return p, err } for _, d := range []string{p.pkg, p.in, p.out} { @@ -873,6 +873,12 @@ func prepare(run string, files Files) (runPaths, error) { return p, nil } +// ignoreAll keeps git out of a directory this tool makes inside a package, whose files can +// hold student data. +func ignoreAll(dir string) error { + return os.WriteFile(filepath.Join(dir, ".gitignore"), []byte("*\n"), 0o644) +} + func ensureRealDir(d string) error { if _, err := os.Lstat(d); errors.Is(err, os.ErrNotExist) { return os.Mkdir(d, 0o755) @@ -1199,6 +1205,7 @@ func RouteMissing(err error) bool { - `cmd/package.go`: new. - `cmd/root.go`: `root.AddCommand(newPackageCmd())` after `newQueryCmd()`. - `internal/packages/init.go`: new. +- `internal/packages/build.go`: new; the default build directory, the `--out` check, writing the zip, and the `.gitignore` both working directories carry. - `internal/packages/template/run.py`: new, embedded. - `cmd/package_test.go`: new. @@ -1231,8 +1238,8 @@ executes the package against your own dataset under the runner's rules; build zi publish registers the zip in the catalog with your cc-data token. What run cannot reproduce from the VM: the package runs as you, with no network sandbox, -and with HOME, PATH and the locale passed through so its own cc-data calls find your -login. A .py entrypoint runs with python3.11 when it is on PATH, as on the VM, else +and with HOME, PATH, the locale and any proxy settings passed through so its own cc-data +calls find your login and reach the server. A .py entrypoint runs with python3.11 when it is on PATH, as on the VM, else python3.` func newPackageCmd() *cobra.Command { @@ -1331,8 +1338,7 @@ func newPackageRunCmd() *cobra.Command { if err != nil { return err } - // The package runs in its own process group, out of reach of the terminal's Ctrl-C, - // so an interrupt has to cancel the run for the group to be killed with it. + // The terminal's Ctrl-C cannot reach the package's own process group, so it cancels the run. ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) defer stop() archive, err := packages.Zip(files) @@ -1499,8 +1505,8 @@ type packagePublishFlags struct { official, asJSON bool } -// run publishes an archive already read and checked; it takes the client so it can be -// exercised against a test server without a stored credential. +// run publishes the archive; it takes the client so it can be exercised against a test server +// without a stored credential. func (f packagePublishFlags) run(ctx context.Context, client *api.Client, archive []byte) error { got, err := client.PublishPackage(ctx, archive, f.origin, f.official) if err != nil { @@ -1555,13 +1561,8 @@ import ( "fmt" "os" "path/filepath" - "strings" ) -// BuildDirName is where build writes by default: inside the package, and skipped by -// Collect like every dot path, so one build's zip never reaches the next. -const BuildDirName = ".cc-data-build" - //go:embed template/run.py var runStub []byte @@ -1606,6 +1607,20 @@ func Init(dir, name string) ([]string, error) { } return []string{"manifest.json", "run.py"}, nil } +``` +```go +package packages + +import ( + "fmt" + "os" + "path/filepath" + "strings" +) + +// BuildDirName is where build writes by default: inside the package, and skipped by +// Collect like every dot path, so one build's zip never reaches the next. +const BuildDirName = ".cc-data-build" // CheckOutsidePackage refuses an explicit build output that a later build would collect. func CheckOutsidePackage(dir, out string) error { @@ -1637,15 +1652,15 @@ func excludedAncestor(rel string) bool { return false } -// WriteBuild writes the archive, creating its directory, with the .gitignore every working -// directory this tool makes inside a package carries. +// WriteBuild writes the archive, creating its directory, and ignores the default build +// directory for git. func WriteBuild(out string, archive []byte) error { dir := filepath.Dir(out) if err := os.MkdirAll(dir, 0o755); err != nil { return err } if filepath.Base(dir) == BuildDirName { - if err := os.WriteFile(filepath.Join(dir, ".gitignore"), []byte("*\n"), 0o644); err != nil { + if err := ignoreAll(dir); err != nil { return err } } From 538c18dd459eec46d97fa0bb7daab5e01ee1e3bf Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 16:05:53 -0400 Subject: [PATCH 08/17] docs: document developing and publishing a package [REPORT-146] The researcher guide gains "Developing and publishing a package": the init, run, build and publish loop, the four-key scope file and where to keep it, what package run reproduces of the runner and what it cannot, the checks report-server makes and the warning an older one gives, the "publishing is not configured" answer, and the student data the two working directories can hold. Later sections renumber from 8. The README's command sketch and the Claude skill gain the four commands, and package run's help now states the local/VM differences that only package's help stated. --- README.md | 4 + cmd/package.go | 12 ++- docs/researcher-guide.md | 97 ++++++++++++++++++- internal/guidance/src/skill_header.md | 13 +++ .../implementation.md | 14 +-- 5 files changed, 126 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 8aac7ca..6d2e9e5 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,10 @@ cc-data get history --dataset / cc-data get attachments --dataset / [--answer ] [--url] [--inline] cc-data query --dataset / "SELECT ..." [--format table|csv|json|jsonl] cc-data repl --dataset / +cc-data package init [dir] [--name ] # manifest.json skeleton + run.py stub for a dashboard package +cc-data package run [dir] --dataset / --scope # run it locally under the runner's rules +cc-data package build [dir] [--out ] # reproducible zip, checked against the catalog's rules +cc-data package publish [--portal ] [--origin projects/] [--official] cc-data mcp # MCP stdio server for Claude Desktop ``` diff --git a/cmd/package.go b/cmd/package.go index 86f5606..aa0e605 100644 --- a/cmd/package.go +++ b/cmd/package.go @@ -17,16 +17,18 @@ import ( "github.com/spf13/cobra" ) +const runDifferences = `What run cannot reproduce from the VM: the package runs as you, with no network +sandbox, and with HOME, PATH, the locale and any proxy settings passed through so its own +cc-data calls find your login and reach the server. A .py entrypoint runs with python3.11 +when it is on PATH, as on the VM, else python3.` + const packageLong = `Develop, test and publish a Researcher Dashboard package. A package is a directory holding manifest.json and an entrypoint. init writes both; run executes the package against your own dataset under the runner's rules; build zips it; publish registers the zip in the catalog with your cc-data token. -What run cannot reproduce from the VM: the package runs as you, with no network sandbox, -and with HOME, PATH, the locale and any proxy settings passed through so its own cc-data -calls find your login and reach the server. A .py entrypoint runs with python3.11 when it is on PATH, as on the VM, else -python3.` +` + runDifferences func newPackageCmd() *cobra.Command { cmd := &cobra.Command{ @@ -93,7 +95,7 @@ func newPackageRunCmd() *cobra.Command { Long: "Run a package against your own dataset under the runner's rules: the same scope.json,\n" + "environment variables, output files, display.md cap and applicability check.\n\n" + "--scope names a file holding the scope's kind, id, classes and assignments.\n" + - "Results are written under /.cc-data-run/, which build never ships.", + "Results are written under /.cc-data-run/, which build never ships.\n\n" + runDifferences, Args: cobra.MaximumNArgs(1), RunE: func(cmd *cobra.Command, args []string) error { if datasetRef == "" || scopePath == "" { diff --git a/docs/researcher-guide.md b/docs/researcher-guide.md index 77fafcb..9521844 100644 --- a/docs/researcher-guide.md +++ b/docs/researcher-guide.md @@ -585,7 +585,98 @@ write the cross-dataset query for you. --- -## 7. A few things to keep in mind +## 7. Developing and publishing a package + +A Researcher Dashboard **package** is a small program the dashboard runs for a +class on a short-lived virtual machine (the "runner"), which pulls the class's +data with `cc-data` and writes a short report. `cc-data package` lets you write +one on your own machine, run it the way the runner will, and publish it to the +dashboard's catalog with the cc-data login you already have. + +### The loop + +``` +cc-data package init my-package # writes manifest.json and run.py +# edit run.py and manifest.json +cc-data package run my-package --dataset learn.concord.org/my-package-dev --scope ~/scopes/class-6.json +cc-data package build my-package # writes my-package/.cc-data-build/-.zip +cc-data package publish my-package/.cc-data-build/my-package-0.1.0.zip --portal prod +``` + +The `run.py` that `init` writes is a complete package: it finds or creates a +Student ID Mapping run for the class, pulls its answers, counts answers and +learners through `cc-data query`, and writes `display.md`, `summary.txt` and +`counts.json`. Change the counting and the display, and keep the rest. + +### The scope file + +`package run` needs to know which class to run for. On the VM the dashboard +supplies that; on your machine a scope file does, holding exactly these four keys: + +```json +{ + "kind": "class", + "id": "", + "classes": [{"class_hash": "", "class_id": 6}], + "assignments": [{"offering_id": 9, "runnable_id": 1234, "name": "Moth 1.2", "url": "https://..."}] +} +``` + +Keep it **outside the package directory**, or under a name starting with a dot +such as `.scope.json`, so `build` never ships a class hash in the zip. + +### What `run` reproduces, and what it cannot + +`run` stages exactly the files `build` would ship into +`/.cc-data-run/pkg/` and runs the package from there, writes the runner's +`scope.json`, sets the runner's environment variables (`RD_DATASET`, +`RD_SCOPE_FILE`, `RD_OUTPUT_DIR` and the rest), applies the same time limit, and +reads the result as the runner does, including the 65,536-byte cap on +`display.md`. A package that reads a file `build` leaves out, such as something +under `local-data/`, fails locally as it would on the VM. + +What it cannot reproduce: + +- **No sandbox.** The package runs as you, with your network. The VM runs it as + its own user behind an egress proxy. +- **A few of your variables pass through.** `HOME`, `PATH`, your locale, the + proxy settings and a handful of others reach the package, so its own `cc-data` + calls find your login. Nothing else from your environment does. +- **Python.** A `.py` entrypoint runs with `python3.11` when it is on your + `PATH`, as on the VM, and otherwise with `python3` after a note saying so. +- **CLUE.** A package with `clue_prepull: true` is refused locally, since + `cc-data` cannot fetch CLUE data yet. + +### Checked against report-server + +`build` and `run` send the zip to report-server, which checks it against the +catalog's rules without publishing it, and `run` asks report-server whether the +package's `urls` patterns apply to the scope's assignments. A refusal stops the +command with report-server's own message. A report-server too old to have these +checks answers with a warning, and the command carries on: the package is then +unchecked until you publish it. + +`build` also warns, and still writes the zip, when that version is already +published (bump `version` in `manifest.json`) or when the portal cannot store +packages yet. + +### Publishing + +`publish` sends the zip to the catalog with your cc-data token. A first publish +creates the package private to you; `--origin projects/` publishes it as a +project's, and `--official` needs the publisher role. A portal whose report-server +has no package storage configured answers `UNPROCESSABLE` "publishing is not +configured for " even though the zip itself passed every check. + +### Files the commands leave in your package + +`.cc-data-run/` (the last run's staged copy, inputs, outputs and the package's +data directory) and `.cc-data-build/` (built zips) each carry a `.gitignore` +of `*`, so git never commits them. **Both can hold student data**: `display.md` +is drawn from it. `build` never ships either, nor any other path starting with a +dot. + +## 8. A few things to keep in mind - **One dataset belongs to one portal.** Run IDs are specific to a portal (and to the report server behind it); the same run number on a different portal is a @@ -608,7 +699,7 @@ write the cross-dataset query for you. --- -## 8. Troubleshooting +## 9. Troubleshooting - **`NOT_AUTHENTICATED`, or a prompt to "run `cc-data login`".** Your token is missing or expired. Run `cc-data login` (add `--portal` if you're not on @@ -628,7 +719,7 @@ write the cross-dataset query for you. --- -## 9. Sensitive data and cleanup +## 10. Sensitive data and cleanup Downloading turns a transient export into a durable local corpus that can contain student names and other PII. Keep only what you need, and purge datasets when you diff --git a/internal/guidance/src/skill_header.md b/internal/guidance/src/skill_header.md index 0f974d4..cf7024d 100644 --- a/internal/guidance/src/skill_header.md +++ b/internal/guidance/src/skill_header.md @@ -51,6 +51,19 @@ guessing flags. These take `--portal` when no default portal is configured. +## Packages + +Researcher Dashboard packages are developed and published with `cc-data package`: + +- `cc-data package init [dir]` writes a `manifest.json` skeleton and a `run.py` stub. +- `cc-data package run [dir] --dataset --scope ` runs it locally the + way the dashboard's runner will, with the scope file holding the class's + `kind`, `id`, `classes` and `assignments`. +- `cc-data package build [dir]` zips it reproducibly and checks it against the + catalog's rules. +- `cc-data package publish ` registers it in the catalog with the cc-data + token. + ## Fetching data - `cc-data get report --dataset ` — the report CSV. A Portal report diff --git a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md index 66e43df..ddcecf0 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md +++ b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md @@ -1231,16 +1231,18 @@ import ( "github.com/spf13/cobra" ) +const runDifferences = `What run cannot reproduce from the VM: the package runs as you, with no network +sandbox, and with HOME, PATH, the locale and any proxy settings passed through so its own +cc-data calls find your login and reach the server. A .py entrypoint runs with python3.11 +when it is on PATH, as on the VM, else python3.` + const packageLong = `Develop, test and publish a Researcher Dashboard package. A package is a directory holding manifest.json and an entrypoint. init writes both; run executes the package against your own dataset under the runner's rules; build zips it; publish registers the zip in the catalog with your cc-data token. -What run cannot reproduce from the VM: the package runs as you, with no network sandbox, -and with HOME, PATH, the locale and any proxy settings passed through so its own cc-data -calls find your login and reach the server. A .py entrypoint runs with python3.11 when it is on PATH, as on the VM, else -python3.` +` + runDifferences func newPackageCmd() *cobra.Command { cmd := &cobra.Command{ @@ -1307,7 +1309,7 @@ func newPackageRunCmd() *cobra.Command { Long: "Run a package against your own dataset under the runner's rules: the same scope.json,\n" + "environment variables, output files, display.md cap and applicability check.\n\n" + "--scope names a file holding the scope's kind, id, classes and assignments.\n" + - "Results are written under /.cc-data-run/, which build never ships.", + "Results are written under /.cc-data-run/, which build never ships.\n\n" + runDifferences, Args: cobra.MaximumNArgs(1), RunE: func(cmd *cobra.Command, args []string) error { if datasetRef == "" || scopePath == "" { @@ -1811,7 +1813,7 @@ Tests (against an `httptest` server that answers 404 for any route it was not gi - `.cc-data-run/` and `.cc-data-build/` (gitignored, and holding student data); - the 422 "publishing is not configured" answer a portal gives before its bucket exists. -`skill_header.md` gains a short "Packages" list naming the four commands. The `--help` texts above already state the differences between a local run and a VM. +`skill_header.md` gains a short "Packages" list naming the four commands. The `--help` texts above state the differences between a local run and a VM, on both `package` and `package run`. The guide's later sections renumber from 8. **Files affected**: - `README.md`. From 67e315dd8255d40368cfa0e3862f6e51478cf5a4 Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 16:06:24 -0400 Subject: [PATCH 09/17] ci: hold pre-release tags back from latest and Homebrew [REPORT-146] The runner image and REPORT-147's publish workflow pin v0.3.0-pre.1, cut after this branch merges. release.yml now classifies the tag once: a tag with a "-" after the version still builds and uploads every archive and checksum, but its GitHub release is marked as a pre-release, so it is never shown as latest, and the Homebrew formula step is skipped, so brew upgrade never offers it. The archives upload inside gh release create, before the formula step, so skipping the formula cannot drop them. A release_test.go guard pins both uses of the classification, and the README's release paragraph says what a pre-release tag publishes. --- .github/workflows/release.yml | 15 ++++++++++++++- README.md | 2 +- release_test.go | 15 +++++++++++++++ .../implementation.md | 5 ++--- 4 files changed, 32 insertions(+), 5 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 1f1d01a..eee552e 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -170,6 +170,17 @@ jobs: echo "ok: ${tarball}" done + # A pre-release tag (v0.3.0-pre.1) exists to be pinned: it is never marked latest + # and never reaches Homebrew. + - name: Classify the tag + id: tag + run: | + if [[ "${GITHUB_REF_NAME}" == *-* ]]; then + echo "prerelease=true" >> "${GITHUB_OUTPUT}" + else + echo "prerelease=false" >> "${GITHUB_OUTPUT}" + fi + - name: Create GitHub release env: GH_TOKEN: ${{ github.token }} @@ -177,9 +188,11 @@ jobs: # Attach the checksum manifests alongside the tarballs so consumers can # verify downloads independently. gh release create "${GITHUB_REF_NAME}" dist/*.tar.gz dist/*.sha256 \ - --title "${GITHUB_REF_NAME}" --generate-notes + --title "${GITHUB_REF_NAME}" --generate-notes \ + --prerelease=${{ steps.tag.outputs.prerelease }} - name: Render and push the Homebrew formula + if: steps.tag.outputs.prerelease == 'false' env: TAP_TOKEN: ${{ secrets.HOMEBREW_TAP_TOKEN }} run: scripts/render-formula.sh "${GITHUB_REF_NAME}" diff --git a/README.md b/README.md index 6d2e9e5..722ea0a 100644 --- a/README.md +++ b/README.md @@ -98,7 +98,7 @@ The CLI is written in Go. DuckDB is embedded via [`duckdb-go`](https://github.co go build ./... ``` -Releases are built with goreleaser on native GitHub Actions runners per platform, with a Homebrew formula for install (`brew install concord-consortium/tap/cc-data`). Note that `brew uninstall` removes only the binary; run `cc-data uninstall` first to remove the Claude skill, the `~/.claude/CLAUDE.md` pointer, and (optionally) your stored credentials. Your datasets are never removed automatically — `cc-data uninstall` prints where they remain. +Releases are built with goreleaser on native GitHub Actions runners per platform, with a Homebrew formula for install (`brew install concord-consortium/tap/cc-data`). A tag with a pre-release suffix, such as `v0.3.0-pre.1`, publishes a GitHub pre-release with the same archives and no formula, for pinning by the runner image and package workflows. Note that `brew uninstall` removes only the binary; run `cc-data uninstall` first to remove the Claude skill, the `~/.claude/CLAUDE.md` pointer, and (optionally) your stored credentials. Your datasets are never removed automatically — `cc-data uninstall` prints where they remain. The macOS signing/notarization and Homebrew-tap credentials the release needs live in this repo's GitHub Actions secrets. Maintainers can re-create those secrets from `cc-data-release-secrets.zip` in the **Developer Admin** 1Password vault, which bundles the credentials and a script that sets them. It exists so the release secrets can be restored, rotated, or handed to a new maintainer without redoing the Apple and GitHub setup from scratch (the tap push token in particular is a fine-grained GitHub PAT that must be rotated at least yearly). diff --git a/release_test.go b/release_test.go index 29a117b..8506558 100644 --- a/release_test.go +++ b/release_test.go @@ -121,3 +121,18 @@ func readFile(t *testing.T, path string) string { } return string(data) } + +// A pre-release tag exists to be pinned, so it must not become GitHub's latest release or +// reach researchers through brew upgrade. Both steps read the one classification. +func TestReleaseWorkflowKeepsPreReleasesBack(t *testing.T) { + rel := readFile(t, ".github/workflows/release.yml") + for _, want := range []string{ + `if [[ "${GITHUB_REF_NAME}" == *-* ]]; then`, + "--prerelease=${{ steps.tag.outputs.prerelease }}", + "if: steps.tag.outputs.prerelease == 'false'", + } { + if !strings.Contains(rel, want) { + t.Fatalf("release workflow must hold a pre-release tag back from latest and from Homebrew; missing %q", want) + } + } +} diff --git a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md index ddcecf0..81da434 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md +++ b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md @@ -1855,9 +1855,8 @@ Tests (against an `httptest` server that answers 404 for any route it was not gi and, before "Create GitHub release": ```yaml - # A tag with a semver pre-release suffix (v0.3.0-pre.1) is published for pinning, - # by the runner image and package workflows, and never offered as the current - # release: GitHub does not mark it latest and Homebrew never sees it. + # A pre-release tag (v0.3.0-pre.1) exists to be pinned: it is never marked latest + # and never reaches Homebrew. - name: Classify the tag id: tag run: | From a69586d73af7d41c6fff492d63edd661186e7ebb Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 16:13:20 -0400 Subject: [PATCH 10/17] fix: close the gaps a spec audit found in the package work [REPORT-146] - run_answers.run_id is cast to BIGINT, as the fallback view already was, so its type no longer depends on whether a dataset has membership. - An unreadable scope file is INVALID_SCOPE, like an invalid one. - A .gitignore that is a link is refused rather than written through, and build refuses a .cc-data-build that is a link, so neither working directory can be used to write outside the package. - publish's usage line and the README sketch name --json. New tests cover the BIGINT column, both link refusals, the python3 fallback note, the build .gitignore, the applies warnings for unread and truncated profiles, publish's exit 3 and its retry advice when the server never answered, and the unreadable scope file. The requirements now say that a linked summary.txt refuses the result and a linked counts.json is ignored, as on the runner, and reconcile the deriver's 240 seconds with report-server's 270. --- README.md | 2 +- cmd/package.go | 13 ++- cmd/package_test.go | 51 +++++++++ internal/duck/run_views_test.go | 23 ++++ internal/duck/views.go | 2 +- internal/packages/build.go | 13 ++- internal/packages/proc_unix.go | 4 +- internal/packages/result.go | 4 +- internal/packages/run.go | 13 +-- internal/packages/run_test.go | 57 ++++++++++ internal/packages/scope.go | 5 +- .../implementation.md | 101 ++++++++++++------ .../requirements.md | 12 +-- 13 files changed, 232 insertions(+), 68 deletions(-) diff --git a/README.md b/README.md index 722ea0a..bf445de 100644 --- a/README.md +++ b/README.md @@ -71,7 +71,7 @@ cc-data repl --dataset / cc-data package init [dir] [--name ] # manifest.json skeleton + run.py stub for a dashboard package cc-data package run [dir] --dataset / --scope # run it locally under the runner's rules cc-data package build [dir] [--out ] # reproducible zip, checked against the catalog's rules -cc-data package publish [--portal ] [--origin projects/] [--official] +cc-data package publish [--portal ] [--origin projects/] [--official] [--json] cc-data mcp # MCP stdio server for Claude Desktop ``` diff --git a/cmd/package.go b/cmd/package.go index aa0e605..98c4e1e 100644 --- a/cmd/package.go +++ b/cmd/package.go @@ -108,7 +108,7 @@ func newPackageRunCmd() *cobra.Command { } raw, err := os.ReadFile(scopePath) if err != nil { - return output.Usagef("reading the scope file: %v", err) + return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: fmt.Sprintf("reading the scope file: %v", err)} } scope, err := packages.ParseLocalScope(raw) if err != nil { @@ -158,13 +158,12 @@ func newPackageRunCmd() *cobra.Command { return cmd } -// derivationTimeout covers report-service's deriver, which may spend up to 240 seconds -// fetching a scope's activities; the client's 60-second default would cut it off. +// derivationTimeout covers report-server's wait of up to 270 seconds on report-service's +// deriver; the client's 60-second default would cut it off. const derivationTimeout = 5 * time.Minute -// validate asks report-server whether a publish of the archive would be accepted, and returns -// its answer. A report-server without the route is not a refusal: the archive is unchecked, -// the answer is nil, and saying so is the whole result. +// validate asks report-server whether a publish of the archive would be accepted. A server +// without the route gets a warning and a nil answer: the archive is unchecked, not refused. func validate(ctx context.Context, client *api.Client, archive []byte, origin string, official bool) (*api.ValidatedPackage, error) { ans, err := client.ValidatePackage(ctx, archive, origin, official) switch { @@ -314,7 +313,7 @@ func (f packagePublishFlags) run(ctx context.Context, client *api.Client, archiv func newPackagePublishCmd() *cobra.Command { var f packagePublishFlags cmd := &cobra.Command{ - Use: "publish [--portal ]", + Use: "publish [--portal ] [--origin projects/] [--official] [--json]", Short: "Publish a built package to the catalog with your cc-data token", Long: "POST the zip to report-server's catalog, which creates the package private on its first\n" + "publish and records the version. --origin projects/ publishes a project package;\n" + diff --git a/cmd/package_test.go b/cmd/package_test.go index ed15a94..7b5bc88 100644 --- a/cmd/package_test.go +++ b/cmd/package_test.go @@ -160,6 +160,9 @@ func TestPackageBuildValidatesThenWrites(t *testing.T) { if !bytes.Equal(reqs[0].body, written) || reqs[0].contentType != "application/zip" { t.Errorf("validate got %d bytes as %q; the zip written is %d bytes", len(reqs[0].body), reqs[0].contentType, len(written)) } + if gi, err := os.ReadFile(filepath.Join(dir, packages.BuildDirName, ".gitignore")); err != nil || string(gi) != "*\n" { + t.Errorf(".cc-data-build/.gitignore = %q, %v", gi, err) + } r, err := zip.NewReader(bytes.NewReader(written), int64(len(written))) if err != nil || len(r.File) != 2 { t.Fatalf("zip: %v, %v", r, err) @@ -473,3 +476,51 @@ func TestPackagePublishPassesTheServersCodeThrough(t *testing.T) { t.Errorf("%d publish requests, want 1", n) } } + +func TestPackageRunWarnsWhatTheDeriverCouldNotRead(t *testing.T) { + srv := newPackageServer(t, map[string]func(http.ResponseWriter, []byte){ + "/api/v1/packages/applies": answer(http.StatusOK, `{"applies": true, "reason": null, "unread": [{"url": "https://a/1", "reason": "timeout"}], "truncated": true}`), + }) + _, errb := capture(t) + if _, err := appliesFor(api.New(srv.URL, "tok"), testScope(t))(context.Background(), packages.URLs{Any: []string{"*x*"}}); err != nil { + t.Fatal(err) + } + for _, want := range []string{"could not read https://a/1 (timeout)", "profile was truncated"} { + if !strings.Contains(errb.String(), want) { + t.Errorf("stderr = %q, want %q", errb.String(), want) + } + } +} + +func TestPackageRunRefusesAnUnreadableScopeFile(t *testing.T) { + dir := writePackage(t) + capture(t) + cmd := newPackageRunCmd() + cmd.SetArgs([]string{dir, "--dataset", "learn.concord.org/d", "--scope", filepath.Join(t.TempDir(), "missing.json")}) + cmd.SetOut(io.Discard) + cmd.SetErr(io.Discard) + if cliErr := asCLIError(t, cmd.Execute()); cliErr.Code != "INVALID_SCOPE" || cliErr.ExitCode != output.ExitUsage { + t.Errorf("err = %+v", cliErr) + } +} + +func TestPackagePublishMapsAuthAndUnansweredFailures(t *testing.T) { + srv := newPackageServer(t, map[string]func(http.ResponseWriter, []byte){ + "/api/v1/packages": answer(http.StatusUnauthorized, `{"error":"NOT_AUTHENTICATED","message":"bad token"}`), + }) + capture(t) + err := (packagePublishFlags{}).run(context.Background(), api.New(srv.URL, "tok"), []byte("zip")) + if cliErr := asCLIError(t, err); cliErr.ExitCode != output.ExitNotAuth { + t.Errorf("NOT_AUTHENTICATED: %+v", cliErr) + } + + dropped := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, _, _ := w.(http.Hijacker).Hijack() + conn.Close() + })) + t.Cleanup(dropped.Close) + err = (packagePublishFlags{}).run(context.Background(), api.New(dropped.URL, "tok"), []byte("zip")) + if cliErr := asCLIError(t, err); !strings.Contains(cliErr.Action, "Publishing again is safe") { + t.Errorf("unanswered: %+v", cliErr) + } +} diff --git a/internal/duck/run_views_test.go b/internal/duck/run_views_test.go index 86486d5..c8b3508 100644 --- a/internal/duck/run_views_test.go +++ b/internal/duck/run_views_test.go @@ -1,6 +1,7 @@ package duck import ( + "context" "encoding/json" "fmt" "testing" @@ -65,3 +66,25 @@ func TestRunAnswersIsEmptyNotMissingOnAFreshDataset(t *testing.T) { t.Errorf("the learner count binds to %d on a fresh dataset, want 0", n) } } + +func TestRunAnswersRunIDIsBigint(t *testing.T) { + d := newDS(t, "ds") + buildStore(t, d, 700, [][]byte{answerRec("s", "https://p/d/e1", "q1", "a")}) + for name, e := range map[string]*Engine{ + "with membership": openEngine(t, []DatasetSpec{{DS: d}}, nil), + "fresh": openEngine(t, []DatasetSpec{{DS: newDS(t, "fresh")}}, nil), + } { + rows, err := e.Query(context.Background(), "SELECT data_type FROM information_schema.columns WHERE table_name = 'run_answers' AND column_name = 'run_id'") + if err != nil { + t.Fatalf("%s: %v", name, err) + } + var typ string + if rows.Next() { + rows.Scan(&typ) + } + rows.Close() + if typ != "BIGINT" { + t.Errorf("%s: run_id is %s, want BIGINT", name, typ) + } + } +} diff --git a/internal/duck/views.go b/internal/duck/views.go index f4a5386..e3d262b 100644 --- a/internal/duck/views.go +++ b/internal/duck/views.go @@ -368,7 +368,7 @@ func (vs viewSet) runAnswersView() viewStmt { name := vs.prefix + `"run_answers"` answers := vs.prefix + sqlIdent(store.TypeAnswers) membership := vs.prefix + `"run_membership"` - primary := fmt.Sprintf("CREATE VIEW %s AS SELECT m.run_id, s.* FROM %s s JOIN %s m USING (source_key, remote_endpoint, question_id) WHERE m.type = %s", + primary := fmt.Sprintf("CREATE VIEW %s AS SELECT CAST(m.run_id AS BIGINT) AS run_id, s.* FROM %s s JOIN %s m USING (source_key, remote_endpoint, question_id) WHERE m.type = %s", name, answers, membership, sqlStr(store.TypeAnswers)) fallback := fmt.Sprintf("CREATE VIEW %s AS SELECT CAST(NULL AS BIGINT) AS run_id, s.* FROM %s s WHERE false", name, answers) return viewStmt{name: name, primary: primary, fallback: fallback} diff --git a/internal/packages/build.go b/internal/packages/build.go index 3629a9c..6c5b52a 100644 --- a/internal/packages/build.go +++ b/internal/packages/build.go @@ -45,13 +45,16 @@ func excludedAncestor(rel string) bool { // directory for git. func WriteBuild(out string, archive []byte) error { dir := filepath.Dir(out) - if err := os.MkdirAll(dir, 0o755); err != nil { - return err - } - if filepath.Base(dir) == BuildDirName { - if err := ignoreAll(dir); err != nil { + if filepath.Base(dir) != BuildDirName { + if err := os.MkdirAll(dir, 0o755); err != nil { return err } + } else if err := os.MkdirAll(filepath.Dir(dir), 0o755); err != nil { + return err + } else if err := ensureRealDir(dir); err != nil { + return err + } else if err := ignoreAll(dir); err != nil { + return err } return os.WriteFile(out, archive, 0o644) } diff --git a/internal/packages/proc_unix.go b/internal/packages/proc_unix.go index 0d9442b..d425bf2 100644 --- a/internal/packages/proc_unix.go +++ b/internal/packages/proc_unix.go @@ -7,8 +7,8 @@ import ( "syscall" ) -// killGroup runs the package in its own process group and kills the whole group on timeout, -// so a child the entrypoint started (cc-data, a shell) does not outlive the bound. +// killGroup runs the package in its own process group and kills the whole group when the run +// is canceled or times out, so a child the entrypoint started does not outlive it. func killGroup(cmd *exec.Cmd) { cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) } diff --git a/internal/packages/result.go b/internal/packages/result.go index fd24440..51cc898 100644 --- a/internal/packages/result.go +++ b/internal/packages/result.go @@ -27,8 +27,8 @@ type Result struct { Counts map[string]any `json:"counts"` } -// OutputRefused is a result the runner would refuse: no display.md, one that is a link or -// not a file, or one over the cap. +// OutputRefused is a result the runner would refuse: a missing, linked or oversized +// display.md, a linked summary.txt, or a linked output directory. type OutputRefused struct{ Reason string } func (e *OutputRefused) Error() string { return e.Reason } diff --git a/internal/packages/run.go b/internal/packages/run.go index f079d64..b7ec4cc 100644 --- a/internal/packages/run.go +++ b/internal/packages/run.go @@ -21,11 +21,8 @@ const TimeoutMargin = 10 * time.Minute // with "." so Collect, and so build, never ships it. const RunDirName = ".cc-data-run" -// passthrough is what a laptop adds to the runner's environment so the package's own cc-data -// calls find the researcher's stored credential (keychain or credentials file), reach the -// server, and run at all. The runner sets HTTPS_PROXY and HTTP_PROXY when it has a proxy, and -// NO_PROXY keeps a laptop's exemptions with them. Matching is case-insensitive, so the -// lower-case spellings pass too. +// passthrough is what a laptop adds so the package's own cc-data calls find the stored +// credential, reach the server through any proxy, and run at all. Matching is case-insensitive. var passthrough = []string{ "HOME", "PATH", "USER", "LOGNAME", "LANG", "TZ", "TMPDIR", "DBUS_SESSION_BUS_ADDRESS", "XDG_RUNTIME_DIR", @@ -198,7 +195,11 @@ func prepare(run string, files Files) (runPaths, error) { // ignoreAll keeps git out of a directory this tool makes inside a package, whose files can // hold student data. func ignoreAll(dir string) error { - return os.WriteFile(filepath.Join(dir, ".gitignore"), []byte("*\n"), 0o644) + p := filepath.Join(dir, ".gitignore") + if info, err := os.Lstat(p); err == nil && !info.Mode().IsRegular() { + return fmt.Errorf("refusing to write %s: it is a link or not a file", p) + } + return os.WriteFile(p, []byte("*\n"), 0o644) } func ensureRealDir(d string) error { diff --git a/internal/packages/run_test.go b/internal/packages/run_test.go index 71e55a6..8de16cc 100644 --- a/internal/packages/run_test.go +++ b/internal/packages/run_test.go @@ -263,3 +263,60 @@ func TestRunRefusesALinkedRunDirectory(t *testing.T) { t.Errorf("a file behind the link was removed: %v", err) } } + +func TestRunRefusesALinkedGitignore(t *testing.T) { + skipOnWindows(t, "links need privileges on Windows") + dir, files := shellPackage(t, `echo ok > "$RD_OUTPUT_DIR/display.md"`+"\n", nil) + target := filepath.Join(t.TempDir(), "precious") + writeTree(t, filepath.Dir(target), map[string]string{"precious": "keep"}) + writeTree(t, dir, map[string]string{RunDirName + "/": ""}) + if err := os.Symlink(target, filepath.Join(dir, RunDirName, ".gitignore")); err != nil { + t.Fatal(err) + } + var stderr bytes.Buffer + if _, err := Run(context.Background(), runOpts(t, dir, files, nil, &stderr)); err == nil { + t.Error("a linked .gitignore was written through") + } + if got, _ := os.ReadFile(target); string(got) != "keep" { + t.Errorf("the link's target became %q", got) + } +} + +func TestWriteBuildRefusesALinkedBuildDirectory(t *testing.T) { + skipOnWindows(t, "links need privileges on Windows") + dir := t.TempDir() + elsewhere := t.TempDir() + if err := os.Symlink(elsewhere, filepath.Join(dir, BuildDirName)); err != nil { + t.Fatal(err) + } + if err := WriteBuild(filepath.Join(dir, BuildDirName, "p-0.1.0.zip"), []byte("zip")); err == nil { + t.Error("a linked build directory was written into") + } + if entries, _ := os.ReadDir(elsewhere); len(entries) != 0 { + t.Errorf("files written behind the link: %v", entries) + } +} + +func TestEntrypointFallsBackToPython3WithANote(t *testing.T) { + look := func(have ...string) func(string) (string, error) { + return func(name string) (string, error) { + for _, h := range have { + if h == name { + return "/usr/bin/" + name, nil + } + } + return "", exec.ErrNotFound + } + } + cmd, _, note, err := entrypointCommand("/pkg", "run.py", look("python3.11", "python3")) + if err != nil || cmd != "/usr/bin/python3.11" || note != "" { + t.Errorf("with python3.11: %s, %q, %v", cmd, note, err) + } + cmd, _, note, err = entrypointCommand("/pkg", "run.py", look("python3")) + if err != nil || cmd != "/usr/bin/python3" || !strings.Contains(note, "python3.11") { + t.Errorf("without python3.11: %s, %q, %v", cmd, note, err) + } + if _, _, _, err := entrypointCommand("/pkg", "run.py", look()); err == nil { + t.Error("no interpreter was not an error") + } +} diff --git a/internal/packages/scope.go b/internal/packages/scope.go index aa26153..1315541 100644 --- a/internal/packages/scope.go +++ b/internal/packages/scope.go @@ -30,9 +30,8 @@ type LocalScope struct { Assignments []Assignment `json:"assignments"` } -// ScopeFile is scope.json as the runner writes it. ClueSource is always "firebase" today, the -// runner's constant; Dataset is the bare dataset name, as the runner writes pkg-, while -// RD_DATASET carries the full ref. +// ScopeFile is scope.json as the runner writes it: ClueSource is its constant "firebase", and +// Dataset the bare name, as it writes pkg-, while RD_DATASET carries the full ref. type ScopeFile struct { LocalScope ClueSource string `json:"clue_source"` diff --git a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md index 81da434..cf2e75a 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md +++ b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md @@ -23,9 +23,9 @@ The package rules and applicability are report-server's (requirements, Dependenc **Estimated diff size**: ~170 lines ```diff ---- /tmp/o 2026-10-08 09:00:24.920084911 -0400 -+++ internal/duck/views.go 2026-10-08 09:00:14.224272861 -0400 -@@ -71,6 +71,7 @@ +--- /tmp/o ++++ internal/duck/views.go +@@ -71,6 +71,7 @@ func (vs viewSet) statements() []viewStmt { stmts = append(stmts, vs.storeView(store.TypeAnswers)) stmts = append(stmts, vs.storeView(store.TypeHistory)) stmts = append(stmts, vs.runMembershipView()) @@ -33,19 +33,18 @@ The package rules and applicability are report-server's (requirements, Dependenc stmts = append(stmts, vs.downloadsView()) stmts = append(stmts, vs.attachmentFilesView()) stmts = append(stmts, vs.attachmentStatesView()) -@@ -360,6 +361,20 @@ +@@ -360,6 +361,19 @@ func (vs viewSet) runMembershipView() viewStmt { return viewStmt{name: name, primary: primary, fallback: fallback, files: files} } -+// runAnswersView is every answer once per run whose answers fetch holds it in membership: the -+// type-qualified membership join, registered on every dataset so a run with no answers counts -+// zero rather than failing to bind the way answers_ does. It must follow both views it -+// reads, and it declares no files, so materializing reads it through theirs. ++// runAnswersView is answers with the run_id of each answers membership. Always bound, unlike ++// answers_; it must follow both views it reads and declares no files, so materialize reads ++// it through theirs. +func (vs viewSet) runAnswersView() viewStmt { + name := vs.prefix + `"run_answers"` + answers := vs.prefix + sqlIdent(store.TypeAnswers) + membership := vs.prefix + `"run_membership"` -+ primary := fmt.Sprintf("CREATE VIEW %s AS SELECT m.run_id, s.* FROM %s s JOIN %s m USING (source_key, remote_endpoint, question_id) WHERE m.type = %s", ++ primary := fmt.Sprintf("CREATE VIEW %s AS SELECT CAST(m.run_id AS BIGINT) AS run_id, s.* FROM %s s JOIN %s m USING (source_key, remote_endpoint, question_id) WHERE m.type = %s", + name, answers, membership, sqlStr(store.TypeAnswers)) + fallback := fmt.Sprintf("CREATE VIEW %s AS SELECT CAST(NULL AS BIGINT) AS run_id, s.* FROM %s s WHERE false", name, answers) + return viewStmt{name: name, primary: primary, fallback: fallback} @@ -62,6 +61,7 @@ The package rules and applicability are report-server's (requirements, Dependenc package duck import ( + "context" "encoding/json" "fmt" "testing" @@ -126,12 +126,35 @@ func TestRunAnswersIsEmptyNotMissingOnAFreshDataset(t *testing.T) { t.Errorf("the learner count binds to %d on a fresh dataset, want 0", n) } } + +func TestRunAnswersRunIDIsBigint(t *testing.T) { + d := newDS(t, "ds") + buildStore(t, d, 700, [][]byte{answerRec("s", "https://p/d/e1", "q1", "a")}) + for name, e := range map[string]*Engine{ + "with membership": openEngine(t, []DatasetSpec{{DS: d}}, nil), + "fresh": openEngine(t, []DatasetSpec{{DS: newDS(t, "fresh")}}, nil), + } { + rows, err := e.Query(context.Background(), "SELECT data_type FROM information_schema.columns WHERE table_name = 'run_answers' AND column_name = 'run_id'") + if err != nil { + t.Fatalf("%s: %v", name, err) + } + var typ string + if rows.Next() { + rows.Scan(&typ) + } + rows.Close() + if typ != "BIGINT" { + t.Errorf("%s: run_id is %s, want BIGINT", name, typ) + } + } +} ``` | Mutation | Test that fails | |---|---| | `m.type = 'answers'` dropped | run 700 answers = 5 (the history membership joins) | | the documented count by endpoint instead of `user_id` | run 700 learners = 3 | +| `run_id` taken from membership without the cast | `TestRunAnswersRunIDIsBigint` (INTEGER) | Documentation (the two guards in `internal/guidance/guard_test.go` fail until both files carry the name; the dashes in `core.md` match that list's existing entries): @@ -451,8 +474,8 @@ type Result struct { Counts map[string]any `json:"counts"` } -// OutputRefused is a result the runner would refuse: no display.md, one that is a link or -// not a file, or one over the cap. +// OutputRefused is a result the runner would refuse: a missing, linked or oversized +// display.md, a linked summary.txt, or a linked output directory. type OutputRefused struct{ Reason string } func (e *OutputRefused) Error() string { return e.Reason } @@ -607,9 +630,8 @@ type LocalScope struct { Assignments []Assignment `json:"assignments"` } -// ScopeFile is scope.json as the runner writes it. ClueSource is always "firebase" today, the -// runner's constant; Dataset is the bare dataset name, as the runner writes pkg-, while -// RD_DATASET carries the full ref. +// ScopeFile is scope.json as the runner writes it: ClueSource is its constant "firebase", and +// Dataset the bare name, as it writes pkg-, while RD_DATASET carries the full ref. type ScopeFile struct { LocalScope ClueSource string `json:"clue_source"` @@ -699,11 +721,8 @@ const TimeoutMargin = 10 * time.Minute // with "." so Collect, and so build, never ships it. const RunDirName = ".cc-data-run" -// passthrough is what a laptop adds to the runner's environment so the package's own cc-data -// calls find the researcher's stored credential (keychain or credentials file), reach the -// server, and run at all. The runner sets HTTPS_PROXY and HTTP_PROXY when it has a proxy, and -// NO_PROXY keeps a laptop's exemptions with them. Matching is case-insensitive, so the -// lower-case spellings pass too. +// passthrough is what a laptop adds so the package's own cc-data calls find the stored +// credential, reach the server through any proxy, and run at all. Matching is case-insensitive. var passthrough = []string{ "HOME", "PATH", "USER", "LOGNAME", "LANG", "TZ", "TMPDIR", "DBUS_SESSION_BUS_ADDRESS", "XDG_RUNTIME_DIR", @@ -876,7 +895,11 @@ func prepare(run string, files Files) (runPaths, error) { // ignoreAll keeps git out of a directory this tool makes inside a package, whose files can // hold student data. func ignoreAll(dir string) error { - return os.WriteFile(filepath.Join(dir, ".gitignore"), []byte("*\n"), 0o644) + p := filepath.Join(dir, ".gitignore") + if info, err := os.Lstat(p); err == nil && !info.Mode().IsRegular() { + return fmt.Errorf("refusing to write %s: it is a link or not a file", p) + } + return os.WriteFile(p, []byte("*\n"), 0o644) } func ensureRealDir(d string) error { @@ -970,8 +993,8 @@ import ( "syscall" ) -// killGroup runs the package in its own process group and kills the whole group on timeout, -// so a child the entrypoint started (cc-data, a shell) does not outlive the bound. +// killGroup runs the package in its own process group and kills the whole group when the run +// is canceled or times out, so a child the entrypoint started does not outlive it. func killGroup(cmd *exec.Cmd) { cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) } @@ -1011,6 +1034,9 @@ Tests (the entrypoint is a `/bin/sh` script, so these skip on Windows): - **`TestRunLeavesNothingBehind`**: a package that backgrounds `(sleep 1; echo late > "$RD_OUTPUT_DIR/late.txt")` and exits 0 succeeds, and 1.5 seconds later `late.txt` does not exist. - **`TestRunKillsTheGroupWhenInterrupted`**: the same package, with the parent context canceled after 300 ms, fails as "interrupted" within 5 seconds. - **`TestRunRefusesALinkedRunDirectory`**: `.cc-data-run` linked to another directory is refused, and a file in that directory survives. +- **`TestRunRefusesALinkedGitignore`**: a `.cc-data-run/.gitignore` linked to another file is refused, and the file keeps its content. +- **`TestEntrypointFallsBackToPython3WithANote`**: `python3.11` when found, else `python3` with a note naming 3.11, else an error. +- **`TestWriteBuildRefusesALinkedBuildDirectory`**: a `.cc-data-build` linked elsewhere is refused, and nothing is written behind it. | Mutation | Test that fails | |---|---| @@ -1322,7 +1348,7 @@ func newPackageRunCmd() *cobra.Command { } raw, err := os.ReadFile(scopePath) if err != nil { - return output.Usagef("reading the scope file: %v", err) + return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: fmt.Sprintf("reading the scope file: %v", err)} } scope, err := packages.ParseLocalScope(raw) if err != nil { @@ -1372,13 +1398,12 @@ func newPackageRunCmd() *cobra.Command { return cmd } -// derivationTimeout covers report-service's deriver, which may spend up to 240 seconds -// fetching a scope's activities; the client's 60-second default would cut it off. +// derivationTimeout covers report-server's wait of up to 270 seconds on report-service's +// deriver; the client's 60-second default would cut it off. const derivationTimeout = 5 * time.Minute -// validate asks report-server whether a publish of the archive would be accepted, and returns -// its answer. A report-server without the route is not a refusal: the archive is unchecked, -// the answer is nil, and saying so is the whole result. +// validate asks report-server whether a publish of the archive would be accepted. A server +// without the route gets a warning and a nil answer: the archive is unchecked, not refused. func validate(ctx context.Context, client *api.Client, archive []byte, origin string, official bool) (*api.ValidatedPackage, error) { ans, err := client.ValidatePackage(ctx, archive, origin, official) switch { @@ -1528,7 +1553,7 @@ func (f packagePublishFlags) run(ctx context.Context, client *api.Client, archiv func newPackagePublishCmd() *cobra.Command { var f packagePublishFlags cmd := &cobra.Command{ - Use: "publish [--portal ]", + Use: "publish [--portal ] [--origin projects/] [--official] [--json]", Short: "Publish a built package to the catalog with your cc-data token", Long: "POST the zip to report-server's catalog, which creates the package private on its first\n" + "publish and records the version. --origin projects/ publishes a project package;\n" + @@ -1658,13 +1683,16 @@ func excludedAncestor(rel string) bool { // directory for git. func WriteBuild(out string, archive []byte) error { dir := filepath.Dir(out) - if err := os.MkdirAll(dir, 0o755); err != nil { - return err - } - if filepath.Base(dir) == BuildDirName { - if err := ignoreAll(dir); err != nil { + if filepath.Base(dir) != BuildDirName { + if err := os.MkdirAll(dir, 0o755); err != nil { return err } + } else if err := os.MkdirAll(filepath.Dir(dir), 0o755); err != nil { + return err + } else if err := ensureRealDir(dir); err != nil { + return err + } else if err := ignoreAll(dir); err != nil { + return err } return os.WriteFile(out, archive, 0o644) } @@ -1768,7 +1796,7 @@ Tests (against an `httptest` server that answers 404 for any route it was not gi - **`TestPackageInitRefusesASecondInit`**. - **`TestPackageBuildValidatesThenWrites`**: - Exactly one validate request is made, carrying the bytes that were written, as `application/zip`. - - The zip is `wildfire-responses-0.1.0.zip` with two files, and the printed checksum is the archive's own. + - The zip is `wildfire-responses-0.1.0.zip` with two files, `.cc-data-build/.gitignore` is `*`, and the printed checksum is the archive's own. - A second build from inside the package gives the same checksum. - **`TestPackageBuildWritesNothingTheServerRefuses`**: a 422 `UNPROCESSABLE` keeps its code, exits 5, and no zip is written. - **`TestPackageBuildWarnsOnAPublishedVersionAndStillWrites`**: `already_published: true` prints "users/7/p 0.1.0 is already published; publish will refuse it until the version changes", and the zip is still written. **`TestPackageBuildIsQuietOnANewVersion`** checks the converse. @@ -1786,6 +1814,9 @@ Tests (against an `httptest` server that answers 404 for any route it was not gi - **`TestPackagePublishSendsTheZipAsTheBody`**: checks POST, `application/zip`, `Content-Length`, `origin=projects/20` present, `official` absent, and the bearer. - **`TestPackagePublishRefusesAChecksumItDidNotSend`**. - **`TestPackagePublishPassesTheServersCodeThrough`**: a 409 `ALREADY_EXISTS` keeps its code, exits 5, and is sent once. +- **`TestPackagePublishMapsAuthAndUnansweredFailures`**: `NOT_AUTHENTICATED` exits 3, and a connection dropped before any answer carries the "Publishing again is safe" action. +- **`TestPackageRunWarnsWhatTheDeriverCouldNotRead`**: each unread URL with its reason, and a truncated profile, reach stderr. +- **`TestPackageRunRefusesAnUnreadableScopeFile`**: a missing scope file is `INVALID_SCOPE`, exit 2. | Mutation | Test that fails | |---|---| diff --git a/specs/REPORT-146-package-commands-and-dataset-views/requirements.md b/specs/REPORT-146-package-commands-and-dataset-views/requirements.md index f5c3321..bd6f5d1 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/requirements.md +++ b/specs/REPORT-146-package-commands-and-dataset-views/requirements.md @@ -137,7 +137,7 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - `--scope` names a local scope file. It holds the four keys rigse supplies on the VM: `kind`, `id`, `classes` (`[{class_hash, class_id}]`) and `assignments` (`[{offering_id, runnable_id, name, url}]`). The shape is checked; any other key is refused. - `package init` does not write a scope file, so the file comes from the author. - **R11. The package sees the runner's layout.** Before the entrypoint runs: - - **A fresh run directory** under `dir/.cc-data-run/`, with `in/` and `out/`, both emptied. `.cc-data-run` and everything `package run` empties or creates under it must be a real directory, never a link. A link is refused before anything is removed, so a stray link can never point the emptying at a directory outside the package. + - **A fresh run directory** under `dir/.cc-data-run/`, with `in/` and `out/`, both emptied. `.cc-data-run` and everything `package run` empties or creates under it must be a real directory, never a link. A link is refused before anything is removed, so a stray link can never point the emptying at a directory outside the package, and a `.gitignore` that is a link is refused rather than written through. - **A `.gitignore` containing `*`** written into `dir/.cc-data-run/`. A package directory is usually a git checkout (cc-data-studies ignores only `local-data/`, `__pycache__/` and `*.pyc`), and `out/display.md` holds counts or text drawn from student data. - **`in/scope.json`** in the runner's exact shape: the scope file's four keys; `clue_source` `"firebase"`; `dataset` as the ref's bare name, as the runner writes `pkg-`; and `output_dir` as the absolute `out/` path. - **The runner's environment:** @@ -154,7 +154,7 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - **The question asked.** When the manifest declares any pattern, `package run` sends the patterns and the scope's assignment URLs to `POST /api/v1/packages/applies`. report-server derives the interactive URLs inside those assignments, so the URLs matched are the ones the runner matches. - **A refusal.** If the package does not apply, `run` refuses with report-server's reason, prefixed as the runner prefixes it ("the package does not apply to this class: "), and does not start the entrypoint. - **Partial profiles.** Each URL the deriver could not read, and a truncated profile, is reported on stderr. - - **The call's timeout** is 5 minutes, since the deriver may spend up to 240 seconds; the client's 60-second default would cut it off. + - **The call's timeout** is 5 minutes, since report-server waits up to 270 seconds for a deriver that may spend 240; the client's 60-second default would cut it off. - **Before the route exists (404).** `run` warns that applicability cannot be checked yet and that the VM's runner is the check, and runs the package. - **R13. Execution.** - **The interpreter.** A `.py` entrypoint runs with `python3.11` when it is on `PATH`, else with `python3`, after a one-line note that the VM runs 3.11. Any other entrypoint is executed directly. @@ -166,8 +166,8 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - **No sandbox.** No network namespace, no other uid and no egress proxy: those are the VM's, and the story forbids needing researcher-dashboard infrastructure locally. - **R14. The result, read as the runner reads it.** After the entrypoint exits 0, the output is read without following links: - `display.md` is required, and is refused over R8's cap with its size and the limit (never truncated). - - `summary.txt` is optional, and its first line, trimmed, is the summary. - - `counts.json`, when present and valid, contributes `answers`, `logs` and `log_freshness_at` only. + - `summary.txt` is optional, and its first line, trimmed, is the summary. A `summary.txt` that is a link or not a file refuses the result, as the runner's reader does. + - `counts.json`, when present and valid, contributes `answers`, `logs` and `log_freshness_at` only. One that is a link, not a file or not JSON is ignored, as on the VM. On success, stdout carries one JSON line: the display path and size in bytes, the summary, the counts and the elapsed seconds. - **R15. A `clue_prepull: true` package is refused locally.** The message says CLUE data cannot be fetched by cc-data yet (section 13), so such a package runs only on the VM. @@ -176,7 +176,7 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - A refusal or failure from report-server's validate or applies route keeps the server's code and exit class: 5 for a contract error, 3 for `NOT_AUTHENTICATED`, as every server call does. - A package the scope does not satisfy, or a `clue_prepull` package, is `PACKAGE_REFUSED`. - A non-zero exit, a timeout or an interrupt is `PACKAGE_FAILED`. - - A missing, linked or oversized `display.md` is `PACKAGE_OUTPUT_REFUSED`. + - A missing, linked or oversized `display.md`, or a linked `summary.txt`, is `PACKAGE_OUTPUT_REFUSED`. The last three exit 1 with the single JSON error envelope on stdout. @@ -191,7 +191,7 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - **R18. Reproducible.** The same tree builds to the same bytes: entries in sorted order, a fixed modification time (which Go's writer also records as an extended-timestamp extra field), and nothing that varies between builds. The same package therefore always has the same checksum. - **R19. `build` refuses what `publish` would refuse,** by asking report-server (R6). It takes `--portal` (default: the configured portal), `--origin projects/` and `--official` so the check is made as the publish will be. A refused archive is not written. When validate answers `already_published`, `build` still writes the zip and warns " is already published; publish will refuse it until the version changes". When `publishing_unavailable` names a reason, `build` warns "this package cannot be published yet: " and still writes the zip. `run` ignores both, since running a package needs neither a new version nor a bucket. - **R20. Output.** - - The default output path is `dir/.cc-data-build/-.zip`. It sits inside an excluded dot directory, with the same `.gitignore` as R11's, so building from inside the package directory never feeds one build's zip into the next. A name or version holding a path separator is `INVALID_MANIFEST` rather than a path, since it is unchecked until validate answers, and for good on a server without the route. + - The default output path is `dir/.cc-data-build/-.zip`. It sits inside an excluded dot directory, which must be a real directory as R11's are, with the same `.gitignore` as R11's, so building from inside the package directory never feeds one build's zip into the next. A name or version holding a path separator is `INVALID_MANIFEST` rather than a path, since it is unchecked until validate answers, and for good on a server without the route. - An explicit `--out` inside `dir` is refused unless R17 would exclude that path anyway. - stdout carries one JSON line: the path, `sha256:`, the size and the file count. From fbaef8c09d993703c11e78b5e90e984dc668f83d Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 16:21:35 -0400 Subject: [PATCH 11/17] test: give package run a test seam [REPORT-146] package run's body after the credential lookup moves into packageRunFlags.run(ctx, client, ...), as build and publish already have, with the package and scope file read and checked by load before any credential is needed. A new test runs a shell package against a fake report-server and checks the validate call, the names and server the package sees, and the stdout result line; skipping validate or swapping the ref and its bare name now fails it. environment uses slices.Contains in place of a hand-rolled helper. --- cmd/package.go | 102 +++++++++------ cmd/package_test.go | 65 ++++++++++ internal/packages/run.go | 12 +- .../implementation.md | 118 +++++++++++------- 4 files changed, 202 insertions(+), 95 deletions(-) diff --git a/cmd/package.go b/cmd/package.go index 98c4e1e..d2df5bc 100644 --- a/cmd/package.go +++ b/cmd/package.go @@ -12,6 +12,7 @@ import ( "time" "github.com/concord-consortium/cc-data-cli/internal/api" + "github.com/concord-consortium/cc-data-cli/internal/dataset" "github.com/concord-consortium/cc-data-cli/internal/output" "github.com/concord-consortium/cc-data-cli/internal/packages" "github.com/spf13/cobra" @@ -87,8 +88,65 @@ func newPackageInitCmd() *cobra.Command { return cmd } +// loadedPackage is a package directory and scope file read and checked, before any credential +// is needed. +type loadedPackage struct { + dir string + manifest packages.Manifest + files packages.Files + scope packages.LocalScope +} + +type packageRunFlags struct { + datasetRef, scopePath string + environ func() []string +} + +func (f packageRunFlags) load(dir string) (loadedPackage, error) { + m, files, err := loadPackage(dir) + if err != nil { + return loadedPackage{}, err + } + raw, err := os.ReadFile(f.scopePath) + if err != nil { + return loadedPackage{}, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: fmt.Sprintf("reading the scope file: %v", err)} + } + scope, err := packages.ParseLocalScope(raw) + if err != nil { + return loadedPackage{}, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: err.Error()} + } + return loadedPackage{dir: dir, manifest: m, files: files, scope: scope}, nil +} + +// run validates, runs and reports a loaded package; it takes the client so it can be exercised +// against a test server without a stored credential. +func (f packageRunFlags) run(ctx context.Context, client *api.Client, pkg loadedPackage, ref dataset.Ref, dataRoot string) error { + archive, err := packages.Zip(pkg.files) + if err != nil { + return output.Internalf("%v", err) + } + // A version that is already published, or a portal that cannot store packages yet, + // still runs; only a publish is refused. + if _, err := validate(ctx, client, archive, "", false); err != nil { + return err + } + start := time.Now() + res, err := packages.Run(ctx, packages.RunOptions{ + Dir: pkg.dir, Manifest: pkg.manifest, Files: pkg.files, Scope: pkg.scope, Applies: appliesFor(client, pkg.scope), + Dataset: ref.String(), Name: ref.Name, Portal: ref.Portal.Host(), DataRoot: dataRoot, + ReportSrv: client.BaseURL, Stderr: output.Stderr(), Environ: f.environ, LookPath: exec.LookPath, + }) + if err != nil { + return packageRunError(err) + } + return output.ResultLine(map[string]any{ + "display": res.DisplayPath, "display_bytes": res.DisplayBytes, "summary": res.Summary, + "counts": res.Counts, "elapsed_seconds": int(time.Since(start).Seconds()), + }) +} + func newPackageRunCmd() *cobra.Command { - var datasetRef, scopePath string + f := packageRunFlags{environ: os.Environ} cmd := &cobra.Command{ Use: "run [dir] --dataset --scope ", Short: "Run a package locally exactly as the dashboard's runner will", @@ -98,27 +156,18 @@ func newPackageRunCmd() *cobra.Command { "Results are written under /.cc-data-run/, which build never ships.\n\n" + runDifferences, Args: cobra.MaximumNArgs(1), RunE: func(cmd *cobra.Command, args []string) error { - if datasetRef == "" || scopePath == "" { + if f.datasetRef == "" || f.scopePath == "" { return output.Usagef("--dataset and --scope are required") } - dir := packageDir(args) - m, files, err := loadPackage(dir) + pkg, err := f.load(packageDir(args)) if err != nil { return err } - raw, err := os.ReadFile(scopePath) - if err != nil { - return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: fmt.Sprintf("reading the scope file: %v", err)} - } - scope, err := packages.ParseLocalScope(raw) - if err != nil { - return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: err.Error()} - } cfg, dataRoot, err := loadRuntime() if err != nil { return err } - ref, err := resolveRef(cfg, datasetRef) + ref, err := resolveRef(cfg, f.datasetRef) if err != nil { return err } @@ -129,32 +178,11 @@ func newPackageRunCmd() *cobra.Command { // The terminal's Ctrl-C cannot reach the package's own process group, so it cancels the run. ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) defer stop() - archive, err := packages.Zip(files) - if err != nil { - return output.Internalf("%v", err) - } - // A version that is already published, or a portal that cannot store packages yet, - // still runs; only a publish is refused. - if _, err := validate(ctx, client, archive, "", false); err != nil { - return err - } - start := time.Now() - res, err := packages.Run(ctx, packages.RunOptions{ - Dir: dir, Manifest: m, Files: files, Scope: scope, Applies: appliesFor(client, scope), - Dataset: ref.String(), Name: ref.Name, Portal: ref.Portal.Host(), DataRoot: dataRoot, - ReportSrv: client.BaseURL, Stderr: output.Stderr(), Environ: os.Environ, LookPath: exec.LookPath, - }) - if err != nil { - return packageRunError(err) - } - return output.ResultLine(map[string]any{ - "display": res.DisplayPath, "display_bytes": res.DisplayBytes, "summary": res.Summary, - "counts": res.Counts, "elapsed_seconds": int(time.Since(start).Seconds()), - }) + return f.run(ctx, client, pkg, ref, dataRoot) }, } - cmd.Flags().StringVar(&datasetRef, "dataset", "", "the dataset the package pulls into and reads: /") - cmd.Flags().StringVar(&scopePath, "scope", "", "a JSON file holding the scope's kind, id, classes and assignments") + cmd.Flags().StringVar(&f.datasetRef, "dataset", "", "the dataset the package pulls into and reads: /") + cmd.Flags().StringVar(&f.scopePath, "scope", "", "a JSON file holding the scope's kind, id, classes and assignments") return cmd } diff --git a/cmd/package_test.go b/cmd/package_test.go index 7b5bc88..45ae365 100644 --- a/cmd/package_test.go +++ b/cmd/package_test.go @@ -11,11 +11,14 @@ import ( "net/http/httptest" "os" "path/filepath" + "runtime" "strings" "sync" "testing" "github.com/concord-consortium/cc-data-cli/internal/api" + "github.com/concord-consortium/cc-data-cli/internal/config" + "github.com/concord-consortium/cc-data-cli/internal/dataset" "github.com/concord-consortium/cc-data-cli/internal/output" "github.com/concord-consortium/cc-data-cli/internal/packages" ) @@ -524,3 +527,65 @@ func TestPackagePublishMapsAuthAndUnansweredFailures(t *testing.T) { t.Errorf("unanswered: %+v", cliErr) } } + +func TestPackageRunValidatesThenRunsWithTheRunnersNames(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("the entrypoint is a shell script") + } + dir := t.TempDir() + manifest := strings.Replace(testManifest, `"entrypoint": "run.py"`, `"entrypoint": "run.sh"`, 1) + script := "#!/bin/sh\nenv > \"$RD_OUTPUT_DIR/../env.txt\"\ncp \"$RD_SCOPE_FILE\" \"$RD_OUTPUT_DIR/../scope.json\"\necho ok > \"$RD_OUTPUT_DIR/display.md\"\n" + for name, content := range map[string]string{"manifest.json": manifest, "run.sh": script} { + if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o755); err != nil { + t.Fatal(err) + } + } + scopePath := filepath.Join(t.TempDir(), "scope.json") + if err := os.WriteFile(scopePath, []byte(`{"kind": "class", "id": "h", + "classes": [{"class_hash": "0123456789abcdef0123456789abcdef0123456789abcdef", "class_id": 6}], "assignments": []}`), 0o644); err != nil { + t.Fatal(err) + } + f := packageRunFlags{scopePath: scopePath, environ: func() []string { return []string{"PATH=" + os.Getenv("PATH")} }} + pkg, err := f.load(dir) + if err != nil { + t.Fatal(err) + } + srv := newPackageServer(t, validateAnswering(validated)) + out, _ := capture(t) + ref := dataset.Ref{Portal: config.MustPortal("learn.concord.org"), Name: "wildfire"} + if err := f.run(context.Background(), api.New(srv.URL, "tok"), pkg, ref, "/data/root"); err != nil { + t.Fatal(err) + } + + reqs := srv.requests("/api/v1/packages/validate") + if len(reqs) != 1 || reqs[0].contentType != "application/zip" || len(reqs[0].body) == 0 { + t.Fatalf("validate requests = %d", len(reqs)) + } + run := filepath.Join(dir, packages.RunDirName) + env, err := os.ReadFile(filepath.Join(run, "env.txt")) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"RD_DATASET=learn.concord.org/wildfire\n", "CC_DATA_PORTAL=learn.concord.org\n", "CC_DATA_ROOT=/data/root\n", "RD_REPORT_SERVER_URL=" + srv.URL + "\n"} { + if !strings.Contains(string(env), want) { + t.Errorf("environment lacks %q", strings.TrimSpace(want)) + } + } + var scope map[string]any + raw, err := os.ReadFile(filepath.Join(run, "scope.json")) + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(raw, &scope); err != nil || scope["dataset"] != "wildfire" { + t.Errorf("scope.json dataset = %v, %v", scope["dataset"], err) + } + var line map[string]any + if err := json.Unmarshal(out.Bytes(), &line); err != nil { + t.Fatalf("stdout %q: %v", out.String(), err) + } + for _, key := range []string{"display", "display_bytes", "summary", "counts", "elapsed_seconds"} { + if _, ok := line[key]; !ok { + t.Errorf("result line lacks %s: %v", key, line) + } + } +} diff --git a/internal/packages/run.go b/internal/packages/run.go index b7ec4cc..b9643a9 100644 --- a/internal/packages/run.go +++ b/internal/packages/run.go @@ -9,6 +9,7 @@ import ( "os" "os/exec" "path/filepath" + "slices" "sort" "strings" "time" @@ -259,7 +260,7 @@ func environment(from []string, set map[string]string) []string { if !ok { continue } - if strings.HasPrefix(k, "LC_") || contains(passthrough, strings.ToUpper(k)) { + if strings.HasPrefix(k, "LC_") || slices.Contains(passthrough, strings.ToUpper(k)) { keep[k] = v } } @@ -273,12 +274,3 @@ func environment(from []string, set map[string]string) []string { sort.Strings(out) return out } - -func contains(list []string, s string) bool { - for _, x := range list { - if x == s { - return true - } - } - return false -} diff --git a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md index cf2e75a..f0795ee 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md +++ b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md @@ -709,6 +709,7 @@ import ( "os" "os/exec" "path/filepath" + "slices" "sort" "strings" "time" @@ -959,7 +960,7 @@ func environment(from []string, set map[string]string) []string { if !ok { continue } - if strings.HasPrefix(k, "LC_") || contains(passthrough, strings.ToUpper(k)) { + if strings.HasPrefix(k, "LC_") || slices.Contains(passthrough, strings.ToUpper(k)) { keep[k] = v } } @@ -973,15 +974,6 @@ func environment(from []string, set map[string]string) []string { sort.Strings(out) return out } - -func contains(list []string, s string) bool { - for _, x := range list { - if x == s { - return true - } - } - return false -} ``` ```go //go:build !windows @@ -1225,7 +1217,7 @@ func RouteMissing(err error) bool { **Summary**: R9, R12, R16, R19 to R23. - **Wiring.** Adds the four subcommands to the root, and `Init` with the embedded stub. - **The report-server calls.** `validate` is shared by `build` and `run` and returns report-server's answer: `build` warns when `already_published` is true or `publishing_unavailable` names a reason, and `run` ignores both, since running a package needs neither a new version nor a bucket. `appliesFor` gives `run` its `Applies` function, with a 5-minute timeout for the deriver. -- **Testability.** `build` and `publish` carry a `run(ctx, client, ...)` method, as `reportCreateFlags` does, so both are tested against `httptest` with no stored credential. +- **Testability.** `run`, `build` and `publish` each carry a `run(ctx, client, ...)` method, as `reportCreateFlags` does, so all three are tested against `httptest` with no stored credential. `package run` reads and checks the package and scope file (`load`) before it looks up a credential. **Files affected**: - `cmd/package.go`: new. @@ -1252,6 +1244,7 @@ import ( "time" "github.com/concord-consortium/cc-data-cli/internal/api" + "github.com/concord-consortium/cc-data-cli/internal/dataset" "github.com/concord-consortium/cc-data-cli/internal/output" "github.com/concord-consortium/cc-data-cli/internal/packages" "github.com/spf13/cobra" @@ -1327,8 +1320,65 @@ func newPackageInitCmd() *cobra.Command { return cmd } +// loadedPackage is a package directory and scope file read and checked, before any credential +// is needed. +type loadedPackage struct { + dir string + manifest packages.Manifest + files packages.Files + scope packages.LocalScope +} + +type packageRunFlags struct { + datasetRef, scopePath string + environ func() []string +} + +func (f packageRunFlags) load(dir string) (loadedPackage, error) { + m, files, err := loadPackage(dir) + if err != nil { + return loadedPackage{}, err + } + raw, err := os.ReadFile(f.scopePath) + if err != nil { + return loadedPackage{}, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: fmt.Sprintf("reading the scope file: %v", err)} + } + scope, err := packages.ParseLocalScope(raw) + if err != nil { + return loadedPackage{}, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: err.Error()} + } + return loadedPackage{dir: dir, manifest: m, files: files, scope: scope}, nil +} + +// run validates, runs and reports a loaded package; it takes the client so it can be exercised +// against a test server without a stored credential. +func (f packageRunFlags) run(ctx context.Context, client *api.Client, pkg loadedPackage, ref dataset.Ref, dataRoot string) error { + archive, err := packages.Zip(pkg.files) + if err != nil { + return output.Internalf("%v", err) + } + // A version that is already published, or a portal that cannot store packages yet, + // still runs; only a publish is refused. + if _, err := validate(ctx, client, archive, "", false); err != nil { + return err + } + start := time.Now() + res, err := packages.Run(ctx, packages.RunOptions{ + Dir: pkg.dir, Manifest: pkg.manifest, Files: pkg.files, Scope: pkg.scope, Applies: appliesFor(client, pkg.scope), + Dataset: ref.String(), Name: ref.Name, Portal: ref.Portal.Host(), DataRoot: dataRoot, + ReportSrv: client.BaseURL, Stderr: output.Stderr(), Environ: f.environ, LookPath: exec.LookPath, + }) + if err != nil { + return packageRunError(err) + } + return output.ResultLine(map[string]any{ + "display": res.DisplayPath, "display_bytes": res.DisplayBytes, "summary": res.Summary, + "counts": res.Counts, "elapsed_seconds": int(time.Since(start).Seconds()), + }) +} + func newPackageRunCmd() *cobra.Command { - var datasetRef, scopePath string + f := packageRunFlags{environ: os.Environ} cmd := &cobra.Command{ Use: "run [dir] --dataset --scope ", Short: "Run a package locally exactly as the dashboard's runner will", @@ -1338,27 +1388,18 @@ func newPackageRunCmd() *cobra.Command { "Results are written under /.cc-data-run/, which build never ships.\n\n" + runDifferences, Args: cobra.MaximumNArgs(1), RunE: func(cmd *cobra.Command, args []string) error { - if datasetRef == "" || scopePath == "" { + if f.datasetRef == "" || f.scopePath == "" { return output.Usagef("--dataset and --scope are required") } - dir := packageDir(args) - m, files, err := loadPackage(dir) + pkg, err := f.load(packageDir(args)) if err != nil { return err } - raw, err := os.ReadFile(scopePath) - if err != nil { - return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: fmt.Sprintf("reading the scope file: %v", err)} - } - scope, err := packages.ParseLocalScope(raw) - if err != nil { - return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: err.Error()} - } cfg, dataRoot, err := loadRuntime() if err != nil { return err } - ref, err := resolveRef(cfg, datasetRef) + ref, err := resolveRef(cfg, f.datasetRef) if err != nil { return err } @@ -1369,32 +1410,11 @@ func newPackageRunCmd() *cobra.Command { // The terminal's Ctrl-C cannot reach the package's own process group, so it cancels the run. ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) defer stop() - archive, err := packages.Zip(files) - if err != nil { - return output.Internalf("%v", err) - } - // A version that is already published, or a portal that cannot store packages yet, - // still runs; only a publish is refused. - if _, err := validate(ctx, client, archive, "", false); err != nil { - return err - } - start := time.Now() - res, err := packages.Run(ctx, packages.RunOptions{ - Dir: dir, Manifest: m, Files: files, Scope: scope, Applies: appliesFor(client, scope), - Dataset: ref.String(), Name: ref.Name, Portal: ref.Portal.Host(), DataRoot: dataRoot, - ReportSrv: client.BaseURL, Stderr: output.Stderr(), Environ: os.Environ, LookPath: exec.LookPath, - }) - if err != nil { - return packageRunError(err) - } - return output.ResultLine(map[string]any{ - "display": res.DisplayPath, "display_bytes": res.DisplayBytes, "summary": res.Summary, - "counts": res.Counts, "elapsed_seconds": int(time.Since(start).Seconds()), - }) + return f.run(ctx, client, pkg, ref, dataRoot) }, } - cmd.Flags().StringVar(&datasetRef, "dataset", "", "the dataset the package pulls into and reads: /") - cmd.Flags().StringVar(&scopePath, "scope", "", "a JSON file holding the scope's kind, id, classes and assignments") + cmd.Flags().StringVar(&f.datasetRef, "dataset", "", "the dataset the package pulls into and reads: /") + cmd.Flags().StringVar(&f.scopePath, "scope", "", "a JSON file holding the scope's kind, id, classes and assignments") return cmd } @@ -1808,6 +1828,7 @@ Tests (against an `httptest` server that answers 404 for any route it was not gi - **`TestPackageBuildRefusesAnOutputItWouldCollect`**. - **`TestPackageBuildRefusesANameThatIsAPath`**: on a server without the validate route, a manifest named `../../escaped` is `INVALID_MANIFEST`, and no zip is written anywhere. - **`TestPackageRunKeepsTheServersCode`**: `packageRunError` given applies' 503 `SERVICE_UNAVAILABLE` returns that code with exit 5, and given `NOT_AUTHENTICATED` exit 3. +- **`TestPackageRunValidatesThenRunsWithTheRunnersNames`**: one validate request carries the zip, the package sees `RD_DATASET` as the full ref, `CC_DATA_PORTAL`, `CC_DATA_ROOT` and `RD_REPORT_SERVER_URL` as the client's server, `scope.json` carries the bare name, and the stdout line has `display`, `display_bytes`, `summary`, `counts` and `elapsed_seconds`. - **`TestPackageRunAsksTheServersMatcher`**: the request carries the patterns and the assignment URLs, and a "does not apply" answer comes back as the verdict. - **`TestPackageRunOnAServerWithoutTheRouteIsUnconfirmed`**. - **`TestPackageRunAlwaysSendsURLs`**: the applies body always carries `urls`, which REPORT-167 requires (a missing or `null` `urls` is 400 there). @@ -1828,6 +1849,7 @@ Tests (against an `httptest` server that answers 404 for any route it was not gi | the warning printed when `publishing_unavailable` is `null` | `TestPackageBuildIsQuietWhenPublishingIsAvailable` | | `publishing_unavailable` read under the wrong name | `TestPackageBuildWarnsWhenThePortalCannotPublishAndStillWrites`, `TestValidateReturnsAnUnavailablePortalAsAnAnswerNotAnError` | | `official` not sent to validate | `TestPackageBuildValidatesAsThePublishWillBe` | +| `package run` skips validate, or swaps the ref and its bare name | `TestPackageRunValidatesThenRunsWithTheRunnersNames` | | `packageRunError` without its `CLIError` case | `TestPackageRunKeepsTheServersCode` (exit 1, `INTERNAL`) | | the default output name not checked | `TestPackageBuildRefusesANameThatIsAPath` (the zip lands two directories up) | | `already_published` read under the wrong name | `TestPackageBuildWarnsOnAPublishedVersionAndStillWrites`, `TestValidateReturnsAPublishedVersionAsAnAnswerNotAnError` | From 1b2fb6277cfb9922ddf244f64820c9057f50bd7a Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 16:25:24 -0400 Subject: [PATCH 12/17] docs: close the package commands spec [REPORT-146] The requirements and implementation plan are folded into one closed spec, specs/REPORT-146-package-commands-and-dataset-views.md, holding the requirements, Done when, technical notes, out of scope, what waits on the merge or on REPORT-167, and every decision with its rationale. The source folder is removed. --- ...146-package-commands-and-dataset-views.md} | 332 ++- .../implementation.md | 2038 ----------------- 2 files changed, 212 insertions(+), 2158 deletions(-) rename specs/{REPORT-146-package-commands-and-dataset-views/requirements.md => REPORT-146-package-commands-and-dataset-views.md} (73%) delete mode 100644 specs/REPORT-146-package-commands-and-dataset-views/implementation.md diff --git a/specs/REPORT-146-package-commands-and-dataset-views/requirements.md b/specs/REPORT-146-package-commands-and-dataset-views.md similarity index 73% rename from specs/REPORT-146-package-commands-and-dataset-views/requirements.md rename to specs/REPORT-146-package-commands-and-dataset-views.md index bd6f5d1..8c8c34f 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views/requirements.md +++ b/specs/REPORT-146-package-commands-and-dataset-views.md @@ -1,95 +1,13 @@ # cc-data-cli: the package commands and the dataset views that replace `_lib` **Jira**: https://concord-consortium.atlassian.net/browse/REPORT-146 -**Repo**: https://github.com/concord-consortium/cc-data-cli -**Implementation Spec**: [implementation.md](implementation.md) -**Status**: **In Development** + +**Status**: **Closed** ## Overview Researchers get four `cc-data package` commands (`init`, `run`, `build`, `publish`) for writing a Researcher Dashboard package on a laptop, testing it the way the dashboard's runner will run it, and publishing it to the package catalog with the cc-data token they already have. A new dataset view, `run_answers`, and documented joins on views cc-data already has replace the counting SQL the spike copied into every package. -## Project Owner Overview - -The Researcher Dashboard runs analysis "packages" for a class on a short-lived virtual machine. In the spike, a package was built and uploaded by a script that called AWS directly. Each package also carried a shared library of SQL describing how cc-data lays out a dataset. As a result, developing a package needed AWS credentials, and the only way to find a mistake was to run the package on a VM. - -This story moves the whole loop onto the researcher's laptop and onto the token they already use for cc-data. `cc-data package run` runs a package against their own dataset under the runner's rules: the same `scope.json`, the same output files and the same size limit on the result. It asks report-server, rather than a copy of its rules, whether the package is valid and whether it applies to the class. An author therefore hits these failures on their laptop, not on a VM. `cc-data package publish` registers the package in the catalog in one step. The shared SQL becomes a named view and documented joins in cc-data itself, so a package writes plain SQL against stable names and carries no library. The first package built this way is the Wildfire package (REPORT-147), whose spec waits on this one. - -**One rule, one home.** The package rules (what a valid package is) and applicability (whether a package suits a class) are decided by report-server alone. cc-data, the dashboard app and the runner ask it and do not keep their own copies. The new report-service routes this needs are one companion story, REPORT-167 (see Dependencies). - -## Background - -The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) sections 5.5 (applicability by URL patterns), 5.7 (publishing and iterating), 10 (`manifest.json`, `scope.json`, the result a package writes) and 13 (data sources). The section numbers below are that document's. - -**What `_lib` was.** On cc-data-studies' unmerged spike branch `RIGSE-365-analysis-packages` (`838a603`), `studies/_lib/reports.py` held three queries. `studies/_lib/ccdata.py` held the cc-data calls a class study makes: it creates the Student Answers and Student Actions runs, pulls them, and keeps run ids in `runs.json`. `publish-package.sh` copied `_lib/` into every archive. REPORT-147 builds on cc-data-studies `main`, where `_lib` never landed, so this story adds new views rather than replacing existing code. `_lib` is only the reference for what the views must cover (stream channel, 2026-10-07). The three queries are: - -1. **Answers scoped to a run**: `answers JOIN run_membership USING (source_key, remote_endpoint, question_id) WHERE m.run_id = AND m.type = 'answers'`. The query is written this way, and not against `answers_`, because the per-run view exists only once the run has membership on disk. A class with no answers would then fail to bind instead of counting zero. -2. **Learners with at least one answer**: a `UNION ALL` over every `res__remote_endpoint` column of `report_` (paired with `user_id`), joined to that run's answers, then `count(DISTINCT user_id)`. The column list differs from class to class, so the spike read it with `DESCRIBE` before building the SQL. -3. **Log freshness**: `count(*), max(event_time) FROM logs WHERE run_id = `. - -`ccdata.py`'s run creation and pulling stay package code. Section 13 says a package makes its own pulls, and the Jira story says the runner does not create a package's report runs. Nothing replaces that half in cc-data. - -**What cc-data already has** (`internal/duck/views.go`, released in v0.2.0, which is `main` at `58c9172`): - -- `run_membership` (run_id, type, identity columns), `answers`, `history`, `reports`, `report_`, `answers_`. Each has a typed-empty fallback, so they always bind. -- `logs`, with `run_id` and `event_time` (REPORT-112). The freshness query is already plain SQL against stable names, so query 3 needs no new view. -- `student_id_mapping`, one row per learner from Student ID Mapping runs, with `user_id` and `run_remote_endpoint` (a bare endpoint already nulled). Joined to a run's answers, it counts learners without double-counting a student who answered in two assignments, which is query 2's goal, so query 2 needs no new view either. - -**Why learners come from Student ID Mapping, not the Student Answers CSV** (REPORT-147's measurements on production, `bca55aa`, decided by Doug 2026-10-08). A whole-class Student Answers run fails above three or four assignments, and real Wildfire classes have 8 to 14. A reused Student Answers run's CSV is fixed when its query ran, so a learner who joins later has answers but no row. A Student ID Mapping run is a Portal report computed on every download, with no assignment limit, and its id fetches answers like any run. So the package route this story teaches is Student ID Mapping, and `_lib`'s Student Answers union (query 2) does not carry over. -- A drift guard (`internal/guidance/guard_test.go`): every name in `StaticViewNames()` must be documented in `internal/guidance/src/core.md`'s Views section and in `docs/researcher-guide.md`'s "How datasets are organized" table, and the reverse. `cc-data query --help` and the MCP `query` tool description list the static views from the same function. - -**What the runner does to a package.** This is RD-4 pass 2 as spec'd and built in its stage-verification throwaway build (`stage5-throwaway-build.patch`, researcher-dashboard branch oob `RD-4-pass-2-pull-loop`). It is not implemented on any branch yet. - -- It validates `manifest.json`: - - required fields; - - `expected_duration_seconds` a positive integer and at most `PACKAGE_MAX_DURATION_SECONDS` (default 7200); - - `clue_prepull` boolean if present; - - `urls` keys only `all`, `any` and `none`, each a list of non-empty strings; - - the entrypoint inside the package; - - `name` equal to the last segment of the identity. -- When `urls` declares any pattern, it matches the patterns against the class profile's `assignment_urls` ∪ `interactive_urls`, using the glob of section 5.5. It refuses with "the package does not apply to this class: ". It also refuses a package that declares patterns on a class with no profile yet. -- It writes `scope.json` with exactly `kind`, `id`, `classes`, `assignments`, `clue_source`, `dataset` (the bare name `pkg-`) and `output_dir`. It names the file in `RD_SCOPE_FILE`. -- It replaces the package's environment with `HOME`, `PATH=/usr/local/bin:/usr/bin:/bin`, `CC_DATA_ROOT`, `CC_DATA_PORTAL`, `CC_DATA_LOCAL`, `RD_DATASET` (`/pkg-`), `RD_SCOPE_FILE`, `RD_DATA_DIR` and `RD_OUTPUT_DIR`, plus `RD_REPORT_SERVER_URL`, `RD_BUCKET` and `RD_STORAGE_PREFIX` when set. -- It runs a `.py` entrypoint with `python3.11`, and any other entrypoint directly. The bound is `expected_duration_seconds` plus a 10-minute margin. -- It empties the output directory before each run. Afterwards it reads, without following links: - - `display.md`, required and refused over 65,536 bytes (never truncated); - - `summary.txt`, whose first line, trimmed, is the summary (or null); - - `counts.json`, for `answers`, `logs` and `log_freshness_at` only. - -**What report-server accepts** (`server/lib/report_server/packages/*.ex` and `package_controller.ex` on report-service `master`, `067a1c4`, live on staging as `1.12.0-pre.2`): - -- **The request.** `POST /api/v1/packages` behind the ordinary API token (the cc-data token). The body is the zip, with `Content-Type: application/zip` and a declared `Content-Length`. Query parameters are `origin=projects/` (absent means the caller's own `users/`) and `official=true|false`. Only a publisher may set `official`. -- **Archive limits.** At most 10 MiB, at most 50 MiB declared uncompressed in total, no absolute or `..` entry, exactly one root `manifest.json` of at most 64 KiB. -- **Manifest rules.** - - `name` matches `^[a-z0-9][a-z0-9-]{0,62}$`. - - `version` is `MAJOR.MINOR.PATCH[-prerelease]`, at most 64 characters. - - `title` is non-empty, at most 200 characters. - - `description` is optional: one line, at most 500 characters. - - `urls` has only `all`/`any`/`none` arrays, each pattern non-empty with no whitespace or control characters and at most 256 characters, and at most 20 patterns in total. - - `clue_prepull` is boolean. - - `entrypoint` is a file in the archive. - - `expected_duration_seconds` is 1 to 28,800. - - No `owner`, `maintainer`, `origin`, `visibility`, `project` or `official` key. -- **On success (201).** The response is `{catalog_id, identity, version, checksum, visibility, official, current_version}`, where `checksum` is `sha256:` of the body. -- **On failure.** The flat envelope `{"error", "message"}` with `BAD_REQUEST`, `UNPROCESSABLE` (422), `FORBIDDEN`, `ALREADY_EXISTS` (409), `NOT_AUTHENTICATED` or `SERVICE_UNAVAILABLE`. -- **Visibility and the version pointer.** A first publish creates the package `private`. For a private package, every publish moves `current_version` to the new version. - -## Dependencies - -**REPORT-167** (report-service: one home for the package rules and applicability, created 2026-10-08) owns the routes cc-data calls. REPORT-146 codes against the contract below, which that story's spec may refine. Until those routes are deployed, `build` and `run` warn and proceed (R6, R12), so REPORT-146 is not blocked on it. - -- **`POST /api/v1/packages/validate`** (cc-data token). The body is the raw zip, sent exactly as for publish, with `origin` and `official` as optional query parameters, as publish takes them. It runs every check `POST /api/v1/packages` runs, including permission and origin, and stops before writing any row or object. It answers 200 with `{identity, version, checksum, visibility, already_published}` (what a publish would record now), or the same coded error a publish would give for any other refusal. **A version that is already published is reported, not refused:** `already_published` is true and the answer is still 200, while only the publish itself answers `ALREADY_EXISTS` (REPORT-167's R12, refined after REPORT-146's first spec commit). **A portal that cannot store packages yet is reported, not refused:** `publishing_unavailable` is `null` or publish's own message (today only "publishing is not configured for ", before the portal has a bucket in `PackageBuckets`), the answer is still 200, and every other check still runs; only the publish answers that 422. Without this, `package run` would refuse every package on staging until its bucket is set (REPORT-167, `1ff2fc0`). The answer is thus `{identity, version, checksum, visibility, already_published, publishing_unavailable}`. - - It is a separate route, not a `dry_run` parameter on publish. Today's publish ignores parameters it does not know, so a `dry_run=true` sent to a server without the feature would publish for real. A missing route answers 404. -- **`POST /api/v1/packages/applies`** (cc-data token, or the researcher token the runner holds). The body is JSON: `{urls: {all, any, none}, assignment_urls: [...], scope_urls: [...]}`. - - **Assignment URLs** are followed by report-service's existing deriver (`deriveProfile` in `functions/src/researcher-dashboard/derive-profile.ts`). report-server reaches it through a new `derive_urls` route on the `api` function, the way it already calls `bulk_read`. - - **Scope URLs** are matched as given; that is the runner's case, since it already holds the class profile. - - **`urls` is required**: a missing or `null` `urls` is 400 `BAD_REQUEST`, while a `null` group inside it counts as empty. cc-data always sends it. - - **The answer** is `{applies, reason, interactive_urls, unread: [{url, reason}], truncated}`. The `reason` uses the runner's wording. - - **The matcher** behind it is the only glob matcher in the system. The app gets the same answer through the package-list call it already makes. -- **One contract fixture** of identity, version and package-key cases lives in report-service, asserted by report-server, the function, the runner and the app. cc-data's only copy of that grammar is `init`'s name check (R9), kept on purpose. -- **Neither route answers 404 on a report-server that has it** (REPORT-167). So a 404 still means only that the route is not deployed. A deriver failure answers 400 or 503, and report-server waits up to 270 seconds for the deriver, inside cc-data's 5-minute timeout. -- **report-server adopts the runner's stricter rules,** so that anything it accepts the runner will run: `expected_duration_seconds` at most 7,200 (today 28,800), and no symbolic-link entries in an archive. - ## Requirements ### Dataset views @@ -218,7 +136,7 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - **One classification.** Both steps read one decision made once in the workflow, and a test pins both uses. - The change ships in this story's PR, so the first tag after merge is already safe. -## Done when +### Done when - A package made with `init`, edited and run with `package run` against a real dataset, produces `display.md`, `summary.txt` and `counts.json` under the runner's layout. - `display.md` over 65,536 bytes fails `package run` at the same size the runner refuses. @@ -230,6 +148,24 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se ## Technical Notes +### Dependencies + +**REPORT-167** (report-service: one home for the package rules and applicability, created 2026-10-08) owns the routes cc-data calls. REPORT-146 codes against the contract below, which that story's spec may refine. Until those routes are deployed, `build` and `run` warn and proceed (R6, R12), so REPORT-146 is not blocked on it. + +- **`POST /api/v1/packages/validate`** (cc-data token). The body is the raw zip, sent exactly as for publish, with `origin` and `official` as optional query parameters, as publish takes them. It runs every check `POST /api/v1/packages` runs, including permission and origin, and stops before writing any row or object. It answers 200 with `{identity, version, checksum, visibility, already_published}` (what a publish would record now), or the same coded error a publish would give for any other refusal. **A version that is already published is reported, not refused:** `already_published` is true and the answer is still 200, while only the publish itself answers `ALREADY_EXISTS` (REPORT-167's R12, refined after REPORT-146's first spec commit). **A portal that cannot store packages yet is reported, not refused:** `publishing_unavailable` is `null` or publish's own message (today only "publishing is not configured for ", before the portal has a bucket in `PackageBuckets`), the answer is still 200, and every other check still runs; only the publish answers that 422. Without this, `package run` would refuse every package on staging until its bucket is set (REPORT-167, `1ff2fc0`). The answer is thus `{identity, version, checksum, visibility, already_published, publishing_unavailable}`. + - It is a separate route, not a `dry_run` parameter on publish. Today's publish ignores parameters it does not know, so a `dry_run=true` sent to a server without the feature would publish for real. A missing route answers 404. +- **`POST /api/v1/packages/applies`** (cc-data token, or the researcher token the runner holds). The body is JSON: `{urls: {all, any, none}, assignment_urls: [...], scope_urls: [...]}`. + - **Assignment URLs** are followed by report-service's existing deriver (`deriveProfile` in `functions/src/researcher-dashboard/derive-profile.ts`). report-server reaches it through a new `derive_urls` route on the `api` function, the way it already calls `bulk_read`. + - **Scope URLs** are matched as given; that is the runner's case, since it already holds the class profile. + - **`urls` is required**: a missing or `null` `urls` is 400 `BAD_REQUEST`, while a `null` group inside it counts as empty. cc-data always sends it. + - **The answer** is `{applies, reason, interactive_urls, unread: [{url, reason}], truncated}`. The `reason` uses the runner's wording. + - **The matcher** behind it is the only glob matcher in the system. The app gets the same answer through the package-list call it already makes. +- **One contract fixture** of identity, version and package-key cases lives in report-service, asserted by report-server, the function, the runner and the app. cc-data's only copy of that grammar is `init`'s name check (R9), kept on purpose. +- **Neither route answers 404 on a report-server that has it** (REPORT-167). So a 404 still means only that the route is not deployed. A deriver failure answers 400 or 503, and report-server waits up to 270 seconds for the deriver, inside cc-data's 5-minute timeout. +- **report-server adopts the runner's stricter rules,** so that anything it accepts the runner will run: `expected_duration_seconds` at most 7,200 (today 28,800), and no symbolic-link entries in an archive. + +### Notes + - **Views live in `internal/duck/views.go`.** Add one builder to `viewSet.statements()`, after `runMembershipView` (`run_answers` reads `answers` and `run_membership`, so it must register after both). Use `vs.prefix` for every referenced view so multi-dataset sessions resolve to the right schema, as `perStoreView` does. - **What the tests can see.** `StaticViewNames()` and the guard pick up the view automatically. `cmd/query.go`'s help and `internal/mcpserver/tools.go`'s `query` description list it with no edit. - **Verified against real staging data.** On `learn.portal.staging.concord.org/staging-smoketest`, run 164: `_lib`'s answers SQL gives 5, and `run_answers` gives 5. REPORT-147 measured the Student ID Mapping learner join against the Student Answers union on production for both Wildfire classes (runs 2390 to 2393): the two count the same today. @@ -256,12 +192,16 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se - Bumping the runner image's cc-data pin; deleting `_lib` or the spike packages; a dashboard control that hands an author a ready-made scope file (RD-3). - Recording the cc-data version in a package's result, which is the runner's image version (section 13). -## Open Questions +## Not Yet Implemented - +- Tagging `main` as `v0.3.0-pre.1` after this story merges, and checking that its GitHub release is marked as a pre-release with the linux/amd64 archive and its `.sha256` and no Homebrew formula (R25, Done when). It is a release step after the merge, not code on the branch. +- Publishing a private package and a version row on staging with `cc-data package publish` (Done when). It runs against staging once the branch is released. +- `build` and `run` refusing what report-server refuses, and `run` refusing a package that does not apply, as observed against a live server (Done when): both wait on REPORT-167's routes being deployed, and until then both warn and proceed. +- A package built on a laptop producing the same output on the VM: moved to REPORT-147 (Doug, 2026-10-08). -### RESOLVED: Which URLs does `package run` match a package's patterns against? +## Decisions + +### Which URLs does `package run` match a package's patterns against? **Context**: The runner matches against the class profile's `assignment_urls` ∪ `interactive_urls`, while `scope.json` carries only assignment URLs, so a pattern on an interactive (the Wildfire case) would be refused locally and accepted on the VM. The profile is derived by report-service's function from public authoring JSON. **Options considered**: - A) A Go port of the deriver's walk in cc-data. @@ -271,20 +211,28 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se **Decision**: D, taken further (Doug, 2026-10-08): report-server holds the only matcher, and cc-data, the dashboard app and the runner all ask it (Dependencies, `POST /api/v1/packages/applies`). The deriver keeps its one copy, and there is no glob matcher anywhere else. The app and the runner already call report-service at startup, so the call adds no new dependency for either. Before the route is deployed, `run` warns and proceeds (R12). -### RESOLVED: Where does the shared pattern fixture live? +--- + +### Where does the shared pattern fixture live? **Decision**: The question dissolves under the decision above (Doug, 2026-10-08). With one matcher in report-server, `url-patterns.json` is an ordinary test file beside it in report-service, and nothing vendors a copy. cc-data has none. -### RESOLVED: How is "a package built on a laptop runs in the VM and produces the same output" verified? +--- + +### How is "a package built on a laptop runs in the VM and produces the same output" verified? **Decision**: REPORT-147 owns it (Doug, 2026-10-08). Its done-when already requires the Wildfire package to run "identically under `cc-data package run` on a laptop and on a VM", and it is the first story that can put a package on a VM. REPORT-146 closes on the criteria in "Done when" below. -### RESOLVED: Judgment call: doing applicability and the rules in report-server rather than in cc-data +--- + +### Doing applicability and the rules in report-server rather than in cc-data **Options considered**: - A) cc-data keeps Go copies of the manifest rules, the archive limits and the matcher, guarded by fixtures. - B) cc-data asks report-server's validate and applies routes, keeping only what it needs to stage and start a package. **Decision**: B (Doug, 2026-10-08, to remove DRY violations without multiplying stories). The copies already disagreed: report-server allows 28,800 seconds where the runner refuses over 7,200. REPORT-167 makes report-server adopt the runner's stricter rules, so its answer is the runner's. -### RESOLVED: Keep `learner_endpoints`, and which route the `init` stub teaches +--- + +### Keep `learner_endpoints`, and which route the `init` stub teaches **Context**: REPORT-147 (`bca55aa`) counts learners through a Student ID Mapping run with `run_answers` and `student_id_mapping`, after measuring on production that whole-class Student Answers runs fail above three or four assignments and that a reused run's learner list goes stale. That left `learner_endpoints` with no consumer, and the stub teaching the failing route. **Options considered**: - A) Drop `learner_endpoints`, document the Student ID Mapping learner count, and switch the stub to Student ID Mapping. @@ -292,21 +240,27 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se **Decision**: A (Doug, 2026-10-08). Nothing uses the view, the route it rests on fails on real classes, and the existing `student_id_mapping` view already makes the count without double-counting. The stub is the template authors copy, so it teaches the route that works. -### RESOLVED: Judgment call: where a local scope comes from +--- + +### Where a local scope comes from **Options considered**: - A) An author-supplied file (`--scope`) carrying the four keys rigse supplies. - B) Synthesized from a pulled Student Answers report (`class_id`, and `res__offering_id`, `_name` and `_resource_url` per assignment). **Decision**: A. The report has no `class_hash` or `runnable_id`, and the scope contract has no room for nulls in either. Any synthesized scope would differ from the VM's in exactly the fields a package might key on. A dashboard control that hands out the real scope is the eventual source (RD-3, out of scope). -### RESOLVED: Judgment call: a `clue_prepull` package under `package run` +--- + +### A `clue_prepull` package under `package run` **Options considered**: - A) Refuse it locally (R15). - B) Run it with `clue_source: "firebase"` and an empty corpus. **Decision**: A. B tells the package something false, and the package would produce numbers the VM would not. Section 13 already says a CLUE package is the one package that cannot be developed locally until REPORT-109 to 111. -### RESOLVED: Low confidence: the passthrough environment in R11 +--- + +### The passthrough environment in R11 **Context**: The runner replaces the environment entirely. Locally, a package's own `cc-data` calls must find the researcher's credential: the OS keychain via `go-keyring`, or `~/.config/cc-data/credentials.json`. **Options considered**: - A) The R11 allowlist. @@ -314,7 +268,9 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se **Decision**: A, verified on this machine (Linux, keyring backend). `env -i` with the R11 list, then `cc-data reports list --portal learn.concord.org --json`, reads the keyring token and lists runs. So does `env -i HOME PATH USER` alone: go-keyring reaches the session bus without `DBUS_SESSION_BUS_ADDRESS` here, so the D-Bus and XDG variables are kept only for setups that need them. On macOS go-keyring calls `/usr/bin/security`, which needs nothing beyond `HOME` and a `PATH` reaching `/usr/bin`. A cc-data query under the same `env -i` reads a dataset. B would let a package pass locally while depending on a variable the VM never sets. -### RESOLVED: Low confidence: `scope.json`'s `dataset` is a bare name +--- + +### `scope.json`'s `dataset` is a bare name **Context**: `final-design.md` section 10 shows `"dataset": ""`. RD-4 pass 2's `scopeFile` writes the bare `pkg-`, while `RD_DATASET` carries the full ref. A package that passes `scope["dataset"]` to `cc-data query --dataset` resolves it under the configured default portal, which may not be the dataset's. **Options considered**: - A) Mirror the runner (bare name), have the `init` stub read `RD_DATASET`, and record the discrepancy for RD-4 and the design to settle. @@ -322,40 +278,176 @@ The design is `final-design.md` (global oob, `streams/researcher-dashboard/`) se **Decision**: A. This story's contract is "exactly as the runner", and B changes a contract RD-4 owns from a story that consumes it. RD-4 pass 2 has since settled it as the bare name, and `final-design.md` section 10 says so (stream channel #28). The `init` stub reads `RD_DATASET`, the one name that is a full ref on both sides. -## Self-Review +--- + +### Emptying the run directory could follow a link out of the package +**Context**: Raised in self-review (Security Engineer). + +**Decision**: R11 empties `dir/.cc-data-run/in` and `out` on every run. If `.cc-data-run` were a symbolic link to another directory, the emptying would delete under that target. Fixed in R11: every directory `package run` empties or creates must be a real directory, and a link is refused before anything is removed. + +--- + +### `publish` would upload a hand-made zip the runner then refuses +**Context**: Raised in self-review (Security Engineer). + +**Decision**: report-server checks entry paths but not entry types, and the runner refuses a symbolic link after unpacking (RD-4's `checkTree`). So a zip made outside `build` could publish and then fail on every VM. Resolved by REPORT-167: report-server refuses link entries at publish, so the one check covers every zip. + +--- + +### Run output written inside a git checkout +**Context**: Raised in self-review (Education Researcher). + +**Decision**: `out/display.md` is derived from student data and lands inside the package directory. cc-data-studies' `.gitignore` ignores only `local-data/`, `__pycache__/` and `*.pyc` (read 2026-10-07), so `git add -A` would stage it. Fixed in R11 and R20: each working directory gets a `.gitignore` of `*`. + +--- + +### `_run_id` reads like run scoping and is not +**Context**: Raised in self-review (Education Researcher). + +**Decision**: The answers store stamps `_run_id` with the run that fetched each stored version (`internal/store/segment.go` `AppendPage`), and a later run's re-fetch overwrites it. Next to `run_answers.run_id`, a researcher could filter `answers` by `_run_id` and silently undercount. Fixed in R5's documentation requirement. + +--- + +### The default build output would be swept into the next build +**Context**: Raised in self-review (Package Author). + +**Decision**: With the output in the current directory and `build` run from inside the package, as authors do, R20 as written either refused its own default or zipped the last build into the next. Fixed in R20: the default is `dir/.cc-data-build/`, which R17 already excludes. + +--- + +### An `init` stub that only reads existing runs reports nothing on the VM +**Context**: Raised in self-review (Package Author). + +**Decision**: On the VM a package's dataset (`pkg-`) starts empty, and section 13 makes run creation package code. A stub that assumed a pulled run would pass locally and count zero on its first VM run. Fixed in R9: the stub reuses or creates its own run, as `_lib/ccdata.py` and RD-4's smoke package do (a Student ID Mapping run since 2026-10-08). + +--- + +### R3 could pass with a fixture that cannot tell the queries apart +**Context**: Raised in self-review (QA Engineer). + +**Decision**: With one assignment per learner, counting endpoints and counting users give the same number, so a broken learner view would pass. Fixed in R3: the synthetic dataset includes a two-assignment learner, a bare endpoint, an answer in two runs and a second class's run. + +--- + +### Staging's answer before `PackageBuckets` exists was unstated +**Context**: Raised in self-review (Operator). + +**Decision**: `Store.bucket_for` returns 422 "publishing is not configured for " after every archive and manifest check has passed. Recorded in Technical Notes, so that answer reads as a well-formed request rather than a failure of this story. + +--- + +### How a raw zip body reaches the server +**Options considered**: +- A) Generalize `Client.do` into `send` with a content type. +- B) Build a separate `http.Request` for the zip routes. + +**Decision**: A. Publish and validate then share the bearer, the cross-origin redirect refusal, the no-retry rule for POST and the error-envelope decoding with every other call. B would re-implement each of them. + +--- + +### The exit class for a refused or failed package +**Options considered**: +- A) Exit 1 with codes `PACKAGE_REFUSED`, `PACKAGE_FAILED` and `PACKAGE_OUTPUT_REFUSED`. +- B) A new exit code 7. + +**Decision**: A. The root help defines exit 1 as "internal/other error", and the JSON envelope's code already distinguishes the cases. A new exit class is a contract change for every script that reads cc-data's exit codes. + +--- + +### Following no link without `O_NOFOLLOW` +**Options considered**: +- A) `Lstat`, then open, then `os.SameFile`. +- B) `syscall.O_NOFOLLOW`. + +**Decision**: A. CI tests on `windows-2022`, where `syscall.O_NOFOLLOW` does not exist. On a laptop the race A leaves open would need the author's own package to swap a file mid-read, and the runner, which runs as root, is the boundary that matters. + +--- + +### What a 404 from the new routes means +**Options considered**: +- A) Any 404 from validate or applies means the route is not deployed: warn and proceed. +- B) Require a specific error code before treating it as "not deployed". + +**Decision**: A. report-server's router has no route either path could match today (`/packages/validate` and `/packages/applies` are one segment, and the only other `POST /packages/...` route takes four). So a 404 can only mean the route is missing, and B would need a code that an older server, by definition, cannot send. + +--- + +### The process-group kill on macOS +**Context**: `TestRunKillsAtTheBound` ran on Linux only. + +**Decision**: `Setpgid` and `kill(-pgid, SIGKILL)` are POSIX and behave the same on Darwin, and `GOOS=darwin go build ./internal/packages/` compiles. CI's `macos-15` and `macos-15-intel` jobs run the test, so a Darwin difference would fail there. + +--- + +### A shell entrypoint would run locally and be refused on the VM +**Context**: Raised in self-review (Senior Engineer). + +**Decision**: The zip originally gave every entry mode 0644, and the runner executes a non-`.py` entrypoint directly. A Go zip with a 0644 `plain.sh` and a 0755 `exec.sh`, unpacked with `unzip`, gives "Permission denied" for the first and runs the second, so `unzip` restores the zip's modes. Locally, the staged copy kept the source's 0755, so `package run` passed. Fixed: +- `Files` records a ship mode per file (0755 if any execute bit is set, else 0644). +- `Zip` and the staging copy both use it. +- `CheckEntrypointMode` refuses a non-executable non-`.py` entrypoint in `run` and `build`. + +R17 states the rule. + +--- + +### A scope file inside the package ships with it +**Context**: Raised in self-review (Operator). + +**Decision**: An author naming the file `scope.json` in the package directory would publish a class hash in every archive. The documentation step says to keep it outside the package or under a dot name, which `Collect` already excludes. + +--- + +### The applicability call would time out before the deriver finishes +**Context**: Raised in self-review (Operator). + +**Decision**: The deriver may spend up to 240 seconds on a scope's activities, and the client's per-attempt timeout for JSON calls is 60 seconds. `appliesFor` copies the client with a 5-minute `RequestTimeout`, which also keeps the shared client's default for every other call. + +--- + +### An applies failure lost its code and exit class +**Context**: Raised in self-review (Senior Engineer). + +**Decision**: `Run` returns `appliesFor`'s `*output.CLIError` unchanged, and `packageRunError` matched only the three package error types, so it fell through to `Internalf`. A throwaway test fed it applies' 503 `SERVICE_UNAVAILABLE` and `NOT_AUTHENTICATED`, and both came back exit 1 `INTERNAL`, against R16. `packageRunError` now returns a `CLIError` as it is, and `TestPackageRunKeepsTheServersCode` pins it. + +--- + +### Ctrl-C left the package running +**Context**: Raised in self-review (Package Author). + +**Decision**: `Setpgid` takes the package out of the terminal's foreground group, and cc-data had no interrupt handling outside `dataset materialize`. Sending `SIGINT` to cc-data's group, as a terminal does, ended cc-data while the package's `sleep 30` kept running. `package run` now cancels on `os.Interrupt` (the `materialize` idiom), which kills the group through the existing `cmd.Cancel`, and `Run` reports "the package was interrupted". Rerun with the fix, the package was gone a second after the signal. + +--- -Roles: senior engineer, security engineer, QA engineer, education researcher, package author, operator. Every finding below was checked against the code before it was written down. Each had one defensible correction, which has been applied to the requirements above. +### `build`'s default path came from unchecked manifest fields +**Context**: Raised in self-review (Security Engineer). -### Security Engineer +**Decision**: On a server without the validate route, a manifest named `../../escaped` built to `escaped-1.0.0.zip` two directories above the package, outside `.cc-data-build` and its `.gitignore`. `build` now refuses a name and version that make a path. It is a path check, not a copy of report-server's name grammar. -#### RESOLVED: Emptying the run directory could follow a link out of the package -R11 empties `dir/.cc-data-run/in` and `out` on every run. If `.cc-data-run` were a symbolic link to another directory, the emptying would delete under that target. Fixed in R11: every directory `package run` empties or creates must be a real directory, and a link is refused before anything is removed. +--- -#### RESOLVED: `publish` would upload a hand-made zip the runner then refuses -report-server checks entry paths but not entry types, and the runner refuses a symbolic link after unpacking (RD-4's `checkTree`). So a zip made outside `build` could publish and then fail on every VM. Resolved by REPORT-167: report-server refuses link entries at publish, so the one check covers every zip. +### The proxy variables were not passed through +**Context**: Raised in self-review (Operator). -### Education Researcher +**Decision**: cc-data's client uses Go's default transport, which reads `HTTPS_PROXY`, and the runner sets the proxy variables when it has a proxy (`package-env.js`). Behind a proxy, `package run`'s own requests would succeed while the package's `cc-data` calls failed. They are now on the passthrough list. -#### RESOLVED: Run output written inside a git checkout -`out/display.md` is derived from student data and lands inside the package directory. cc-data-studies' `.gitignore` ignores only `local-data/`, `__pycache__/` and `*.pyc` (read 2026-10-07), so `git add -A` would stage it. Fixed in R11 and R20: each working directory gets a `.gitignore` of `*`. +--- -#### RESOLVED: `_run_id` reads like run scoping and is not -The answers store stamps `_run_id` with the run that fetched each stored version (`internal/store/segment.go` `AppendPage`), and a later run's re-fetch overwrites it. Next to `run_answers.run_id`, a researcher could filter `answers` by `_run_id` and silently undercount. Fixed in R5's documentation requirement. +### "No extra fields" in the zip was untrue +**Context**: Raised in self-review (Senior Engineer). -### Package Author +**Decision**: `archive/zip` writes `Modified` as an extended-timestamp extra field (`5554...` on every entry of a scratch build). It is as fixed as the time, so R18 holds; the comment and R18 now say so. -#### RESOLVED: The default build output would be swept into the next build -With the output in the current directory and `build` run from inside the package, as authors do, R20 as written either refused its own default or zipped the last build into the next. Fixed in R20: the default is `dir/.cc-data-build/`, which R17 already excludes. +--- -#### RESOLVED: An `init` stub that only reads existing runs reports nothing on the VM -On the VM a package's dataset (`pkg-`) starts empty, and section 13 makes run creation package code. A stub that assumed a pulled run would pass locally and count zero on its first VM run. Fixed in R9: the stub reuses or creates its own run, as `_lib/ccdata.py` and RD-4's smoke package do (a Student ID Mapping run since 2026-10-08). +### `publish` kept a local copy of the origin rule +**Context**: Raised in self-review (One Source of Truth). -### QA Engineer +**Decision**: `publish` refused an `--origin` without the `projects/` prefix, `build` did not, and report-server answers the same case with 422 "origin must be projects/; a user origin is always your own". The local check is deleted, so both commands give report-server's answer. -#### RESOLVED: R3 could pass with a fixture that cannot tell the queries apart -With one assignment per learner, counting endpoints and counting users give the same number, so a broken learner view would pass. Fixed in R3: the synthetic dataset includes a two-assignment learner, a bare endpoint, an answer in two runs and a second class's run. +--- -### Operator +### Assumptions the second self-review checked and kept +**Context**: Raised in self-review (second round), each checked against the code or the live sources. -#### RESOLVED: Staging's answer before `PackageBuckets` exists was unstated -`Store.bucket_for` returns 422 "publishing is not configured for " after every archive and manifest check has passed. Recorded in Technical Notes, so that answer reads as a well-formed request rather than a failure of this story. +**Decision**: Found sound: the R3 test (dropping `m.run_id = ` fails it at run 800, 2 learners against 1); a second `get answers` on a fetched run re-fetches and merges new answers, so the stub stays current; `reports list --json`'s `report_filter` carries `filters` and `class` as the stub reads them; `query --format json` answers an array of objects; report-server's 270-second wait sits between the deriver's 240 and cc-data's 300; an old report-server answers both new routes with a JSON 404 even without a token; and no publish check refuses an ordinary user's own origin, so `run`'s validate call cannot lock out a researcher who may use the API. diff --git a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md b/specs/REPORT-146-package-commands-and-dataset-views/implementation.md deleted file mode 100644 index f0795ee..0000000 --- a/specs/REPORT-146-package-commands-and-dataset-views/implementation.md +++ /dev/null @@ -1,2038 +0,0 @@ -# Implementation Plan: cc-data-cli: the package commands and the dataset views that replace `_lib` - -**Jira**: https://concord-consortium.atlassian.net/browse/REPORT-146 -**Requirements Spec**: [requirements.md](requirements.md) -**Status**: **In Development** - -## Implementation Plan - -The plan comes in seven commits. Each one builds and passes `go test ./...` on its own, in this order. All of the code below was compiled and tested in a scratch copy of `main` (`58c9172`) while this plan was written: the full suite passes, `go vet` is clean for Linux, and `GOOS=windows go vet` is clean for `internal/packages` and `internal/api`. A table under each step names mutations its tests catch; every one listed was applied and made a test fail. The scratch copy has been deleted. - -The package rules and applicability are report-server's (requirements, Dependencies). cc-data calls `POST /api/v1/packages/validate` and `POST /api/v1/packages/applies`, which REPORT-167 adds. A 404 from either means an older report-server, and is a warning, not a failure. So this plan ships and works before that story lands, and gains its checks when it does. - -### The run-scoped view `run_answers`, and the documented counts - -**Summary**: R1, R3 to R5. Adds the static view `run_answers` to `internal/duck/views.go`, and documents it with the learner count through the existing `student_id_mapping` view and the log-freshness query, where the drift guards check. It tests both counts on a dataset where each wrong way of counting gives a different number. Static views are derived everywhere else, so `cmd/query.go`'s help and the MCP `query` description pick it up with no edit. - -**Files affected**: -- `internal/duck/views.go`: one builder, registered after `runMembershipView`. -- `internal/duck/run_views_test.go`: new. -- `internal/guidance/src/core.md`: the `run_answers` entry, the learner count, and the freshness query. -- `docs/researcher-guide.md`: one table row. - -**Estimated diff size**: ~170 lines - -```diff ---- /tmp/o -+++ internal/duck/views.go -@@ -71,6 +71,7 @@ func (vs viewSet) statements() []viewStmt { - stmts = append(stmts, vs.storeView(store.TypeAnswers)) - stmts = append(stmts, vs.storeView(store.TypeHistory)) - stmts = append(stmts, vs.runMembershipView()) -+ stmts = append(stmts, vs.runAnswersView()) - stmts = append(stmts, vs.downloadsView()) - stmts = append(stmts, vs.attachmentFilesView()) - stmts = append(stmts, vs.attachmentStatesView()) -@@ -360,6 +361,19 @@ func (vs viewSet) runMembershipView() viewStmt { - return viewStmt{name: name, primary: primary, fallback: fallback, files: files} - } - -+// runAnswersView is answers with the run_id of each answers membership. Always bound, unlike -+// answers_; it must follow both views it reads and declares no files, so materialize reads -+// it through theirs. -+func (vs viewSet) runAnswersView() viewStmt { -+ name := vs.prefix + `"run_answers"` -+ answers := vs.prefix + sqlIdent(store.TypeAnswers) -+ membership := vs.prefix + `"run_membership"` -+ primary := fmt.Sprintf("CREATE VIEW %s AS SELECT CAST(m.run_id AS BIGINT) AS run_id, s.* FROM %s s JOIN %s m USING (source_key, remote_endpoint, question_id) WHERE m.type = %s", -+ name, answers, membership, sqlStr(store.TypeAnswers)) -+ fallback := fmt.Sprintf("CREATE VIEW %s AS SELECT CAST(NULL AS BIGINT) AS run_id, s.* FROM %s s WHERE false", name, answers) -+ return viewStmt{name: name, primary: primary, fallback: fallback} -+} -+ - // downloadsView is a VALUES dimension table from the manifest. - // - // hide_names separates the two meanings of a name column: where a run hid names, student_name -``` - -`run_answers` declares no files, so `dataset materialize` leaves it a join over the materialized `answers` and `run_membership` (checked against a copy of `staging-smoketest`). The learner count is tested here even though it adds no view, because the guidance presents it as the way to count, and a guidance query that silently stopped counting correctly is the failure this test exists to catch. - -```go -package duck - -import ( - "context" - "encoding/json" - "fmt" - "testing" - "time" - - "github.com/concord-consortium/cc-data-cli/internal/store" -) - -// The documented learner count, run-scoped on both sides, as the guidance gives it. -const learnersOfRun = "SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = %d AND m.run_id = %d" - -// Each wrong way to count gives a different number here: counting endpoints instead of users -// (user 1 answered in two assignments), joining the bare-endpoint learner (who would borrow -// another bare answer), counting an answer once instead of once per run (e3/q1 is in runs 700 -// and 800), and joining membership untyped (run 700's history holds e1a/q1 too). -func TestRunAnswersAndTheDocumentedLearnerCount(t *testing.T) { - d := newDS(t, "ds") - buildStore(t, d, 700, [][]byte{ - answerRec("s", "https://p/d/e1a", "q1", "a"), answerRec("s", "https://p/d/e1b", "q1", "b"), - answerRec("s", "https://p/d/e3", "q1", "c"), answerRec("s", "https://p/d/", "q1", "bare"), - }) - hist, _ := json.Marshal(map[string]any{"source_key": "s", "remote_endpoint": "https://p/d/e1a", "question_id": "q1", "history_id": "h1"}) - seg := store.OpenSegment(d.Dir, store.TypeHistory, 700) - if err := seg.AppendPage([][]byte{hist}, time.Unix(700, 0).UTC(), 700); err != nil { - t.Fatal(err) - } - seg.WriteCursor(&store.Cursor{Items: 1}) - if _, err := d.MergeCompact(store.TypeHistory, 700, seg); err != nil { - t.Fatal(err) - } - buildStore(t, d, 800, [][]byte{answerRec("s", "https://p/d/e3", "q1", "c"), answerRec("s", "https://p/d/e9", "q1", "z")}) - row := func(learner, user, class, offering int, endpoint string) string { - return fmt.Sprintf("%d,%d,%d,s%d,%d,%d,https://act/1,%s", learner, user, user, user, class, offering, endpoint) - } - addDimensionCSV(t, d, dimFixture{run: 700, slug: "student-id-mapping", fetchedAt: at(1), filter: `{}`, csv: mappingCSV( - row(101, 1, 1, 70, "https://p/d/e1a"), row(102, 1, 1, 71, "https://p/d/e1b"), - row(103, 3, 1, 70, "https://p/d/e3"), row(104, 4, 1, 70, "https://p/d/"))}) - addDimensionCSV(t, d, dimFixture{run: 800, slug: "student-id-mapping", fetchedAt: at(2), filter: `{}`, csv: mappingCSV( - row(109, 9, 2, 90, "https://p/d/e9"))}) - e := openEngine(t, []DatasetSpec{{DS: d}}, nil) - - if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 700"); n != 4 { - t.Errorf("run 700 answers = %d, want 4", n) - } - if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 800"); n != 2 { - t.Errorf("run 800 answers = %d, want 2", n) - } - if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 700, 700)); n != 2 { - t.Errorf("run 700 learners = %d, want 2 (users 1 and 3; user 4's endpoint is bare)", n) - } - if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 800, 800)); n != 1 { - t.Errorf("run 800 learners = %d, want 1", n) - } -} - -func TestRunAnswersIsEmptyNotMissingOnAFreshDataset(t *testing.T) { - e := openEngine(t, []DatasetSpec{{DS: newDS(t, "ds")}}, nil) - if n := queryInt(t, e, "SELECT count(*) FROM run_answers"); n != 0 { - t.Errorf("run_answers has %d rows", n) - } - if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 1, 1)); n != 0 { - t.Errorf("the learner count binds to %d on a fresh dataset, want 0", n) - } -} - -func TestRunAnswersRunIDIsBigint(t *testing.T) { - d := newDS(t, "ds") - buildStore(t, d, 700, [][]byte{answerRec("s", "https://p/d/e1", "q1", "a")}) - for name, e := range map[string]*Engine{ - "with membership": openEngine(t, []DatasetSpec{{DS: d}}, nil), - "fresh": openEngine(t, []DatasetSpec{{DS: newDS(t, "fresh")}}, nil), - } { - rows, err := e.Query(context.Background(), "SELECT data_type FROM information_schema.columns WHERE table_name = 'run_answers' AND column_name = 'run_id'") - if err != nil { - t.Fatalf("%s: %v", name, err) - } - var typ string - if rows.Next() { - rows.Scan(&typ) - } - rows.Close() - if typ != "BIGINT" { - t.Errorf("%s: run_id is %s, want BIGINT", name, typ) - } - } -} -``` - -| Mutation | Test that fails | -|---|---| -| `m.type = 'answers'` dropped | run 700 answers = 5 (the history membership joins) | -| the documented count by endpoint instead of `user_id` | run 700 learners = 3 | -| `run_id` taken from membership without the cast | `TestRunAnswersRunIDIsBigint` (INTEGER) | - -Documentation (the two guards in `internal/guidance/guard_test.go` fail until both files carry the name; the dashes in `core.md` match that list's existing entries): - -```diff ---- /tmp/o 2026-10-08 09:00:24.902084889 -0400 -+++ internal/guidance/src/core.md 2026-10-08 09:00:14.224676445 -0400 -@@ -122,6 +122,24 @@ - **type-qualified**: `answers a JOIN run_membership m USING - (source_key, remote_endpoint, question_id) WHERE m.run_id = 584 AND m.type = - 'answers'`. History joins add `history_id` to the USING list. -+- `run_answers` — every `answers` row with `run_id`, once for each run whose -+ answers fetch holds it (a Student Answers or a Student ID Mapping run), so one -+ run's answers are `SELECT count(*) FROM -+ run_answers WHERE run_id = 584`. It is the type-qualified membership join above, -+ always present, so a run with no answers counts zero rather than failing to bind -+ as `answers_` does. Its `run_id` is membership; the store's own `_run_id` is -+ only the run that last fetched the record, so filtering `answers` by `_run_id` -+ undercounts a run whose answers a later run re-fetched. -+- A run's learners with at least one answer come from its Student ID Mapping -+ report: `SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN -+ run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = 584 -+ AND m.run_id = 584`, where 584 is a Student ID Mapping run whose answers were -+ fetched. Count by `user_id`, never by endpoint, since a student has one endpoint -+ per assignment. A Student ID Mapping run is computed live, so re-reading it with -+ `--refresh` picks up learners who joined since; a Student Answers run is fixed -+ when its query ran, and a whole-class one fails above three or four assignments. -+- A run's log freshness is `SELECT count(*) AS logs, max(event_time) AS -+ log_freshness_at FROM logs WHERE run_id = `. - - Reports-to-stores join: `reports.res__remote_endpoint = - answers.remote_endpoint`, with `res___*` pairing to - `answers.question_id`. ---- /tmp/o 2026-10-08 09:00:24.914084903 -0400 -+++ docs/researcher-guide.md 2026-10-08 08:57:22.784965017 -0400 -@@ -346,6 +346,7 @@ - | `attachment_content` | The text/JSON content of every saved CODAP/SageModeler snapshot, queryable and diffable. | - | `student_id_mapping` | One row per learner from Student ID Mapping runs, with the ids that join them to their answers and history. | - | `student_metadata` | One row per learner from Student Metadata runs: name, username, class, school, teachers, permission forms. | -+| `run_answers` | Every answer with the `run_id` of each run that holds it: `SELECT count(*) FROM run_answers WHERE run_id = 584` counts one run's answers. Join it to `student_id_mapping` on `remote_endpoint = run_remote_endpoint` to count learners by `user_id`. | - | `run_membership`, `downloads` | Provenance: which run's fetch covered which records, and what each download was, including whether its run hid names. | - - ### Speeding up a large dataset -``` - ---- - -### A package's files: the manifest fields, collecting and zipping, the result - -**Summary**: R6 (cc-data's part), R8, R14, R17, R18. Adds `internal/packages`: -- **Reading the manifest.** It reads only the fields the runner acts on. -- **Collecting the package.** It gathers the file set `build` ships and `run` stages, each file with the mode it ships with, so an executable entrypoint stays executable after `unzip`. -- **Zipping.** It writes a reproducible zip. -- **Reading the result.** It reads `display.md`, `summary.txt` and `counts.json` as the runner does. - -Links are checked with `Lstat` then `os.SameFile` rather than `O_NOFOLLOW`, which Windows (CI only) lacks. The package has no callers yet and is independently reviewable. - -**Files affected**: -- `internal/packages/manifest.go`: new. -- `internal/packages/archive.go`: new. -- `internal/packages/result.go`: new. -- `internal/packages/files_test.go`: new; holds the shared helpers `goodManifest` and `writeTree`. - -**Estimated diff size**: ~430 lines - -```go -// Package packages is the local side of a Researcher Dashboard package: reading the manifest -// fields the runner acts on, collecting and zipping the package, staging and running it the -// way the runner does, and reading the result files. The package rules themselves, and -// applicability, are report-server's, which cc-data asks rather than copies. -package packages - -import ( - "encoding/json" - "fmt" - "regexp" - "strings" - "unicode" -) - -var nameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,62}$`) - -// ValidName reports whether name fits the catalog's name grammar, which init needs before -// report-server has seen anything. -func ValidName(name string) bool { return nameRe.MatchString(name) } - -// URLs is a manifest's applicability patterns, passed to report-server as written. -type URLs struct { - All []string `json:"all"` - Any []string `json:"any"` - None []string `json:"none"` -} - -// DeclaresPatterns reports whether any sense holds a pattern; a package that declares none -// applies to every scope. -func (u URLs) DeclaresPatterns() bool { - return len(u.All)+len(u.Any)+len(u.None) > 0 -} - -// Manifest is the part of manifest.json the runner acts on. Everything else, and every limit, -// is report-server's to check (cmd's validate call). -type Manifest struct { - Name string `json:"name"` - Title string `json:"title"` - Version string `json:"version"` - Description string `json:"description,omitempty"` - URLs URLs `json:"urls"` - CluePrepull bool `json:"clue_prepull"` - Entrypoint string `json:"entrypoint"` - ExpectedDurationSeconds int `json:"expected_duration_seconds"` -} - -// ReadManifest reads the fields a run acts on, refusing an entrypoint outside files and a -// duration that is not a positive integer; every other rule is report-server's. -func ReadManifest(data []byte, files map[string]bool) (Manifest, error) { - var m Manifest - if err := json.Unmarshal(data, &m); err != nil { - return m, fmt.Errorf("manifest.json: %v", err) - } - if !files[m.Entrypoint] || unsafePath(m.Entrypoint) { - return m, fmt.Errorf("manifest.json: entrypoint %q is not a file in the package", m.Entrypoint) - } - if m.ExpectedDurationSeconds < 1 { - return m, fmt.Errorf("manifest.json: expected_duration_seconds must be a positive integer") - } - return m, nil -} - -// unsafePath is report-server's Archive.unsafe_path?: absolute, a drive letter, or a ".." segment. -func unsafePath(name string) bool { - if strings.HasPrefix(name, "/") || strings.HasPrefix(name, `\`) { - return true - } - if len(name) >= 2 && name[1] == ':' && unicode.IsLetter(rune(name[0])) { - return true - } - for _, seg := range strings.FieldsFunc(name, func(r rune) bool { return r == '/' || r == '\\' }) { - if seg == ".." { - return true - } - } - return false -} -``` -```go -package packages - -import ( - "archive/zip" - "bytes" - "crypto/sha256" - "encoding/hex" - "fmt" - "io" - "io/fs" - "os" - "path" - "path/filepath" - "runtime" - "sort" - "strings" - "time" -) - -// buildTime is every entry's modification time, so the same tree always zips to the same -// bytes and the same checksum. -var buildTime = time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC) - -// Files is a package's contents: slash-separated paths relative to its directory, sorted, -// each with the mode it ships with. -type Files struct { - Dir string - Paths []string - Modes map[string]os.FileMode - Bytes int64 -} - -// shipMode is the one mode a file keeps through build and unzip: 0755 when any execute bit is -// set, else 0644. unzip restores it on the VM, where a non-.py entrypoint is executed directly. -func shipMode(m fs.FileMode) os.FileMode { - if m.Perm()&0o111 != 0 { - return 0o755 - } - return 0o644 -} - -// Set is the paths as a lookup, the shape ReadManifest checks the entrypoint against. -func (f Files) Set() map[string]bool { - set := make(map[string]bool, len(f.Paths)) - for _, p := range f.Paths { - set[p] = true - } - return set -} - -// excluded is what a build leaves out: any dot path (.git, .cc-data-run, .cc-data-build), -// __pycache__, *.pyc and local-data. -func excluded(rel string, isDir bool) bool { - base := path.Base(rel) - if strings.HasPrefix(base, ".") || base == "__pycache__" || (isDir && base == "local-data") { - return true - } - return !isDir && strings.HasSuffix(base, ".pyc") -} - -// Collect walks dir for the files a package is made of. A link or other non-regular file is -// refused rather than skipped, since the runner refuses an archive holding one. -func Collect(dir string) (Files, error) { - files := Files{Dir: dir, Modes: map[string]os.FileMode{}} - err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error { - if err != nil { - return err - } - if p == dir { - return nil - } - rel, err := filepath.Rel(dir, p) - if err != nil { - return err - } - rel = filepath.ToSlash(rel) - if excluded(rel, d.IsDir()) { - if d.IsDir() { - return filepath.SkipDir - } - return nil - } - switch { - case d.Type()&fs.ModeSymlink != 0: - return fmt.Errorf("%s is a symbolic link; a package may hold only regular files", rel) - case d.IsDir(): - return nil - case !d.Type().IsRegular(): - return fmt.Errorf("%s is not a regular file", rel) - } - info, err := d.Info() - if err != nil { - return err - } - files.Paths = append(files.Paths, rel) - files.Modes[rel] = shipMode(info.Mode()) - files.Bytes += info.Size() - return nil - }) - sort.Strings(files.Paths) - return files, err -} - -// Zip writes a reproducible archive: sorted entries, Deflate, a fixed time and each file's ship -// mode. The size limits are report-server's. -func Zip(files Files) ([]byte, error) { - var buf bytes.Buffer - w := zip.NewWriter(&buf) - for _, rel := range files.Paths { - h := &zip.FileHeader{Name: rel, Method: zip.Deflate, Modified: buildTime} - h.SetMode(files.Modes[rel]) - dst, err := w.CreateHeader(h) - if err != nil { - return nil, err - } - src, err := os.Open(filepath.Join(files.Dir, filepath.FromSlash(rel))) - if err != nil { - return nil, err - } - _, err = io.Copy(dst, src) - src.Close() - if err != nil { - return nil, err - } - } - if err := w.Close(); err != nil { - return nil, err - } - return buf.Bytes(), nil -} - -// CheckEntrypointMode refuses a non-.py entrypoint that would not be executable once unzipped, -// since the runner executes it directly. Windows records no execute bit to check. -func (f Files) CheckEntrypointMode(entrypoint string) error { - if runtime.GOOS == "windows" || strings.HasSuffix(entrypoint, ".py") || f.Modes[entrypoint]&0o111 != 0 { - return nil - } - return fmt.Errorf("the entrypoint %s is not executable; the runner executes a non-.py entrypoint directly (chmod +x it)", entrypoint) -} - -// Checksum is the catalog's spelling of an archive's digest. -func Checksum(b []byte) string { - sum := sha256.Sum256(b) - return "sha256:" + hex.EncodeToString(sum[:]) -} -``` -```go -package packages - -import ( - "encoding/json" - "errors" - "fmt" - "io" - "io/fs" - "os" - "path/filepath" - "strings" -) - -// DisplayLimitBytes is the runner's cap on display.md in UTF-8 bytes, decided only in -// final-design.md section 10. -const DisplayLimitBytes = 64 * 1024 - -// countKeys are the counts.json keys the runner reads; anything else a package writes there -// is ignored, as it is on the VM. -var countKeys = []string{"answers", "logs", "log_freshness_at"} - -// Result is what the runner keeps of a run: the display, the summary line and the counts. -type Result struct { - DisplayPath string `json:"display"` - DisplayBytes int64 `json:"display_bytes"` - Summary *string `json:"summary"` - Counts map[string]any `json:"counts"` -} - -// OutputRefused is a result the runner would refuse: a missing, linked or oversized -// display.md, a linked summary.txt, or a linked output directory. -type OutputRefused struct{ Reason string } - -func (e *OutputRefused) Error() string { return e.Reason } - -// ReadResult reads the output directory as the runner does, following no link: the -// directory must be itself, and each file must be a regular file. -func ReadResult(outDir string) (Result, error) { - var res Result - if err := RealDir(outDir); err != nil { - return res, &OutputRefused{Reason: "the package replaced its output directory with a link"} - } - _, size, err := readOwnFile(outDir, "display.md", DisplayLimitBytes) - switch { - case errors.Is(err, fs.ErrNotExist): - return res, &OutputRefused{Reason: "the package wrote no display.md"} - case err != nil: - return res, err - case size > DisplayLimitBytes: - return res, &OutputRefused{Reason: fmt.Sprintf("the package's display.md is %d bytes, over the %d-byte limit", size, DisplayLimitBytes)} - } - res.DisplayPath = filepath.Join(outDir, "display.md") - res.DisplayBytes = size - if text, _, err := readOwnFile(outDir, "summary.txt", DisplayLimitBytes); err == nil { - line := strings.TrimSpace(strings.SplitN(strings.ReplaceAll(text, "\r\n", "\n"), "\n", 2)[0]) - if line != "" { - res.Summary = &line - } - } else if !errors.Is(err, fs.ErrNotExist) { - return res, err - } - res.Counts = map[string]any{} - if text, _, err := readOwnFile(outDir, "counts.json", DisplayLimitBytes); err == nil { - var parsed map[string]any - if json.Unmarshal([]byte(text), &parsed) == nil { - for _, k := range countKeys { - if v, ok := parsed[k]; ok { - res.Counts[k] = v - } - } - } - } - return res, nil -} - -// readOwnFile returns up to limit+1 bytes of a regular, unlinked file and its full size. The -// Lstat-then-SameFile pair stands in for O_NOFOLLOW, which Windows lacks. -func readOwnFile(dir, name string, limit int64) (string, int64, error) { - p := filepath.Join(dir, name) - before, err := os.Lstat(p) - if err != nil { - return "", 0, err - } - if before.Mode()&fs.ModeSymlink != 0 { - return "", 0, &OutputRefused{Reason: fmt.Sprintf("the package's %s is a link, not a file", name)} - } - if !before.Mode().IsRegular() { - return "", 0, &OutputRefused{Reason: fmt.Sprintf("the package's %s is not a file", name)} - } - f, err := os.Open(p) - if err != nil { - return "", 0, err - } - defer f.Close() - after, err := f.Stat() - if err != nil || !os.SameFile(before, after) { - return "", 0, &OutputRefused{Reason: fmt.Sprintf("the package's %s changed while it was read", name)} - } - b, err := io.ReadAll(io.LimitReader(f, limit+1)) - return string(b), after.Size(), err -} - -// RealDir reports an error unless p is a directory reached without following a link. -func RealDir(p string) error { - info, err := os.Lstat(p) - if err != nil { - return err - } - if info.Mode()&fs.ModeSymlink != 0 || !info.IsDir() { - return fmt.Errorf("%s is a link or not a directory", p) - } - return nil -} -``` - -Tests (the link and mode tests skip on Windows, where links need privileges and files have no execute bit): -- **`TestReadManifestNeedsWhatTheRunActsOn`**: a missing entrypoint is refused, and so are `"300"`, `1.5` and `0` for the duration. -- **`TestCollectLeavesOutWhatBuildMustNotShip`**: the tree holds `.git/`, `.cc-data-run/`, `.cc-data-build/`, `__pycache__/cache.txt`, `lib/y.pyc`, `local-data/` and `.DS_Store`. Exactly `lib/q.sql,manifest.json,run.py` is collected. `__pycache__` holds a non-`.pyc` file so the directory rule is tested on its own. -- **`TestCollectRefusesALink`**. -- **`TestZipIsReproducible`**: a touched mtime leaves the checksum unchanged. -- **`TestModesSurviveTheZip`**: `run.py` (0755 on disk) zips as 0755 and a 0600 `manifest.json` as 0644. A 0644 `run.sh` entrypoint fails `CheckEntrypointMode`, and a 0644 `run.py` passes it. -- **`TestDisplayCapEdges`**: 65,536 bytes pass and 65,537 are refused, and the constant equals 65,536. -- **`TestResultReadsOnlyTheRunnersKeys`**: the summary is the first line, trimmed, and `learners` in `counts.json` is dropped. -- **`TestResultRefusesALinkedDisplay`**. - -| Mutation | Test that fails | -|---|---| -| `size >= DisplayLimitBytes` | `TestDisplayCapEdges` | -| `__pycache__` not excluded | `TestCollectLeavesOutWhatBuildMustNotShip` | -| every entry zipped 0644 | `TestModesSurviveTheZip` | -| a duration of 0 accepted | `TestReadManifestNeedsWhatTheRunActsOn` | - ---- - -### `package run`'s engine: scope, layout, environment, execution - -**Summary**: R10 to R16. -- **Scope.** Parses the author's scope file. -- **Layout.** Rebuilds `.cc-data-run/` (`pkg/`, `in/`, `out/`, `data/` and a `.gitignore` of `*`) and writes the runner's `scope.json`. -- **Applicability.** Asks an injected `Applies` function, which the commands step wires to report-server. -- **Environment and interpreter.** Builds the runner's environment plus the passthrough list, and picks the interpreter. -- **Execution.** Runs the entrypoint from the staged copy in its own process group, kills the group at the bound or on an interrupt, and kills what is left of it once the entrypoint exits. -- **Result.** Reads it with `ReadResult`. - -**Files affected**: -- `internal/packages/scope.go`: new. -- `internal/packages/run.go`: new. -- `internal/packages/proc_unix.go` and `proc_windows.go`: new. -- `internal/packages/run_test.go`: new; holds `scopeJSON`, `runOpts` and `shellPackage`. - -**Estimated diff size**: ~520 lines - -```go -package packages - -import ( - "bytes" - "encoding/json" - "fmt" - "regexp" -) - -var classHashRe = regexp.MustCompile(`^[0-9a-f]{48}$`) - -// Class and Assignment are scope.json's entries. -type Class struct { - ClassHash string `json:"class_hash"` - ClassID int64 `json:"class_id"` -} - -type Assignment struct { - OfferingID int64 `json:"offering_id"` - RunnableID int64 `json:"runnable_id"` - Name *string `json:"name"` - URL string `json:"url"` -} - -// LocalScope is the four keys rigse supplies on the VM, which an author writes for a laptop. -type LocalScope struct { - Kind string `json:"kind"` - ID string `json:"id"` - Classes []Class `json:"classes"` - Assignments []Assignment `json:"assignments"` -} - -// ScopeFile is scope.json as the runner writes it: ClueSource is its constant "firebase", and -// Dataset the bare name, as it writes pkg-, while RD_DATASET carries the full ref. -type ScopeFile struct { - LocalScope - ClueSource string `json:"clue_source"` - Dataset string `json:"dataset"` - OutputDir string `json:"output_dir"` -} - -// MarshalJSON keeps the runner's key order, so a diff between a local and a VM scope.json is -// only the values. -func (s ScopeFile) MarshalJSON() ([]byte, error) { - type ordered struct { - Kind string `json:"kind"` - ID string `json:"id"` - Classes []Class `json:"classes"` - Assignments []Assignment `json:"assignments"` - ClueSource string `json:"clue_source"` - Dataset string `json:"dataset"` - OutputDir string `json:"output_dir"` - } - return json.Marshal(ordered{s.Kind, s.ID, s.Classes, s.Assignments, s.ClueSource, s.Dataset, s.OutputDir}) -} - -// ParseLocalScope reads an author's scope file, refusing any key the four do not name (the -// other three are the runner's to set) and any shape the runner would never send. -func ParseLocalScope(data []byte) (LocalScope, error) { - var s LocalScope - dec := json.NewDecoder(bytes.NewReader(data)) - dec.DisallowUnknownFields() - if err := dec.Decode(&s); err != nil { - return s, fmt.Errorf("scope file: %v (it holds exactly kind, id, classes and assignments)", err) - } - switch { - case s.Kind == "": - return s, fmt.Errorf("scope file: kind is required") - case s.ID == "": - return s, fmt.Errorf("scope file: id is required") - case len(s.Classes) == 0: - return s, fmt.Errorf("scope file: classes must name at least one class") - case s.Assignments == nil: - return s, fmt.Errorf("scope file: assignments is required (an empty list is allowed)") - } - for i, c := range s.Classes { - if !classHashRe.MatchString(c.ClassHash) || c.ClassID < 1 { - return s, fmt.Errorf("scope file: classes[%d] needs a 48-hex class_hash and a positive class_id", i) - } - } - for i, a := range s.Assignments { - if a.OfferingID < 1 || a.RunnableID < 1 { - return s, fmt.Errorf("scope file: assignments[%d] needs a positive offering_id and runnable_id", i) - } - } - return s, nil -} - -// AssignmentURLs is the scope's half of the URLs a package's patterns are matched against. -func (s LocalScope) AssignmentURLs() []string { - var urls []string - for _, a := range s.Assignments { - if a.URL != "" { - urls = append(urls, a.URL) - } - } - return urls -} -``` -```go -package packages - -import ( - "context" - "encoding/json" - "errors" - "fmt" - "io" - "os" - "os/exec" - "path/filepath" - "slices" - "sort" - "strings" - "time" -) - -// TimeoutMargin is what the runner adds to expected_duration_seconds before it kills a run. -const TimeoutMargin = 10 * time.Minute - -// RunDirName is the working tree package run keeps inside the package directory. It starts -// with "." so Collect, and so build, never ships it. -const RunDirName = ".cc-data-run" - -// passthrough is what a laptop adds so the package's own cc-data calls find the stored -// credential, reach the server through any proxy, and run at all. Matching is case-insensitive. -var passthrough = []string{ - "HOME", "PATH", "USER", "LOGNAME", "LANG", "TZ", "TMPDIR", - "DBUS_SESSION_BUS_ADDRESS", "XDG_RUNTIME_DIR", - "HTTPS_PROXY", "HTTP_PROXY", "NO_PROXY", - // Windows has no release, but CI builds and tests there. - "SYSTEMROOT", "USERPROFILE", "APPDATA", "LOCALAPPDATA", "PATHEXT", "COMSPEC", -} - -// Verdict is report-server's answer to whether a package's patterns match a scope. Unconfirmed -// means the report-server answering has no applicability route yet, so nothing was checked. -type Verdict struct { - Applies bool - Reason string - Unconfirmed bool -} - -// Refused is a package the runner would not start: it does not apply, or it needs the CLUE -// pre-pull, which cc-data cannot do. -type Refused struct{ Reason string } - -func (e *Refused) Error() string { return e.Reason } - -// Failed is a package that started and did not finish cleanly. -type Failed struct{ Reason string } - -func (e *Failed) Error() string { return e.Reason } - -// RunOptions is everything package run decides before the entrypoint starts. -type RunOptions struct { - Dir string // the package directory - Manifest Manifest // validated against Files - Files Files // what build would ship - Scope LocalScope // the author's scope file - Applies func(context.Context, URLs) (Verdict, error) // report-server's applicability answer - Dataset string // the full ref, / - Name string // the ref's bare name - Portal string // the ref's portal host - DataRoot string // CC_DATA_ROOT as package run resolved it - ReportSrv string // the stored credential's server, or "" - Stderr io.Writer // where the package's own output goes - Environ func() []string - LookPath func(string) (string, error) - Timeout time.Duration // zero means expected_duration_seconds + TimeoutMargin -} - -// Run prepares the runner's layout, runs the entrypoint and reads its result. -func Run(ctx context.Context, o RunOptions) (Result, error) { - if o.Manifest.CluePrepull { - return Result{}, &Refused{Reason: "the package sets clue_prepull, and cc-data cannot fetch CLUE data yet, so it runs only on the VM"} - } - if o.Manifest.URLs.DeclaresPatterns() { - v, err := o.Applies(ctx, o.Manifest.URLs) - switch { - case err != nil: - return Result{}, err - case v.Unconfirmed: - fmt.Fprintln(o.Stderr, "warning: report-server cannot check applicability yet; the package runs, and the VM's runner is the check") - case !v.Applies: - return Result{}, &Refused{Reason: "the package does not apply to this class: " + v.Reason} - } - } - run := filepath.Join(o.Dir, RunDirName) - paths, err := prepare(run, o.Files) - if err != nil { - return Result{}, err - } - scope := ScopeFile{LocalScope: o.Scope, ClueSource: "firebase", Dataset: o.Name, OutputDir: paths.out} - b, err := json.MarshalIndent(scope, "", " ") - if err != nil { - return Result{}, err - } - scopePath := filepath.Join(paths.in, "scope.json") - if err := os.WriteFile(scopePath, b, 0o644); err != nil { - return Result{}, err - } - - env := map[string]string{ - "CC_DATA_ROOT": o.DataRoot, - "CC_DATA_PORTAL": o.Portal, - "CC_DATA_LOCAL": paths.data, - "RD_DATASET": o.Dataset, - "RD_SCOPE_FILE": scopePath, - "RD_DATA_DIR": paths.data, - "RD_OUTPUT_DIR": paths.out, - } - if o.ReportSrv != "" { - env["RD_REPORT_SERVER_URL"] = o.ReportSrv - } - command, args, note, err := entrypointCommand(paths.pkg, o.Manifest.Entrypoint, o.LookPath) - if err != nil { - return Result{}, err - } - if note != "" { - fmt.Fprintln(o.Stderr, note) - } - - timeout := o.Timeout - if timeout == 0 { - timeout = time.Duration(o.Manifest.ExpectedDurationSeconds)*time.Second + TimeoutMargin - } - runCtx, cancel := context.WithTimeout(ctx, timeout) - defer cancel() - cmd := exec.CommandContext(runCtx, command, args...) - cmd.Dir = paths.pkg - cmd.Env = environment(o.Environ(), env) - cmd.Stdout = o.Stderr - cmd.Stderr = o.Stderr - killGroup(cmd) - err = cmd.Run() - // As on the VM, nothing the package started outlives it or writes after it. - reapGroup(cmd) - if err != nil { - switch { - case errors.Is(runCtx.Err(), context.DeadlineExceeded): - return Result{}, &Failed{Reason: fmt.Sprintf("the package ran past its %ds bound", int(timeout.Seconds()))} - case ctx.Err() != nil: - return Result{}, &Failed{Reason: "the package was interrupted"} - } - return Result{}, &Failed{Reason: fmt.Sprintf("the package exited unsuccessfully: %v", err)} - } - return ReadResult(paths.out) -} - -type runPaths struct{ pkg, in, out, data string } - -// prepare empties and rebuilds the run tree. Every directory it removes or creates under the -// package must be a real directory, so a link left there can never aim the removal outside. -func prepare(run string, files Files) (runPaths, error) { - p := runPaths{ - pkg: filepath.Join(run, "pkg"), - in: filepath.Join(run, "in"), - out: filepath.Join(run, "out"), - data: filepath.Join(run, "data"), - } - if err := ensureRealDir(run); err != nil { - return p, err - } - if err := ignoreAll(run); err != nil { - return p, err - } - for _, d := range []string{p.pkg, p.in, p.out} { - if _, err := os.Lstat(d); err == nil { - if err := RealDir(d); err != nil { - return p, fmt.Errorf("refusing to empty %s: it is a link or not a directory", d) - } - if err := os.RemoveAll(d); err != nil { - return p, err - } - } - if err := os.Mkdir(d, 0o755); err != nil { - return p, err - } - } - if err := ensureRealDir(p.data); err != nil { - return p, err - } - abs, err := filepath.Abs(run) - if err != nil { - return p, err - } - p = runPaths{filepath.Join(abs, "pkg"), filepath.Join(abs, "in"), filepath.Join(abs, "out"), filepath.Join(abs, "data")} - for _, rel := range files.Paths { - if err := copyFile(filepath.Join(files.Dir, filepath.FromSlash(rel)), filepath.Join(p.pkg, filepath.FromSlash(rel)), files.Modes[rel]); err != nil { - return p, err - } - } - return p, nil -} - -// ignoreAll keeps git out of a directory this tool makes inside a package, whose files can -// hold student data. -func ignoreAll(dir string) error { - p := filepath.Join(dir, ".gitignore") - if info, err := os.Lstat(p); err == nil && !info.Mode().IsRegular() { - return fmt.Errorf("refusing to write %s: it is a link or not a file", p) - } - return os.WriteFile(p, []byte("*\n"), 0o644) -} - -func ensureRealDir(d string) error { - if _, err := os.Lstat(d); errors.Is(err, os.ErrNotExist) { - return os.Mkdir(d, 0o755) - } - if err := RealDir(d); err != nil { - return fmt.Errorf("refusing to use %s: it is a link or not a directory", d) - } - return nil -} - -// copyFile stages one file with the mode build gives it, so the staged package is the one -// unzip produces on the VM. -func copyFile(src, dst string, mode os.FileMode) error { - if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { - return err - } - in, err := os.Open(src) - if err != nil { - return err - } - defer in.Close() - out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_EXCL, mode) - if err != nil { - return err - } - if _, err := io.Copy(out, in); err != nil { - out.Close() - return err - } - return out.Close() -} - -// entrypointCommand is the runner's choice of interpreter: python3.11 for a .py entrypoint, -// otherwise the entrypoint itself. A laptop without python3.11 falls back to python3, saying so. -func entrypointCommand(pkgDir, entrypoint string, lookPath func(string) (string, error)) (string, []string, string, error) { - path := filepath.Join(pkgDir, filepath.FromSlash(entrypoint)) - if !strings.HasSuffix(entrypoint, ".py") { - return path, nil, "", nil - } - if p, err := lookPath("python3.11"); err == nil { - return p, []string{path}, "", nil - } - if p, err := lookPath("python3"); err == nil { - return p, []string{path}, "note: python3.11, the VM's interpreter, is not on PATH; running with python3", nil - } - return "", nil, "", fmt.Errorf("neither python3.11 nor python3 is on PATH") -} - -// environment is the passthrough taken from the caller's environment (LC_* included), then -// the runner's variables over it; nothing else is inherited. -func environment(from []string, set map[string]string) []string { - keep := map[string]string{} - for _, kv := range from { - k, v, ok := strings.Cut(kv, "=") - if !ok { - continue - } - if strings.HasPrefix(k, "LC_") || slices.Contains(passthrough, strings.ToUpper(k)) { - keep[k] = v - } - } - for k, v := range set { - keep[k] = v - } - out := make([]string, 0, len(keep)) - for k, v := range keep { - out = append(out, k+"="+v) - } - sort.Strings(out) - return out -} -``` -```go -//go:build !windows - -package packages - -import ( - "os/exec" - "syscall" -) - -// killGroup runs the package in its own process group and kills the whole group when the run -// is canceled or times out, so a child the entrypoint started does not outlive it. -func killGroup(cmd *exec.Cmd) { - cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} - cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) } -} - -// reapGroup kills whatever is left of the package's process group once the entrypoint has -// exited. The group outlives its leader while any member does, so the kill still finds it. -func reapGroup(cmd *exec.Cmd) { - if cmd.Process != nil { - syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) - } -} -``` -```go -//go:build windows - -package packages - -import "os/exec" - -// killGroup and reapGroup kill only the entrypoint on Windows, which has no release. -func killGroup(cmd *exec.Cmd) {} - -func reapGroup(cmd *exec.Cmd) {} -``` - -Tests (the entrypoint is a `/bin/sh` script, so these skip on Windows): -- **`TestLocalScopeRefusesTheRunnersKeys`**: a scope file setting `dataset` is refused. -- **`TestRunGivesThePackageTheRunnersLayout`**: - - The package dumps its environment, copies `scope.json` and lists its working directory. - - The environment carries `RD_DATASET` (the full ref), `CC_DATA_PORTAL`, `RD_SCOPE_FILE` a passed-through `LC_ALL` and lower-case `https_proxy`, and not an injected `AWS_SECRET_ACCESS_KEY`. - - `scope.json` has the bare `dataset`, `clue_source: "firebase"` and the absolute `output_dir`. - - The working directory holds exactly `manifest.json` and `run.sh` (no `local-data/`), the staged `run.sh` keeps mode 0755, and `.gitignore` is `*`. -- **`TestRunRefusesBeforeStarting`**: when `Applies` says no, the error is the runner's prefix plus the reason, and the entrypoint never starts. A `clue_prepull` package is refused. -- **`TestRunWarnsAndRunsWhenApplicabilityIsUnconfirmed`**: an unconfirmed verdict prints the warning and runs. -- **`TestRunKillsAtTheBound`**: `sleep 30 & wait` under a 300 ms bound fails as "past its bound" within 5 seconds. -- **`TestRunLeavesNothingBehind`**: a package that backgrounds `(sleep 1; echo late > "$RD_OUTPUT_DIR/late.txt")` and exits 0 succeeds, and 1.5 seconds later `late.txt` does not exist. -- **`TestRunKillsTheGroupWhenInterrupted`**: the same package, with the parent context canceled after 300 ms, fails as "interrupted" within 5 seconds. -- **`TestRunRefusesALinkedRunDirectory`**: `.cc-data-run` linked to another directory is refused, and a file in that directory survives. -- **`TestRunRefusesALinkedGitignore`**: a `.cc-data-run/.gitignore` linked to another file is refused, and the file keeps its content. -- **`TestEntrypointFallsBackToPython3WithANote`**: `python3.11` when found, else `python3` with a note naming 3.11, else an error. -- **`TestWriteBuildRefusesALinkedBuildDirectory`**: a `.cc-data-build` linked elsewhere is refused, and nothing is written behind it. - -| Mutation | Test that fails | -|---|---| -| no `Setpgid` | `TestRunKillsAtTheBound` (the orphaned `sleep` holds the output pipe for 30 s) | -| link check on the run directory removed | `TestRunRefusesALinkedRunDirectory` | -| the whole environment inherited | `TestRunGivesThePackageTheRunnersLayout` | -| run from the source directory, not the staged copy | `TestRunGivesThePackageTheRunnersLayout` | -| a "does not apply" verdict ignored | `TestRunRefusesBeforeStarting` | -| a canceled context reported as an ordinary failure | `TestRunKillsTheGroupWhenInterrupted` | -| no `reapGroup` after the entrypoint exits | `TestRunLeavesNothingBehind` (the background job writes `late.txt`) | -| an unconfirmed verdict treated as a refusal | `TestRunWarnsAndRunsWhenApplicabilityIsUnconfirmed` | - ---- - -### The API client: publish, validate and applies - -**Summary**: R12, R19, R21 to R23, client side. -- **`send`.** `Client.do` keeps its signature and delegates to a new `send` that names the body's media type; `attempt` takes it as a parameter. -- **`PublishPackage` and `ValidatePackage`** post the raw zip. A POST is already never retried (`isIdempotent`), and `net/http` sets `Content-Length` from the `*bytes.Reader`, which report-server requires. -- **`PackageApplies`** posts JSON. -- **`RouteMissing`** recognizes the 404 an older report-server gives. - -**Files affected**: -- `internal/api/client.go`: `send`, plus a content-type parameter on `attempt`. -- `internal/api/packages.go`: new. - -**Estimated diff size**: ~140 lines; the requests' shapes are tested at the command level below - -```diff ---- /tmp/o 2026-10-08 05:22:45.439149318 -0400 -+++ internal/api/client.go 2026-10-07 17:17:44.313590304 -0400 -@@ -95,6 +95,11 @@ - // do runs a request under the retry policy and returns the 2xx body, or a typed - // *APIError (contract) / *TransientError (budget exhausted). - func (c *Client) do(ctx context.Context, method, path string, query url.Values, body []byte) ([]byte, error) { -+ return c.send(ctx, method, path, query, body, "application/json") -+} -+ -+// send is do with the body's media type named, for the routes that take raw bytes. -+func (c *Client) send(ctx context.Context, method, path string, query url.Values, body []byte, contentType string) ([]byte, error) { - u := c.BaseURL + path - if len(query) > 0 { - u += "?" + query.Encode() -@@ -110,7 +115,7 @@ - return nil, err - } - -- data, status, err := c.attempt(ctx, method, u, body) -+ data, status, err := c.attempt(ctx, method, u, body, contentType) - if err != nil { - last = err - // A non-idempotent request (POST) may have reached the server, so -@@ -139,7 +144,7 @@ - - // attempt performs one HTTP request under a per-attempt deadline and returns the - // body and status; a transport error (including deadline) is returned as err. --func (c *Client) attempt(ctx context.Context, method, u string, body []byte) ([]byte, int, error) { -+func (c *Client) attempt(ctx context.Context, method, u string, body []byte, contentType string) ([]byte, int, error) { - reqCtx := ctx - var cancel context.CancelFunc - if c.RequestTimeout > 0 { -@@ -160,7 +165,7 @@ - } - req.Header.Set("Accept", "application/json") - if body != nil { -- req.Header.Set("Content-Type", "application/json") -+ req.Header.Set("Content-Type", contentType) - } - - resp, err := c.HTTP.Do(req) -``` -```go -package api - -import ( - "context" - "encoding/json" - "errors" - "net/http" - "net/url" -) - -// PublishedPackage is report-server's answer to a publish. -type PublishedPackage struct { - CatalogID int64 `json:"catalog_id"` - Identity string `json:"identity"` - Version string `json:"version"` - Checksum string `json:"checksum"` - Visibility string `json:"visibility"` - Official bool `json:"official"` - CurrentVersion string `json:"current_version"` -} - -// PublishPackage posts a package archive as the raw request body. origin is "" for the -// caller's own users/, or projects/; official asks the server to mark it official, -// which only a publisher may. Like every POST it is never retried. -func (c *Client) PublishPackage(ctx context.Context, archive []byte, origin string, official bool) (*PublishedPackage, error) { - data, err := c.send(ctx, http.MethodPost, "/api/v1/packages", packageQuery(origin, official), archive, "application/zip") - if err != nil { - return nil, err - } - var out PublishedPackage - if err := json.Unmarshal(data, &out); err != nil { - return nil, err - } - return &out, nil -} - -// ValidatedPackage is what a publish of the archive would record. The last two are not -// refusals: each says only a publish would be refused now, with publish's own message for the -// second. -type ValidatedPackage struct { - Identity string `json:"identity"` - Version string `json:"version"` - Checksum string `json:"checksum"` - Visibility string `json:"visibility"` - AlreadyPublished bool `json:"already_published"` - PublishingUnavailable *string `json:"publishing_unavailable"` -} - -// ValidatePackage asks report-server to apply every publish check to the archive and stop -// before storing it, with the same origin and official a publish would send. A refusal is the -// same coded error a publish would answer. -func (c *Client) ValidatePackage(ctx context.Context, archive []byte, origin string, official bool) (*ValidatedPackage, error) { - data, err := c.send(ctx, http.MethodPost, "/api/v1/packages/validate", packageQuery(origin, official), archive, "application/zip") - if err != nil { - return nil, err - } - var out ValidatedPackage - if err := json.Unmarshal(data, &out); err != nil { - return nil, err - } - return &out, nil -} - -// AppliesRequest names the patterns and the scope's assignment URLs, which report-service's -// deriver follows into their activities, so the URLs matched are the runner's. -type AppliesRequest struct { - URLs map[string][]string `json:"urls"` - AssignmentURLs []string `json:"assignment_urls,omitempty"` -} - -// AppliesAnswer is report-server's verdict, with what the deriver read and could not. -type AppliesAnswer struct { - Applies bool `json:"applies"` - Reason string `json:"reason"` - Unread []struct { - URL string `json:"url"` - Reason string `json:"reason"` - } `json:"unread"` - Truncated bool `json:"truncated"` -} - -// PackageApplies asks report-server's one matcher whether the patterns match the scope. -func (c *Client) PackageApplies(ctx context.Context, req AppliesRequest) (*AppliesAnswer, error) { - var out AppliesAnswer - if err := c.postJSON(ctx, "/api/v1/packages/applies", req, &out); err != nil { - return nil, err - } - return &out, nil -} - -func packageQuery(origin string, official bool) url.Values { - q := url.Values{} - if origin != "" { - q.Set("origin", origin) - } - if official { - q.Set("official", "true") - } - return q -} - -// RouteMissing reports a 404 from a route this cc-data expects and the server does not have -// yet: a report-server older than the validate and applies routes. -func RouteMissing(err error) bool { - var apiErr *APIError - return errors.As(err, &apiErr) && apiErr.Status == http.StatusNotFound -} -``` - ---- - -### The `cc-data package` commands and the `init` stub - -**Summary**: R9, R12, R16, R19 to R23. -- **Wiring.** Adds the four subcommands to the root, and `Init` with the embedded stub. -- **The report-server calls.** `validate` is shared by `build` and `run` and returns report-server's answer: `build` warns when `already_published` is true or `publishing_unavailable` names a reason, and `run` ignores both, since running a package needs neither a new version nor a bucket. `appliesFor` gives `run` its `Applies` function, with a 5-minute timeout for the deriver. -- **Testability.** `run`, `build` and `publish` each carry a `run(ctx, client, ...)` method, as `reportCreateFlags` does, so all three are tested against `httptest` with no stored credential. `package run` reads and checks the package and scope file (`load`) before it looks up a credential. - -**Files affected**: -- `cmd/package.go`: new. -- `cmd/root.go`: `root.AddCommand(newPackageCmd())` after `newQueryCmd()`. -- `internal/packages/init.go`: new. -- `internal/packages/build.go`: new; the default build directory, the `--out` check, writing the zip, and the `.gitignore` both working directories carry. -- `internal/packages/template/run.py`: new, embedded. -- `cmd/package_test.go`: new. - -**Estimated diff size**: ~600 lines - -```go -package cmd - -import ( - "context" - "errors" - "fmt" - "os" - "os/exec" - "os/signal" - "path/filepath" - "strings" - "time" - - "github.com/concord-consortium/cc-data-cli/internal/api" - "github.com/concord-consortium/cc-data-cli/internal/dataset" - "github.com/concord-consortium/cc-data-cli/internal/output" - "github.com/concord-consortium/cc-data-cli/internal/packages" - "github.com/spf13/cobra" -) - -const runDifferences = `What run cannot reproduce from the VM: the package runs as you, with no network -sandbox, and with HOME, PATH, the locale and any proxy settings passed through so its own -cc-data calls find your login and reach the server. A .py entrypoint runs with python3.11 -when it is on PATH, as on the VM, else python3.` - -const packageLong = `Develop, test and publish a Researcher Dashboard package. - -A package is a directory holding manifest.json and an entrypoint. init writes both; run -executes the package against your own dataset under the runner's rules; build zips it; -publish registers the zip in the catalog with your cc-data token. - -` + runDifferences - -func newPackageCmd() *cobra.Command { - cmd := &cobra.Command{ - Use: "package", - Short: "Develop, test and publish a Researcher Dashboard package", - Long: packageLong, - } - cmd.AddCommand(newPackageInitCmd(), newPackageRunCmd(), newPackageBuildCmd(), newPackagePublishCmd()) - return cmd -} - -// packageDir is the optional [dir] argument, defaulting to the current directory. -func packageDir(args []string) string { - if len(args) == 1 { - return args[0] - } - return "." -} - -// loadPackage collects a package directory and validates its manifest against what it holds. -func loadPackage(dir string) (packages.Manifest, packages.Files, error) { - files, err := packages.Collect(dir) - if err != nil { - return packages.Manifest{}, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_PACKAGE", Message: err.Error()} - } - data, err := os.ReadFile(filepath.Join(dir, "manifest.json")) - if err != nil { - return packages.Manifest{}, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_MANIFEST", Message: fmt.Sprintf("no manifest.json in %s", dir)} - } - m, err := packages.ReadManifest(data, files.Set()) - if err != nil { - return m, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_MANIFEST", Message: err.Error()} - } - if err := files.CheckEntrypointMode(m.Entrypoint); err != nil { - return m, files, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_PACKAGE", Message: err.Error()} - } - return m, files, nil -} - -func newPackageInitCmd() *cobra.Command { - var name string - cmd := &cobra.Command{ - Use: "init [dir]", - Short: "Write a manifest.json skeleton and a run.py stub", - Args: cobra.MaximumNArgs(1), - RunE: func(cmd *cobra.Command, args []string) error { - dir := packageDir(args) - written, err := packages.Init(dir, name) - if err != nil { - return output.Usagef("%v", err) - } - return output.ResultLine(map[string]any{"dir": dir, "written": written}) - }, - } - cmd.Flags().StringVar(&name, "name", "", "the package name (default: the directory's name, when it fits ^[a-z0-9][a-z0-9-]{0,62}$)") - return cmd -} - -// loadedPackage is a package directory and scope file read and checked, before any credential -// is needed. -type loadedPackage struct { - dir string - manifest packages.Manifest - files packages.Files - scope packages.LocalScope -} - -type packageRunFlags struct { - datasetRef, scopePath string - environ func() []string -} - -func (f packageRunFlags) load(dir string) (loadedPackage, error) { - m, files, err := loadPackage(dir) - if err != nil { - return loadedPackage{}, err - } - raw, err := os.ReadFile(f.scopePath) - if err != nil { - return loadedPackage{}, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: fmt.Sprintf("reading the scope file: %v", err)} - } - scope, err := packages.ParseLocalScope(raw) - if err != nil { - return loadedPackage{}, &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_SCOPE", Message: err.Error()} - } - return loadedPackage{dir: dir, manifest: m, files: files, scope: scope}, nil -} - -// run validates, runs and reports a loaded package; it takes the client so it can be exercised -// against a test server without a stored credential. -func (f packageRunFlags) run(ctx context.Context, client *api.Client, pkg loadedPackage, ref dataset.Ref, dataRoot string) error { - archive, err := packages.Zip(pkg.files) - if err != nil { - return output.Internalf("%v", err) - } - // A version that is already published, or a portal that cannot store packages yet, - // still runs; only a publish is refused. - if _, err := validate(ctx, client, archive, "", false); err != nil { - return err - } - start := time.Now() - res, err := packages.Run(ctx, packages.RunOptions{ - Dir: pkg.dir, Manifest: pkg.manifest, Files: pkg.files, Scope: pkg.scope, Applies: appliesFor(client, pkg.scope), - Dataset: ref.String(), Name: ref.Name, Portal: ref.Portal.Host(), DataRoot: dataRoot, - ReportSrv: client.BaseURL, Stderr: output.Stderr(), Environ: f.environ, LookPath: exec.LookPath, - }) - if err != nil { - return packageRunError(err) - } - return output.ResultLine(map[string]any{ - "display": res.DisplayPath, "display_bytes": res.DisplayBytes, "summary": res.Summary, - "counts": res.Counts, "elapsed_seconds": int(time.Since(start).Seconds()), - }) -} - -func newPackageRunCmd() *cobra.Command { - f := packageRunFlags{environ: os.Environ} - cmd := &cobra.Command{ - Use: "run [dir] --dataset --scope ", - Short: "Run a package locally exactly as the dashboard's runner will", - Long: "Run a package against your own dataset under the runner's rules: the same scope.json,\n" + - "environment variables, output files, display.md cap and applicability check.\n\n" + - "--scope names a file holding the scope's kind, id, classes and assignments.\n" + - "Results are written under /.cc-data-run/, which build never ships.\n\n" + runDifferences, - Args: cobra.MaximumNArgs(1), - RunE: func(cmd *cobra.Command, args []string) error { - if f.datasetRef == "" || f.scopePath == "" { - return output.Usagef("--dataset and --scope are required") - } - pkg, err := f.load(packageDir(args)) - if err != nil { - return err - } - cfg, dataRoot, err := loadRuntime() - if err != nil { - return err - } - ref, err := resolveRef(cfg, f.datasetRef) - if err != nil { - return err - } - client, err := api.ForPortal(ref.Portal) - if err != nil { - return err - } - // The terminal's Ctrl-C cannot reach the package's own process group, so it cancels the run. - ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) - defer stop() - return f.run(ctx, client, pkg, ref, dataRoot) - }, - } - cmd.Flags().StringVar(&f.datasetRef, "dataset", "", "the dataset the package pulls into and reads: /") - cmd.Flags().StringVar(&f.scopePath, "scope", "", "a JSON file holding the scope's kind, id, classes and assignments") - return cmd -} - -// derivationTimeout covers report-server's wait of up to 270 seconds on report-service's -// deriver; the client's 60-second default would cut it off. -const derivationTimeout = 5 * time.Minute - -// validate asks report-server whether a publish of the archive would be accepted. A server -// without the route gets a warning and a nil answer: the archive is unchecked, not refused. -func validate(ctx context.Context, client *api.Client, archive []byte, origin string, official bool) (*api.ValidatedPackage, error) { - ans, err := client.ValidatePackage(ctx, archive, origin, official) - switch { - case api.RouteMissing(err): - output.Warnf("report-server has no validate route yet, so this package is not checked against the catalog's rules until publish") - return nil, nil - case err != nil: - return nil, api.AsCLIError(err) - } - return ans, nil -} - -// appliesFor is package run's applicability check: report-server's one matcher, against the -// scope's assignment URLs and the interactive URLs report-service's deriver finds in them. -func appliesFor(client *api.Client, scope packages.LocalScope) func(context.Context, packages.URLs) (packages.Verdict, error) { - return func(ctx context.Context, urls packages.URLs) (packages.Verdict, error) { - long := *client - long.RequestTimeout = derivationTimeout - ans, err := long.PackageApplies(ctx, api.AppliesRequest{ - URLs: map[string][]string{"all": urls.All, "any": urls.Any, "none": urls.None}, - AssignmentURLs: scope.AssignmentURLs(), - }) - switch { - case api.RouteMissing(err): - return packages.Verdict{Unconfirmed: true}, nil - case err != nil: - return packages.Verdict{}, api.AsCLIError(err) - } - for _, u := range ans.Unread { - output.Warnf("the class profile could not read %s (%s), so its interactives are not matched", u.URL, u.Reason) - } - if ans.Truncated { - output.Warnf("the class profile was truncated, so not every interactive URL was matched") - } - return packages.Verdict{Applies: ans.Applies, Reason: ans.Reason}, nil - } -} - -func packageRunError(err error) error { - var refused *packages.Refused - var failed *packages.Failed - var out *packages.OutputRefused - var cliErr *output.CLIError - switch { - case errors.As(err, &cliErr): - return cliErr - case errors.As(err, &refused): - return &output.CLIError{ExitCode: output.ExitInternal, Code: "PACKAGE_REFUSED", Message: err.Error()} - case errors.As(err, &failed): - return &output.CLIError{ExitCode: output.ExitInternal, Code: "PACKAGE_FAILED", Message: err.Error()} - case errors.As(err, &out): - return &output.CLIError{ExitCode: output.ExitInternal, Code: "PACKAGE_OUTPUT_REFUSED", Message: err.Error()} - } - return output.Internalf("%v", err) -} - -type packageBuildFlags struct { - out, portal, origin string - official bool -} - -// run builds, validates and writes; it takes the client so it can be exercised against a test -// server without a stored credential. -func (f packageBuildFlags) run(ctx context.Context, client *api.Client, dir string) error { - m, files, err := loadPackage(dir) - if err != nil { - return err - } - out := f.out - if out == "" { - // The manifest is unchecked until validate answers, and unchecked for good on a server - // without the route, so its name and version must not be able to steer the path. - name := fmt.Sprintf("%s-%s.zip", m.Name, m.Version) - if strings.ContainsAny(name, `/\`) { - return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_MANIFEST", Message: fmt.Sprintf("the manifest's name and version make %q, which is not a file name; fix them or pass --out", name)} - } - out = filepath.Join(dir, packages.BuildDirName, name) - } else if err := packages.CheckOutsidePackage(dir, out); err != nil { - return output.Usagef("%v", err) - } - archive, err := packages.Zip(files) - if err != nil { - return output.Internalf("%v", err) - } - ans, err := validate(ctx, client, archive, f.origin, f.official) - if err != nil { - return err - } - if ans != nil && ans.AlreadyPublished { - output.Warnf("%s %s is already published; publish will refuse it until the version changes", ans.Identity, ans.Version) - } - if ans != nil && ans.PublishingUnavailable != nil { - output.Warnf("this package cannot be published yet: %s", *ans.PublishingUnavailable) - } - if err := packages.WriteBuild(out, archive); err != nil { - return output.Internalf("%v", err) - } - return output.ResultLine(map[string]any{ - "path": out, "checksum": packages.Checksum(archive), "bytes": len(archive), "files": len(files.Paths), - }) -} - -func newPackageBuildCmd() *cobra.Command { - var f packageBuildFlags - cmd := &cobra.Command{ - Use: "build [dir]", - Short: "Zip a package reproducibly, check it against the catalog's rules and print its checksum", - Args: cobra.MaximumNArgs(1), - RunE: func(cmd *cobra.Command, args []string) error { - client, err := reportsClient(f.portal) - if err != nil { - return err - } - return f.run(context.Background(), client, packageDir(args)) - }, - } - cmd.Flags().StringVar(&f.out, "out", "", "where to write the zip (default: /.cc-data-build/-.zip)") - cmd.Flags().StringVar(&f.portal, "portal", "", "portal whose catalog rules to check against: an environment alias or a hostname") - cmd.Flags().StringVar(&f.origin, "origin", "", "projects/, when the package will be published as a project's") - cmd.Flags().BoolVar(&f.official, "official", false, "check it as an official publish will be checked (publisher role only)") - return cmd -} - -type packagePublishFlags struct { - portal, origin string - official, asJSON bool -} - -// run publishes the archive; it takes the client so it can be exercised against a test server -// without a stored credential. -func (f packagePublishFlags) run(ctx context.Context, client *api.Client, archive []byte) error { - got, err := client.PublishPackage(ctx, archive, f.origin, f.official) - if err != nil { - return api.AsWriteCLIError(err, "Publishing again is safe: ALREADY_EXISTS then means this attempt landed") - } - if want := packages.Checksum(archive); got.Checksum != want { - return output.Internalf("report-server recorded checksum %s, but the archive sent is %s", got.Checksum, want) - } - if f.asJSON { - return output.ResultLine(got) - } - fmt.Fprintf(output.Stdout(), "identity %s\nversion %s\nvisibility %s\nofficial %t\ncurrent version %s\nchecksum %s\n", - got.Identity, got.Version, got.Visibility, got.Official, got.CurrentVersion, got.Checksum) - return nil -} - -func newPackagePublishCmd() *cobra.Command { - var f packagePublishFlags - cmd := &cobra.Command{ - Use: "publish [--portal ] [--origin projects/] [--official] [--json]", - Short: "Publish a built package to the catalog with your cc-data token", - Long: "POST the zip to report-server's catalog, which creates the package private on its first\n" + - "publish and records the version. --origin projects/ publishes a project package;\n" + - "--official needs the publisher role. The request is never retried.", - Args: cobra.ExactArgs(1), - RunE: func(cmd *cobra.Command, args []string) error { - archive, err := os.ReadFile(args[0]) - if err != nil { - return output.Usagef("reading %s: %v", args[0], err) - } - client, err := reportsClient(f.portal) - if err != nil { - return err - } - return f.run(context.Background(), client, archive) - }, - } - cmd.Flags().StringVar(&f.portal, "portal", "", "portal whose catalog to publish to: an environment alias or a hostname") - cmd.Flags().StringVar(&f.origin, "origin", "", "projects/ to publish a project's package") - cmd.Flags().BoolVar(&f.official, "official", false, "mark the package official (publisher role only)") - cmd.Flags().BoolVar(&f.asJSON, "json", false, "emit the catalog's answer as JSON") - return cmd -} -``` -```go -package packages - -import ( - _ "embed" - "encoding/json" - "errors" - "fmt" - "os" - "path/filepath" -) - -//go:embed template/run.py -var runStub []byte - -// Init writes manifest.json and run.py into dir, creating it if needed. It reads nothing in -// dir except to refuse when either file already exists, in which case it writes neither. -func Init(dir, name string) ([]string, error) { - if name == "" { - abs, err := filepath.Abs(dir) - if err != nil { - return nil, err - } - name = filepath.Base(abs) - if !ValidName(name) { - return nil, fmt.Errorf("the directory name %q is not a package name; pass --name (^[a-z0-9][a-z0-9-]{0,62}$)", name) - } - } else if !ValidName(name) { - return nil, fmt.Errorf("--name %q must match ^[a-z0-9][a-z0-9-]{0,62}$", name) - } - for _, f := range []string{"manifest.json", "run.py"} { - if _, err := os.Lstat(filepath.Join(dir, f)); !errors.Is(err, os.ErrNotExist) { - return nil, fmt.Errorf("%s already exists in %s; init writes only into a package that has neither file", f, dir) - } - } - manifest := Manifest{ - Name: name, Title: name, Version: "0.1.0", - Description: "One line saying what this package shows.", - URLs: URLs{All: []string{}, Any: []string{}, None: []string{}}, - Entrypoint: "run.py", ExpectedDurationSeconds: 300, - } - b, err := json.MarshalIndent(manifest, "", " ") - if err != nil { - return nil, err - } - if err := os.MkdirAll(dir, 0o755); err != nil { - return nil, err - } - if err := os.WriteFile(filepath.Join(dir, "manifest.json"), append(b, '\n'), 0o644); err != nil { - return nil, err - } - if err := os.WriteFile(filepath.Join(dir, "run.py"), runStub, 0o755); err != nil { - return nil, err - } - return []string{"manifest.json", "run.py"}, nil -} -``` -```go -package packages - -import ( - "fmt" - "os" - "path/filepath" - "strings" -) - -// BuildDirName is where build writes by default: inside the package, and skipped by -// Collect like every dot path, so one build's zip never reaches the next. -const BuildDirName = ".cc-data-build" - -// CheckOutsidePackage refuses an explicit build output that a later build would collect. -func CheckOutsidePackage(dir, out string) error { - absDir, err := filepath.Abs(dir) - if err != nil { - return err - } - absOut, err := filepath.Abs(out) - if err != nil { - return err - } - rel, err := filepath.Rel(absDir, absOut) - if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { - return nil - } - if excluded(filepath.ToSlash(rel), false) || excludedAncestor(filepath.ToSlash(rel)) { - return nil - } - return fmt.Errorf("--out %s is inside the package, where the next build would include it", out) -} - -func excludedAncestor(rel string) bool { - parts := strings.Split(rel, "/") - for i := 1; i < len(parts); i++ { - if excluded(strings.Join(parts[:i], "/"), true) { - return true - } - } - return false -} - -// WriteBuild writes the archive, creating its directory, and ignores the default build -// directory for git. -func WriteBuild(out string, archive []byte) error { - dir := filepath.Dir(out) - if filepath.Base(dir) != BuildDirName { - if err := os.MkdirAll(dir, 0o755); err != nil { - return err - } - } else if err := os.MkdirAll(filepath.Dir(dir), 0o755); err != nil { - return err - } else if err := ensureRealDir(dir); err != nil { - return err - } else if err := ignoreAll(dir); err != nil { - return err - } - return os.WriteFile(out, archive, 0o644) -} -``` -```python -#!/usr/bin/env python3 -"""A Researcher Dashboard package: counts a class's answers and the learners who wrote them. - -It runs the same way under `cc-data package run` on a laptop and on the dashboard's VM. It -reads the scope from RD_SCOPE_FILE, pulls its own data into RD_DATASET with cc-data, counts -through cc-data's views, and writes display.md, summary.txt and counts.json to the scope's -output_dir. Stdlib only: the VM installs no Python packages. - -It reads the class through a Student ID Mapping run, which the portal computes on every -download, so re-reading it picks up learners who joined since, and which has no limit on how -many assignments a class has. It reuses the researcher's own run for the class when there is -one and otherwise creates one, on the server the dataset's portal names. -""" -import json -import os -import subprocess -import sys - - -def cc_data(*args): - result = subprocess.run(["cc-data", *args], capture_output=True, text=True) - if result.returncode != 0: - detail = " ".join(p for p in (result.stderr.strip(), result.stdout.strip()) if p) - raise RuntimeError(f"cc-data {' '.join(args[:2])} exited {result.returncode}: {detail or '(no output)'}") - return result.stdout - - -def query(dataset, sql): - return json.loads(cc_data("query", "--dataset", dataset, sql, "--format", "json")) - - -def mapping_run(portal, class_id): - """The newest Student ID Mapping run filtered to exactly this class, else a new one.""" - runs = json.loads(cc_data("reports", "list", "--portal", portal, "--json"))["runs"] - mine = [r["run_id"] for r in runs - if r.get("slug") == "student-id-mapping" - and (r.get("report_filter") or {}).get("filters") == ["class"] - and (r.get("report_filter") or {}).get("class") == [class_id]] - if mine: - return max(int(run) for run in mine) - created = json.loads(cc_data( - "reports", "create", "--portal", portal, "--report-slug", "student-id-mapping", - "--report-filter", json.dumps({"class": [class_id]}), "--json")) - return int(created["run"]["run_id"]) - - -def main(): - with open(os.environ["RD_SCOPE_FILE"], encoding="utf-8") as handle: - scope = json.load(handle) - # RD_DATASET is the full / ref; scope["dataset"] is the bare name. - dataset = os.environ["RD_DATASET"] - class_id = int(scope["classes"][0]["class_id"]) - - try: - cc_data("dataset", "create", dataset) - except RuntimeError as err: - if "exists" not in str(err).lower(): - raise - run = mapping_run(os.environ["CC_DATA_PORTAL"], class_id) - cc_data("get", "report", str(run), "--dataset", dataset, "--refresh") - cc_data("get", "answers", str(run), "--dataset", dataset) - - rows = query(dataset, f"SELECT count(*) AS n FROM student_id_mapping WHERE run_id = {run}")[0]["n"] - if rows == 0: - raise RuntimeError("no report-service access to this class, or the class has no data") - answers = query(dataset, f"SELECT count(*) AS n FROM run_answers WHERE run_id = {run}")[0]["n"] - # By user_id, never by endpoint: a student has one endpoint per assignment. - learners = query(dataset, f""" - SELECT count(DISTINCT m.user_id) AS n FROM student_id_mapping m - JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint - WHERE a.run_id = {run} AND m.run_id = {run}""")[0]["n"] - - out = scope["output_dir"] - with open(os.path.join(out, "display.md"), "w", encoding="utf-8") as handle: - handle.write(f"# Class {class_id}\n\n- {answers} answers\n- {learners} learners with at least one answer\n") - with open(os.path.join(out, "summary.txt"), "w", encoding="utf-8") as handle: - handle.write(f"{answers} answers from {learners} learners\n") - with open(os.path.join(out, "counts.json"), "w", encoding="utf-8") as handle: - json.dump({"answers": answers}, handle) - - -if __name__ == "__main__": - try: - main() - except Exception as err: - print(str(err), file=sys.stderr) - sys.exit(1) -``` - -The stub was run through `Run` against a synthetic dataset holding a Student ID Mapping run and its answers, with a wrapper `cc-data` that fakes `reports list`, `reports create` and `get`, and forwards `dataset` and `query` to the scratch build. -- **The run it picks.** Out of a `reports list` holding a Student ID Mapping run filtered by class and school, one filtered by the class alone, a Student Answers run and another class's run, it took the class-only Student ID Mapping run. -- **The result.** It wrote "3 answers / 2 learners with at least one answer" and `counts.json` `{"answers": 3}`: a learner in two assignments counted once, and a learner with no answers not at all. -- **With no matching run** it called `reports create --report-slug student-id-mapping --report-filter {"class": [6]}`. With nothing pulled, it stopped at the empty-scope guard with its message. - -Tests (against an `httptest` server that answers 404 for any route it was not given): -- **`TestPackageInitRefusesASecondInit`**. -- **`TestPackageBuildValidatesThenWrites`**: - - Exactly one validate request is made, carrying the bytes that were written, as `application/zip`. - - The zip is `wildfire-responses-0.1.0.zip` with two files, `.cc-data-build/.gitignore` is `*`, and the printed checksum is the archive's own. - - A second build from inside the package gives the same checksum. -- **`TestPackageBuildWritesNothingTheServerRefuses`**: a 422 `UNPROCESSABLE` keeps its code, exits 5, and no zip is written. -- **`TestPackageBuildWarnsOnAPublishedVersionAndStillWrites`**: `already_published: true` prints "users/7/p 0.1.0 is already published; publish will refuse it until the version changes", and the zip is still written. **`TestPackageBuildIsQuietOnANewVersion`** checks the converse. -- **`TestPackageBuildValidatesAsThePublishWillBe`**: `--origin projects/20 --official` reach validate's query, and `official` is absent without the flag. -- **`TestValidateReturnsAPublishedVersionAsAnAnswerNotAnError`**: the shared `validate` returns the answer, so `run` proceeds on a published version. -- **`TestPackageBuildWarnsWhenThePortalCannotPublishAndStillWrites`**: `publishing_unavailable: "publishing is not configured for x"` prints "this package cannot be published yet: publishing is not configured for x", and the zip is still written. **`TestPackageBuildIsQuietWhenPublishingIsAvailable`** checks `null`. -- **`TestValidateReturnsAnUnavailablePortalAsAnAnswerNotAnError`**: so `run` proceeds on a portal with no bucket yet, such as staging before `PackageBuckets` is set. -- **`TestPackageBuildOnAServerWithoutTheRouteStillBuilds`**: a 404 is a warning. -- **`TestPackageBuildRefusesAnOutputItWouldCollect`**. -- **`TestPackageBuildRefusesANameThatIsAPath`**: on a server without the validate route, a manifest named `../../escaped` is `INVALID_MANIFEST`, and no zip is written anywhere. -- **`TestPackageRunKeepsTheServersCode`**: `packageRunError` given applies' 503 `SERVICE_UNAVAILABLE` returns that code with exit 5, and given `NOT_AUTHENTICATED` exit 3. -- **`TestPackageRunValidatesThenRunsWithTheRunnersNames`**: one validate request carries the zip, the package sees `RD_DATASET` as the full ref, `CC_DATA_PORTAL`, `CC_DATA_ROOT` and `RD_REPORT_SERVER_URL` as the client's server, `scope.json` carries the bare name, and the stdout line has `display`, `display_bytes`, `summary`, `counts` and `elapsed_seconds`. -- **`TestPackageRunAsksTheServersMatcher`**: the request carries the patterns and the assignment URLs, and a "does not apply" answer comes back as the verdict. -- **`TestPackageRunOnAServerWithoutTheRouteIsUnconfirmed`**. -- **`TestPackageRunAlwaysSendsURLs`**: the applies body always carries `urls`, which REPORT-167 requires (a missing or `null` `urls` is 400 there). -- **`TestPackagePublishSendsTheZipAsTheBody`**: checks POST, `application/zip`, `Content-Length`, `origin=projects/20` present, `official` absent, and the bearer. -- **`TestPackagePublishRefusesAChecksumItDidNotSend`**. -- **`TestPackagePublishPassesTheServersCodeThrough`**: a 409 `ALREADY_EXISTS` keeps its code, exits 5, and is sent once. -- **`TestPackagePublishMapsAuthAndUnansweredFailures`**: `NOT_AUTHENTICATED` exits 3, and a connection dropped before any answer carries the "Publishing again is safe" action. -- **`TestPackageRunWarnsWhatTheDeriverCouldNotRead`**: each unread URL with its reason, and a truncated profile, reach stderr. -- **`TestPackageRunRefusesAnUnreadableScopeFile`**: a missing scope file is `INVALID_SCOPE`, exit 2. - -| Mutation | Test that fails | -|---|---| -| a 404 from validate treated as a refusal | `TestPackageBuildOnAServerWithoutTheRouteStillBuilds` | -| `build` skips validate | `TestPackageBuildValidatesThenWrites`, `...WritesNothingTheServerRefuses` | -| assignment URLs not sent to applies | `TestPackageRunAsksTheServersMatcher` | -| no already-published warning | `TestPackageBuildWarnsOnAPublishedVersionAndStillWrites` | -| no unavailable-portal warning | `TestPackageBuildWarnsWhenThePortalCannotPublishAndStillWrites` | -| the warning printed when `publishing_unavailable` is `null` | `TestPackageBuildIsQuietWhenPublishingIsAvailable` | -| `publishing_unavailable` read under the wrong name | `TestPackageBuildWarnsWhenThePortalCannotPublishAndStillWrites`, `TestValidateReturnsAnUnavailablePortalAsAnAnswerNotAnError` | -| `official` not sent to validate | `TestPackageBuildValidatesAsThePublishWillBe` | -| `package run` skips validate, or swaps the ref and its bare name | `TestPackageRunValidatesThenRunsWithTheRunnersNames` | -| `packageRunError` without its `CLIError` case | `TestPackageRunKeepsTheServersCode` (exit 1, `INTERNAL`) | -| the default output name not checked | `TestPackageBuildRefusesANameThatIsAPath` (the zip lands two directories up) | -| `already_published` read under the wrong name | `TestPackageBuildWarnsOnAPublishedVersionAndStillWrites`, `TestValidateReturnsAPublishedVersionAsAnAnswerNotAnError` | - ---- - -### Documentation - -**Summary**: R24. The README's "Commands (sketch)" gains the four `package` lines. The researcher guide gains a section, "Developing and publishing a package", with these parts: -- the loop (`init`, edit, `run`, `build`, `publish`); -- the scope file's four keys, with an example, kept outside the package or under a dot name such as `.scope.json` so `build` never ships a class hash; -- what `run` reproduces and what it cannot (the sandbox, the passthrough variables, python3.11); -- that `build` and `run` check against report-server, and the warning an older report-server gives; -- `.cc-data-run/` and `.cc-data-build/` (gitignored, and holding student data); -- the 422 "publishing is not configured" answer a portal gives before its bucket exists. - -`skill_header.md` gains a short "Packages" list naming the four commands. The `--help` texts above state the differences between a local run and a VM, on both `package` and `package run`. The guide's later sections renumber from 8. - -**Files affected**: -- `README.md`. -- `docs/researcher-guide.md`. -- `internal/guidance/src/skill_header.md`. - -**Estimated diff size**: ~90 lines - ---- - -### Pre-release tags in the release workflow - -**Summary**: R25. `.github/workflows/release.yml` classifies the tag once, and the release and formula steps both read that answer: a pre-release tag (a `-` after the version, as in `v0.3.0-pre.1`) creates a GitHub pre-release with every archive and checksum, and skips the Homebrew formula. The archives upload inside `gh release create` itself, before the formula step, so skipping the formula cannot lose them. A test in `release_test.go`, beside the workflow's other guards, pins both uses. - -**Files affected**: -- `.github/workflows/release.yml`: one step, one flag, one condition. -- `release_test.go`: one test. -- `README.md`: one sentence in the release paragraph. - -**Estimated diff size**: ~40 lines - -```diff - - name: Create GitHub release - env: - GH_TOKEN: ${{ github.token }} - run: | - # Attach the checksum manifests alongside the tarballs so consumers can - # verify downloads independently. - gh release create "${GITHUB_REF_NAME}" dist/*.tar.gz dist/*.sha256 \ -- --title "${GITHUB_REF_NAME}" --generate-notes -+ --title "${GITHUB_REF_NAME}" --generate-notes \ -+ --prerelease=${{ steps.tag.outputs.prerelease }} - - - name: Render and push the Homebrew formula -+ if: steps.tag.outputs.prerelease == 'false' - env: -``` - -and, before "Create GitHub release": - -```yaml - # A pre-release tag (v0.3.0-pre.1) exists to be pinned: it is never marked latest - # and never reaches Homebrew. - - name: Classify the tag - id: tag - run: | - if [[ "${GITHUB_REF_NAME}" == *-* ]]; then - echo "prerelease=true" >> "${GITHUB_OUTPUT}" - else - echo "prerelease=false" >> "${GITHUB_OUTPUT}" - fi -``` - -```go -// A pre-release tag exists to be pinned, so it must not become GitHub's latest release or -// reach researchers through brew upgrade. Both steps read the one classification. -func TestReleaseWorkflowKeepsPreReleasesBack(t *testing.T) { - rel := readFile(t, ".github/workflows/release.yml") - for _, want := range []string{ - `if [[ "${GITHUB_REF_NAME}" == *-* ]]; then`, - "--prerelease=${{ steps.tag.outputs.prerelease }}", - "if: steps.tag.outputs.prerelease == 'false'", - } { - if !strings.Contains(rel, want) { - t.Fatalf("release workflow must hold a pre-release tag back from latest and from Homebrew; missing %q", want) - } - } -} -``` - -README, after the sentence naming the Homebrew formula: "A tag with a pre-release suffix, such as `v0.3.0-pre.1`, publishes a GitHub pre-release with the same archives and no formula, for pinning by the runner image and package workflows." - -Checked in a scratch copy: the workflow still parses, with the new step between "Verify checksums" and "Create GitHub release"; the classification writes `prerelease=true` for `v0.3.0-pre.1` and `false` for `v0.3.0`; and `gh release create` (2.100.0) parses `--prerelease=false` as a boolean, where `--prerelease=bogus` is refused. - -| Mutation | Test that fails | -|---|---| -| the formula step's condition removed | `TestReleaseWorkflowKeepsPreReleasesBack` | - -## Open Questions - - - -### RESOLVED: Judgment call: how a raw zip body reaches the server -**Options considered**: -- A) Generalize `Client.do` into `send` with a content type. -- B) Build a separate `http.Request` for the zip routes. - -**Decision**: A. Publish and validate then share the bearer, the cross-origin redirect refusal, the no-retry rule for POST and the error-envelope decoding with every other call. B would re-implement each of them. - -### RESOLVED: Judgment call: the exit class for a refused or failed package -**Options considered**: -- A) Exit 1 with codes `PACKAGE_REFUSED`, `PACKAGE_FAILED` and `PACKAGE_OUTPUT_REFUSED`. -- B) A new exit code 7. - -**Decision**: A. The root help defines exit 1 as "internal/other error", and the JSON envelope's code already distinguishes the cases. A new exit class is a contract change for every script that reads cc-data's exit codes. - -### RESOLVED: Judgment call: following no link without `O_NOFOLLOW` -**Options considered**: -- A) `Lstat`, then open, then `os.SameFile`. -- B) `syscall.O_NOFOLLOW`. - -**Decision**: A. CI tests on `windows-2022`, where `syscall.O_NOFOLLOW` does not exist. On a laptop the race A leaves open would need the author's own package to swap a file mid-read, and the runner, which runs as root, is the boundary that matters. - -### RESOLVED: Judgment call: what a 404 from the new routes means -**Options considered**: -- A) Any 404 from validate or applies means the route is not deployed: warn and proceed. -- B) Require a specific error code before treating it as "not deployed". - -**Decision**: A. report-server's router has no route either path could match today (`/packages/validate` and `/packages/applies` are one segment, and the only other `POST /packages/...` route takes four). So a 404 can only mean the route is missing, and B would need a code that an older server, by definition, cannot send. - -### RESOLVED: Low confidence: the process-group kill on macOS -**Context**: `TestRunKillsAtTheBound` ran on Linux only. - -**Decision**: `Setpgid` and `kill(-pgid, SIGKILL)` are POSIX and behave the same on Darwin, and `GOOS=darwin go build ./internal/packages/` compiles. CI's `macos-15` and `macos-15-intel` jobs run the test, so a Darwin difference would fail there. - -## Self-Review - -Roles: commit reviewer, test writer, senior engineer, operator, security engineer. Each finding was checked against the scratch build before it was written down. - -### Senior Engineer - -#### RESOLVED: A shell entrypoint would run locally and be refused on the VM -The zip originally gave every entry mode 0644, and the runner executes a non-`.py` entrypoint directly. A Go zip with a 0644 `plain.sh` and a 0755 `exec.sh`, unpacked with `unzip`, gives "Permission denied" for the first and runs the second, so `unzip` restores the zip's modes. Locally, the staged copy kept the source's 0755, so `package run` passed. Fixed: -- `Files` records a ship mode per file (0755 if any execute bit is set, else 0644). -- `Zip` and the staging copy both use it. -- `CheckEntrypointMode` refuses a non-executable non-`.py` entrypoint in `run` and `build`. - -R17 states the rule. - -#### RESOLVED: `RunOptions.Now` was never read -The field and its assignments were dead, so they are removed. Elapsed time is measured in `cmd/package.go`, where it is reported. - -### Commit Reviewer - -#### RESOLVED: Shared test helpers would not compile step by step -The scratch build kept every test in one file. The plan places each helper in the first step whose tests use it: `goodManifest` and `writeTree` in `files_test.go`, and the run helpers in `run_test.go`. - -### Operator - -#### RESOLVED: A scope file inside the package ships with it -An author naming the file `scope.json` in the package directory would publish a class hash in every archive. The documentation step says to keep it outside the package or under a dot name, which `Collect` already excludes. - -#### RESOLVED: The applicability call would time out before the deriver finishes -The deriver may spend up to 240 seconds on a scope's activities, and the client's per-attempt timeout for JSON calls is 60 seconds. `appliesFor` copies the client with a 5-minute `RequestTimeout`, which also keeps the shared client's default for every other call. - -### Second round - -Roles: senior engineer, package author, security engineer, operator, and a reviewer for one source of truth. Every finding was confirmed against `main`, against the plan's code extracted into a scratch copy, or against the runner and REPORT-167 sources, and each had one defensible fix, which has been applied above. The scratch copy built and passed `go test ./...` on Linux with the fixes, and `go vet` was clean for `internal/packages` under `GOOS=windows` and `GOOS=darwin`. - -#### RESOLVED: An applies failure lost its code and exit class (Senior Engineer) -`Run` returns `appliesFor`'s `*output.CLIError` unchanged, and `packageRunError` matched only the three package error types, so it fell through to `Internalf`. A throwaway test fed it applies' 503 `SERVICE_UNAVAILABLE` and `NOT_AUTHENTICATED`, and both came back exit 1 `INTERNAL`, against R16. `packageRunError` now returns a `CLIError` as it is, and `TestPackageRunKeepsTheServersCode` pins it. - -#### RESOLVED: Ctrl-C left the package running (Package Author) -`Setpgid` takes the package out of the terminal's foreground group, and cc-data had no interrupt handling outside `dataset materialize`. Sending `SIGINT` to cc-data's group, as a terminal does, ended cc-data while the package's `sleep 30` kept running. `package run` now cancels on `os.Interrupt` (the `materialize` idiom), which kills the group through the existing `cmd.Cancel`, and `Run` reports "the package was interrupted". Rerun with the fix, the package was gone a second after the signal. - -#### RESOLVED: `build`'s default path came from unchecked manifest fields (Security Engineer) -On a server without the validate route, a manifest named `../../escaped` built to `escaped-1.0.0.zip` two directories above the package, outside `.cc-data-build` and its `.gitignore`. `build` now refuses a name and version that make a path. It is a path check, not a copy of report-server's name grammar. - -#### RESOLVED: The proxy variables were not passed through (Operator) -cc-data's client uses Go's default transport, which reads `HTTPS_PROXY`, and the runner sets the proxy variables when it has a proxy (`package-env.js`). Behind a proxy, `package run`'s own requests would succeed while the package's `cc-data` calls failed. They are now on the passthrough list. - -#### RESOLVED: "No extra fields" in the zip was untrue (Senior Engineer) -`archive/zip` writes `Modified` as an extended-timestamp extra field (`5554...` on every entry of a scratch build). It is as fixed as the time, so R18 holds; the comment and R18 now say so. - -#### RESOLVED: `publish` kept a local copy of the origin rule (One Source of Truth) -`publish` refused an `--origin` without the `projects/` prefix, `build` did not, and report-server answers the same case with 422 "origin must be projects/; a user origin is always your own". The local check is deleted, so both commands give report-server's answer. - -Checked and found sound: the R3 test (dropping `m.run_id = ` fails it at run 800, 2 learners against 1); a second `get answers` on a fetched run re-fetches and merges new answers, so the stub stays current; `reports list --json`'s `report_filter` carries `filters` and `class` as the stub reads them; `query --format json` answers an array of objects; report-server's 270-second wait sits between the deriver's 240 and cc-data's 300; an old report-server answers both new routes with a JSON 404 even without a token; and no publish check refuses an ordinary user's own origin, so `run`'s validate call cannot lock out a researcher who may use the API. - From 8d2a55dcc38b7a77ebbc4f96e25ef4bb4e8241ed Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Thu, 8 Oct 2026 16:40:31 -0400 Subject: [PATCH 13/17] fix: address Copilot's review of the package work [REPORT-146] - The documented learner count no longer filters student_id_mapping by run_id. That view keeps each learner's row from whichever mapping run was fetched last, so the filter dropped a learner whom a later mapping run of the same class also held. The answers are already scoped by run_answers.run_id, and a run's answers belong only to its own learners. The init stub's empty-scope guard counts the run's own report_ for the same reason. The test gains a later mapping run of the same class, so filtering the mapping by run now fails it. - build writes the zip to a temporary file and renames it into place, so a link already at the output path is replaced, not followed. - A scope file must hold one JSON object and nothing after it. - result.go says why summary.txt and counts.json are read under the display cap: the runner reads them that way too. --- internal/duck/run_views_test.go | 23 +++++++++------ internal/guidance/src/core.md | 10 ++++--- internal/packages/build.go | 24 ++++++++++++++- internal/packages/result.go | 1 + internal/packages/run_test.go | 29 ++++++++++++++++++- internal/packages/scope.go | 4 +++ internal/packages/template/run.py | 4 +-- ...-146-package-commands-and-dataset-views.md | 27 ++++++++++++++--- 8 files changed, 101 insertions(+), 21 deletions(-) diff --git a/internal/duck/run_views_test.go b/internal/duck/run_views_test.go index c8b3508..3ada34f 100644 --- a/internal/duck/run_views_test.go +++ b/internal/duck/run_views_test.go @@ -10,13 +10,15 @@ import ( "github.com/concord-consortium/cc-data-cli/internal/store" ) -// The documented learner count, run-scoped on both sides, as the guidance gives it. -const learnersOfRun = "SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = %d AND m.run_id = %d" +// The documented learner count, scoped by the answers' run as the guidance gives it. +const learnersOfRun = "SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = %d" // Each wrong way to count gives a different number here: counting endpoints instead of users // (user 1 answered in two assignments), joining the bare-endpoint learner (who would borrow // another bare answer), counting an answer once instead of once per run (e3/q1 is in runs 700 -// and 800), and joining membership untyped (run 700's history holds e1a/q1 too). +// and 900), joining membership untyped (run 700's history holds e1a/q1 too), leaving the +// answers unscoped (run 800 is another class), and scoping the mapping by run (run 900, a later +// mapping run of the same class, holds user 3's row in the deduplicated view). func TestRunAnswersAndTheDocumentedLearnerCount(t *testing.T) { d := newDS(t, "ds") buildStore(t, d, 700, [][]byte{ @@ -32,7 +34,8 @@ func TestRunAnswersAndTheDocumentedLearnerCount(t *testing.T) { if _, err := d.MergeCompact(store.TypeHistory, 700, seg); err != nil { t.Fatal(err) } - buildStore(t, d, 800, [][]byte{answerRec("s", "https://p/d/e3", "q1", "c"), answerRec("s", "https://p/d/e9", "q1", "z")}) + buildStore(t, d, 800, [][]byte{answerRec("s", "https://p/d/e9", "q1", "z")}) + buildStore(t, d, 900, [][]byte{answerRec("s", "https://p/d/e3", "q1", "c")}) row := func(learner, user, class, offering int, endpoint string) string { return fmt.Sprintf("%d,%d,%d,s%d,%d,%d,https://act/1,%s", learner, user, user, user, class, offering, endpoint) } @@ -41,18 +44,20 @@ func TestRunAnswersAndTheDocumentedLearnerCount(t *testing.T) { row(103, 3, 1, 70, "https://p/d/e3"), row(104, 4, 1, 70, "https://p/d/"))}) addDimensionCSV(t, d, dimFixture{run: 800, slug: "student-id-mapping", fetchedAt: at(2), filter: `{}`, csv: mappingCSV( row(109, 9, 2, 90, "https://p/d/e9"))}) + addDimensionCSV(t, d, dimFixture{run: 900, slug: "student-id-mapping", fetchedAt: at(3), filter: `{}`, csv: mappingCSV( + row(103, 3, 1, 70, "https://p/d/e3"))}) e := openEngine(t, []DatasetSpec{{DS: d}}, nil) if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 700"); n != 4 { t.Errorf("run 700 answers = %d, want 4", n) } - if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 800"); n != 2 { - t.Errorf("run 800 answers = %d, want 2", n) + if n := queryInt(t, e, "SELECT count(*) FROM run_answers WHERE run_id = 900"); n != 1 { + t.Errorf("run 900 answers = %d, want 1", n) } - if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 700, 700)); n != 2 { + if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 700)); n != 2 { t.Errorf("run 700 learners = %d, want 2 (users 1 and 3; user 4's endpoint is bare)", n) } - if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 800, 800)); n != 1 { + if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 800)); n != 1 { t.Errorf("run 800 learners = %d, want 1", n) } } @@ -62,7 +67,7 @@ func TestRunAnswersIsEmptyNotMissingOnAFreshDataset(t *testing.T) { if n := queryInt(t, e, "SELECT count(*) FROM run_answers"); n != 0 { t.Errorf("run_answers has %d rows", n) } - if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 1, 1)); n != 0 { + if n := queryInt(t, e, fmt.Sprintf(learnersOfRun, 1)); n != 0 { t.Errorf("the learner count binds to %d on a fresh dataset, want 0", n) } } diff --git a/internal/guidance/src/core.md b/internal/guidance/src/core.md index 987cce2..28625bc 100644 --- a/internal/guidance/src/core.md +++ b/internal/guidance/src/core.md @@ -132,10 +132,12 @@ like `wildfire_2026.answers`): undercounts a run whose answers a later run re-fetched. - A run's learners with at least one answer come from its Student ID Mapping report: `SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN - run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = 584 - AND m.run_id = 584`, where 584 is a Student ID Mapping run whose answers were - fetched. Count by `user_id`, never by endpoint, since a student has one endpoint - per assignment. A Student ID Mapping run is computed live, so re-reading it with + run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = + 584`, where 584 is a Student ID Mapping run whose answers and report were + fetched. Scope it by the answers' `run_id` only: `student_id_mapping` keeps each + learner's row from whichever run was fetched last, so `m.run_id = 584` drops a + learner another mapping run of the class also holds. Count by `user_id`, never + by endpoint, since a student has one endpoint per assignment. A Student ID Mapping run is computed live, so re-reading it with `--refresh` picks up learners who joined since; a Student Answers run is fixed when its query ran, and a whole-class one fails above three or four assignments. - A run's log freshness is `SELECT count(*) AS logs, max(event_time) AS diff --git a/internal/packages/build.go b/internal/packages/build.go index 6c5b52a..c8d1800 100644 --- a/internal/packages/build.go +++ b/internal/packages/build.go @@ -56,5 +56,27 @@ func WriteBuild(out string, archive []byte) error { } else if err := ignoreAll(dir); err != nil { return err } - return os.WriteFile(out, archive, 0o644) + return replaceFile(out, archive) +} + +// replaceFile writes data beside path and renames it into place, so a link already at path is +// replaced rather than followed. +func replaceFile(path string, data []byte) error { + tmp, err := os.CreateTemp(filepath.Dir(path), ".build-*") + if err != nil { + return err + } + defer os.Remove(tmp.Name()) + if _, err := tmp.Write(data); err != nil { + tmp.Close() + return err + } + if err := tmp.Chmod(0o644); err != nil { + tmp.Close() + return err + } + if err := tmp.Close(); err != nil { + return err + } + return os.Rename(tmp.Name(), path) } diff --git a/internal/packages/result.go b/internal/packages/result.go index 51cc898..a27daaa 100644 --- a/internal/packages/result.go +++ b/internal/packages/result.go @@ -50,6 +50,7 @@ func ReadResult(outDir string) (Result, error) { return res, &OutputRefused{Reason: fmt.Sprintf("the package's display.md is %d bytes, over the %d-byte limit", size, DisplayLimitBytes)} } res.DisplayPath = filepath.Join(outDir, "display.md") + // The runner reads summary.txt and counts.json under the display cap too, truncating them. res.DisplayBytes = size if text, _, err := readOwnFile(outDir, "summary.txt", DisplayLimitBytes); err == nil { line := strings.TrimSpace(strings.SplitN(strings.ReplaceAll(text, "\r\n", "\n"), "\n", 2)[0]) diff --git a/internal/packages/run_test.go b/internal/packages/run_test.go index 8de16cc..65e357e 100644 --- a/internal/packages/run_test.go +++ b/internal/packages/run_test.go @@ -87,7 +87,10 @@ func TestLocalScopeRefusesTheRunnersKeys(t *testing.T) { if _, err := ParseLocalScope([]byte(withDataset)); err == nil { t.Error("a scope file setting dataset was accepted") } - if _, err := ParseLocalScope([]byte(scopeJSON)); err != nil { + if _, err := ParseLocalScope([]byte(scopeJSON + ` {"dataset": "other"}`)); err == nil { + t.Error("a scope file with a second JSON value after it was accepted") + } + if _, err := ParseLocalScope([]byte(scopeJSON + "\n")); err != nil { t.Errorf("a good scope file was refused: %v", err) } } @@ -320,3 +323,27 @@ func TestEntrypointFallsBackToPython3WithANote(t *testing.T) { t.Error("no interpreter was not an error") } } + +func TestWriteBuildReplacesALinkedZipRatherThanFollowingIt(t *testing.T) { + skipOnWindows(t, "links need privileges on Windows") + dir := t.TempDir() + target := filepath.Join(t.TempDir(), "precious") + writeTree(t, filepath.Dir(target), map[string]string{"precious": "keep"}) + writeTree(t, dir, map[string]string{BuildDirName + "/": ""}) + out := filepath.Join(dir, BuildDirName, "p-0.1.0.zip") + if err := os.Symlink(target, out); err != nil { + t.Fatal(err) + } + if err := WriteBuild(out, []byte("zip")); err != nil { + t.Fatal(err) + } + if got, _ := os.ReadFile(target); string(got) != "keep" { + t.Errorf("the link's target became %q", got) + } + if info, err := os.Lstat(out); err != nil || !info.Mode().IsRegular() { + t.Errorf("the zip is not a regular file: %v, %v", info, err) + } + if entries, _ := os.ReadDir(filepath.Join(dir, BuildDirName)); len(entries) != 2 { + t.Errorf("build directory holds %v, want the zip and .gitignore only", entries) + } +} diff --git a/internal/packages/scope.go b/internal/packages/scope.go index 1315541..3d78c59 100644 --- a/internal/packages/scope.go +++ b/internal/packages/scope.go @@ -4,6 +4,7 @@ import ( "bytes" "encoding/json" "fmt" + "io" "regexp" ) @@ -63,6 +64,9 @@ func ParseLocalScope(data []byte) (LocalScope, error) { if err := dec.Decode(&s); err != nil { return s, fmt.Errorf("scope file: %v (it holds exactly kind, id, classes and assignments)", err) } + if err := dec.Decode(&struct{}{}); err != io.EOF { + return s, fmt.Errorf("scope file: it must hold one JSON object and nothing after it") + } switch { case s.Kind == "": return s, fmt.Errorf("scope file: kind is required") diff --git a/internal/packages/template/run.py b/internal/packages/template/run.py index f0ce06f..5974892 100644 --- a/internal/packages/template/run.py +++ b/internal/packages/template/run.py @@ -60,7 +60,7 @@ def main(): cc_data("get", "report", str(run), "--dataset", dataset, "--refresh") cc_data("get", "answers", str(run), "--dataset", dataset) - rows = query(dataset, f"SELECT count(*) AS n FROM student_id_mapping WHERE run_id = {run}")[0]["n"] + rows = query(dataset, f"SELECT count(*) AS n FROM report_{run}")[0]["n"] if rows == 0: raise RuntimeError("no report-service access to this class, or the class has no data") answers = query(dataset, f"SELECT count(*) AS n FROM run_answers WHERE run_id = {run}")[0]["n"] @@ -68,7 +68,7 @@ def main(): learners = query(dataset, f""" SELECT count(DISTINCT m.user_id) AS n FROM student_id_mapping m JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint - WHERE a.run_id = {run} AND m.run_id = {run}""")[0]["n"] + WHERE a.run_id = {run}""")[0]["n"] out = scope["output_dir"] with open(os.path.join(out, "display.md"), "w", encoding="utf-8") as handle: diff --git a/specs/REPORT-146-package-commands-and-dataset-views.md b/specs/REPORT-146-package-commands-and-dataset-views.md index 8c8c34f..1a23a5e 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views.md +++ b/specs/REPORT-146-package-commands-and-dataset-views.md @@ -16,12 +16,12 @@ Researchers get four `cc-data package` commands (`init`, `run`, `build`, `publis - **R2. Dropped (Doug, 2026-10-08): no `learner_endpoints` view.** It would have unioned each Student Answers CSV's `res__remote_endpoint` columns, so it is empty for a Student ID Mapping run and rests on the route REPORT-147 measured as failing on real classes. Nothing uses it, so nothing builds it. - **R3. The documented counts.** For a run whose answers were fetched: - answers: `SELECT count(*) FROM run_answers WHERE run_id = `; - - learners with at least one answer, for a Student ID Mapping run: `SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = AND m.run_id = `, by `user_id` and never by endpoint, since a student has one endpoint per assignment. + - learners with at least one answer, for a Student ID Mapping run: `SELECT count(DISTINCT m.user_id) FROM student_id_mapping m JOIN run_answers a ON a.remote_endpoint = m.run_remote_endpoint WHERE a.run_id = `, by `user_id` and never by endpoint, since a student has one endpoint per assignment. It is scoped by the answers' run only: `student_id_mapping` keeps each learner's row from whichever mapping run was fetched last, so filtering it by `run_id` drops a learner another mapping run of the class also holds. The tests run both on a synthetic dataset built so that each wrong answer differs from the right one. It has: - a learner with answers in two assignments, so counting endpoints gives 2 where counting users gives 1; - a learner with a bare `.../` endpoint, who must not join; - - an answer whose membership puts it in two runs, so it counts once per run; + - an answer whose membership puts it in two runs, so it counts once per run, the second run being a later mapping run of the same class that holds that learner's row in `student_id_mapping`; - a history membership for one of the answers, which an untyped join would count; - a second Student ID Mapping run for another class, which must not leak into the first run's counts. @@ -44,7 +44,7 @@ Researchers get four `cc-data package` commands (`init`, `run`, `build`, `publis - The stub is stdlib-only Python and is the smallest complete package. It reads `RD_SCOPE_FILE` and `RD_DATASET`. - **Runs.** It takes the researcher's newest Student ID Mapping run filtered to exactly the scope's class (from `cc-data reports list --json`: slug `student-id-mapping`, `report_filter.filters` equal to `["class"]` and `report_filter.class` equal to `[]`), as REPORT-147's R7 does. Otherwise it creates one with `cc-data reports create --report-slug student-id-mapping --report-filter '{"class":[]}'`. - **Pulls and counts.** It re-reads the run with `get report --refresh` (computed live, so learners who joined since appear) and pulls `get answers` on it, then counts with the R3 queries through `cc-data query`, never by locating dataset files. - - **Guard.** It keeps the empty-scope guard that section 13 places in package code: no `student_id_mapping` rows for the run is an error. + - **Guard.** It keeps the empty-scope guard that section 13 places in package code: no rows in the run's own `report_` is an error, which, unlike `student_id_mapping`, holds every learner of that run. - **Output.** It writes `display.md`, `summary.txt` and `counts.json` to the scope's `output_dir`. - **Why the stub creates runs.** On the VM its dataset starts empty, so a stub that only read an existing run would report nothing there. The VM path is what a package exists for. @@ -450,4 +450,23 @@ R17 states the rule. ### Assumptions the second self-review checked and kept **Context**: Raised in self-review (second round), each checked against the code or the live sources. -**Decision**: Found sound: the R3 test (dropping `m.run_id = ` fails it at run 800, 2 learners against 1); a second `get answers` on a fetched run re-fetches and merges new answers, so the stub stays current; `reports list --json`'s `report_filter` carries `filters` and `class` as the stub reads them; `query --format json` answers an array of objects; report-server's 270-second wait sits between the deriver's 240 and cc-data's 300; an old report-server answers both new routes with a JSON 404 even without a token; and no publish check refuses an ordinary user's own origin, so `run`'s validate call cannot lock out a researcher who may use the API. +**Decision**: Found sound: a second `get answers` on a fetched run re-fetches and merges new answers, so the stub stays current; `reports list --json`'s `report_filter` carries `filters` and `class` as the stub reads them; `query --format json` answers an array of objects; report-server's 270-second wait sits between the deriver's 240 and cc-data's 300; an old report-server answers both new routes with a JSON 404 even without a token; and no publish check refuses an ordinary user's own origin, so `run`'s validate call cannot lock out a researcher who may use the API. + +--- + +### The documented learner count filtered the deduplicated mapping by run +**Context**: Raised in Copilot's review of PR #18. `student_id_mapping` keeps one row per learner across every mapping run, latest fetch winning (`TestPerRunDuplicateCheckIsNotConfusedByLaterRuns` already showed that filtering it by an older `run_id` loses learners). The count's `AND m.run_id = ` therefore dropped a learner whom a later-fetched mapping run of the same class also held, and the `init` stub's empty-scope guard, which counted `student_id_mapping` rows for the run, could fail on a class with data. +**Options considered**: +- A) A new per-run mapping view, deduplicated within each run only. +- B) Count over `report_`, the raw per-run CSV, repeating the bare-endpoint rule in the query. +- C) Drop the mapping-side filter: the answers are already scoped by `a.run_id`, a run's answers fetch holds only its own learners' answers, and an endpoint maps to one learner whichever mapping run's row won. + +**Decision**: C for the count, and the guard counts `report_`, which the stub has just pulled. No new view, and the bare-endpoint rule stays in `student_id_mapping` alone. The test gains a later mapping run of the same class, so scoping the mapping by run fails it. + +--- + +### Copilot's other findings on PR #18 +**Context**: Raised in Copilot's review of PR #18. + +**Decision**: The default build zip is written to a temporary file and renamed into place, so a link already at that path is replaced rather than followed. The scope file must hold one JSON object and nothing after it. `summary.txt` and `counts.json` stay read under the display cap, since the runner reads them that way too, and the code now says so. + From c6f1d6a5717e2792264fcf4ce2d068bc5fbfa1b7 Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Fri, 9 Oct 2026 09:34:22 -0400 Subject: [PATCH 14/17] fix: address Ethan McElroy's review of the package work [REPORT-146] - .cc-data-run and every folder package run creates in it are 0700, tightened if an earlier run left them wider, and scope.json and .gitignore are 0600, as every other student-data folder is. - package run makes the data root absolute before the package sees it, since the package runs from its staged copy. - build's --out refuses a directory and any path inside .cc-data-run. - A package folder that is itself a link is resolved before it is walked; links inside it are still refused. - When the running binary is named cc-data, its folder goes first on the package's PATH, so the package calls the same cc-data; under any other name, run warns which one the package will reach. - init reports the real error when it cannot check the folder, and build renames its zip through fsutil.RenameAtomic. - RouteMissing's comment records that validate and applies never answer 404 themselves. - release.yml ignores build metadata after "+" when it classifies a tag, and a test now runs that step against sample tags. New tests cover the three package error codes, a package that fails or writes no display.md, publish sending once when the connection drops, an applies 503 through package run, validate running before the run, HOME, the run folder's modes, and the learner query staying the same in the test, the guidance and the stub. The leftover-process test checks that the process is dead rather than racing a timed write. Comments and help text that overstated behavior are corrected, Files.Bytes is removed, and the stub ignores only "already exists". --- .github/workflows/release.yml | 2 +- cmd/package.go | 33 ++++- cmd/package_test.go | 133 +++++++++++++++--- docs/researcher-guide.md | 10 +- internal/api/packages.go | 5 +- internal/duck/run_views_test.go | 21 +++ internal/guidance/src/core.md | 7 +- internal/packages/archive.go | 14 +- internal/packages/build.go | 15 +- internal/packages/files_test.go | 17 +++ internal/packages/init.go | 6 +- internal/packages/proc_windows.go | 3 +- internal/packages/result.go | 6 +- internal/packages/run.go | 46 ++++-- internal/packages/run_test.go | 66 ++++++--- internal/packages/run_unix_test.go | 56 ++++++++ internal/packages/template/run.py | 2 +- release_test.go | 30 +++- ...-146-package-commands-and-dataset-views.md | 24 +++- 19 files changed, 414 insertions(+), 82 deletions(-) create mode 100644 internal/packages/run_unix_test.go diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index eee552e..129165d 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -175,7 +175,7 @@ jobs: - name: Classify the tag id: tag run: | - if [[ "${GITHUB_REF_NAME}" == *-* ]]; then + if [[ "${GITHUB_REF_NAME%%+*}" == *-* ]]; then echo "prerelease=true" >> "${GITHUB_OUTPUT}" else echo "prerelease=false" >> "${GITHUB_OUTPUT}" diff --git a/cmd/package.go b/cmd/package.go index d2df5bc..73c5c07 100644 --- a/cmd/package.go +++ b/cmd/package.go @@ -21,7 +21,8 @@ import ( const runDifferences = `What run cannot reproduce from the VM: the package runs as you, with no network sandbox, and with HOME, PATH, the locale and any proxy settings passed through so its own cc-data calls find your login and reach the server. A .py entrypoint runs with python3.11 -when it is on PATH, as on the VM, else python3.` +when it is on PATH, as on the VM, else python3. A clue_prepull package is refused, since +cc-data cannot fetch CLUE data yet.` const packageLong = `Develop, test and publish a Researcher Dashboard package. @@ -121,6 +122,11 @@ func (f packageRunFlags) load(dir string) (loadedPackage, error) { // run validates, runs and reports a loaded package; it takes the client so it can be exercised // against a test server without a stored credential. func (f packageRunFlags) run(ctx context.Context, client *api.Client, pkg loadedPackage, ref dataset.Ref, dataRoot string) error { + // The package runs from its staged copy, where a relative root would name another folder. + dataRoot, err := filepath.Abs(dataRoot) + if err != nil { + return output.Internalf("%v", err) + } archive, err := packages.Zip(pkg.files) if err != nil { return output.Internalf("%v", err) @@ -130,11 +136,16 @@ func (f packageRunFlags) run(ctx context.Context, client *api.Client, pkg loaded if _, err := validate(ctx, client, archive, "", false); err != nil { return err } + exe, _ := os.Executable() + binDir, warning := packageCCData(exe, exec.LookPath) + if warning != "" { + output.Warnf("%s", warning) + } start := time.Now() res, err := packages.Run(ctx, packages.RunOptions{ Dir: pkg.dir, Manifest: pkg.manifest, Files: pkg.files, Scope: pkg.scope, Applies: appliesFor(client, pkg.scope), Dataset: ref.String(), Name: ref.Name, Portal: ref.Portal.Host(), DataRoot: dataRoot, - ReportSrv: client.BaseURL, Stderr: output.Stderr(), Environ: f.environ, LookPath: exec.LookPath, + ReportSrv: client.BaseURL, BinDir: binDir, Stderr: output.Stderr(), Environ: f.environ, LookPath: exec.LookPath, }) if err != nil { return packageRunError(err) @@ -145,11 +156,25 @@ func (f packageRunFlags) run(ctx context.Context, client *api.Client, pkg loaded }) } +// packageCCData picks the cc-data a package's own calls reach: this binary, by putting its +// folder first on the package's PATH, when it is named cc-data. A binary with another name (go +// run, a test) cannot be reached by name, so the warning names the cc-data the package will call. +func packageCCData(exe string, lookPath func(string) (string, error)) (binDir, warning string) { + if exe != "" && strings.TrimSuffix(filepath.Base(exe), ".exe") == "cc-data" { + return filepath.Dir(exe), "" + } + found, err := lookPath("cc-data") + if err != nil { + return "", "this binary is not named cc-data and no cc-data is on PATH, so the package's own cc-data calls will fail" + } + return "", fmt.Sprintf("this binary is not named cc-data, so the package's own cc-data calls reach %s", found) +} + func newPackageRunCmd() *cobra.Command { f := packageRunFlags{environ: os.Environ} cmd := &cobra.Command{ Use: "run [dir] --dataset --scope ", - Short: "Run a package locally exactly as the dashboard's runner will", + Short: "Run a package locally under the dashboard runner's rules", Long: "Run a package against your own dataset under the runner's rules: the same scope.json,\n" + "environment variables, output files, display.md cap and applicability check.\n\n" + "--scope names a file holding the scope's kind, id, classes and assignments.\n" + @@ -269,7 +294,7 @@ func (f packageBuildFlags) run(ctx context.Context, client *api.Client, dir stri return &output.CLIError{ExitCode: output.ExitUsage, Code: "INVALID_MANIFEST", Message: fmt.Sprintf("the manifest's name and version make %q, which is not a file name; fix them or pass --out", name)} } out = filepath.Join(dir, packages.BuildDirName, name) - } else if err := packages.CheckOutsidePackage(dir, out); err != nil { + } else if err := packages.CheckBuildOutput(dir, out); err != nil { return output.Usagef("%v", err) } archive, err := packages.Zip(files) diff --git a/cmd/package_test.go b/cmd/package_test.go index 45ae365..2f432fe 100644 --- a/cmd/package_test.go +++ b/cmd/package_test.go @@ -10,10 +10,12 @@ import ( "net/http" "net/http/httptest" "os" + "os/exec" "path/filepath" "runtime" "strings" "sync" + "sync/atomic" "testing" "github.com/concord-consortium/cc-data-cli/internal/api" @@ -140,6 +142,13 @@ func TestPackageInitRefusesASecondInit(t *testing.T) { if got, _ := os.ReadFile(filepath.Join(dir, "run.py")); string(got) != "mine" { t.Errorf("the second init overwrote run.py: %q", got) } + notADir := filepath.Join(t.TempDir(), "a-file") + if err := os.WriteFile(notADir, nil, 0o644); err != nil { + t.Fatal(err) + } + if _, err := packages.Init(notADir, ""); err == nil || strings.Contains(err.Error(), "already exists") { + t.Errorf("init into a regular file: err = %v, want the real error", err) + } if _, err := packages.Init(filepath.Join(t.TempDir(), "Not_A_Name"), ""); err == nil { t.Error("init accepted a directory name outside the grammar without --name") } @@ -312,6 +321,12 @@ func TestPackageBuildRefusesAnOutputItWouldCollect(t *testing.T) { if cliErr := asCLIError(t, err); cliErr.ExitCode != output.ExitUsage { t.Errorf("err = %+v", cliErr) } + for _, out := range []string{dir, filepath.Join(dir, packages.RunDirName, "pkg", "p.zip")} { + err := (packageBuildFlags{out: out}).run(context.Background(), api.New(srv.URL, "tok"), dir) + if cliErr := asCLIError(t, err); cliErr.ExitCode != output.ExitUsage { + t.Errorf("--out %s: err = %+v, want a usage error", out, cliErr) + } + } if err := (packageBuildFlags{out: filepath.Join(dir, ".builds", "p.zip")}).run(context.Background(), api.New(srv.URL, "tok"), dir); err != nil { t.Errorf("an --out under a dot directory was refused: %v", err) } @@ -347,17 +362,42 @@ func TestPackageBuildRefusesANameThatIsAPath(t *testing.T) { } } +const testScopeJSON = `{"kind": "class", "id": "h", + "classes": [{"class_hash": "0123456789abcdef0123456789abcdef0123456789abcdef", "class_id": 6}], + "assignments": [{"offering_id": 9, "runnable_id": 1234, "name": "Wildfire", "url": "https://activity-player.concord.org/?sequence=830"}]}` + func testScope(t *testing.T) packages.LocalScope { t.Helper() - scope, err := packages.ParseLocalScope([]byte(`{"kind": "class", "id": "h", - "classes": [{"class_hash": "0123456789abcdef0123456789abcdef0123456789abcdef", "class_id": 6}], - "assignments": [{"offering_id": 9, "runnable_id": 1234, "name": "Wildfire", "url": "https://activity-player.concord.org/?sequence=830"}]}`)) + scope, err := packages.ParseLocalScope([]byte(testScopeJSON)) if err != nil { t.Fatal(err) } return scope } +// loadRunPackage writes a package of the given manifest and files and a scope file, and loads +// them as package run does before it looks up a credential. +func loadRunPackage(t *testing.T, manifest string, files map[string]string, environ func() []string) (packageRunFlags, loadedPackage) { + t.Helper() + dir := t.TempDir() + files["manifest.json"] = manifest + for name, content := range files { + if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o755); err != nil { + t.Fatal(err) + } + } + scopePath := filepath.Join(t.TempDir(), "scope.json") + if err := os.WriteFile(scopePath, []byte(testScopeJSON), 0o644); err != nil { + t.Fatal(err) + } + f := packageRunFlags{scopePath: scopePath, environ: environ} + pkg, err := f.load(dir) + if err != nil { + t.Fatal(err) + } + return f, pkg +} + func TestPackageRunAsksTheServersMatcher(t *testing.T) { srv := newPackageServer(t, map[string]func(http.ResponseWriter, []byte){ "/api/v1/packages/applies": answer(http.StatusOK, `{"applies": false, "reason": "no URL in this class matches the required pattern *//wildfire.concord.org/*", "interactive_urls": [], "unread": [], "truncated": false}`), @@ -517,7 +557,9 @@ func TestPackagePublishMapsAuthAndUnansweredFailures(t *testing.T) { t.Errorf("NOT_AUTHENTICATED: %+v", cliErr) } + var attempts atomic.Int32 dropped := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + attempts.Add(1) conn, _, _ := w.(http.Hijacker).Hijack() conn.Close() })) @@ -526,36 +568,39 @@ func TestPackagePublishMapsAuthAndUnansweredFailures(t *testing.T) { if cliErr := asCLIError(t, err); !strings.Contains(cliErr.Action, "Publishing again is safe") { t.Errorf("unanswered: %+v", cliErr) } + if n := attempts.Load(); n != 1 { + t.Errorf("an unanswered publish was sent %d times, want 1", n) + } } func TestPackageRunValidatesThenRunsWithTheRunnersNames(t *testing.T) { if runtime.GOOS == "windows" { t.Skip("the entrypoint is a shell script") } - dir := t.TempDir() manifest := strings.Replace(testManifest, `"entrypoint": "run.py"`, `"entrypoint": "run.sh"`, 1) script := "#!/bin/sh\nenv > \"$RD_OUTPUT_DIR/../env.txt\"\ncp \"$RD_SCOPE_FILE\" \"$RD_OUTPUT_DIR/../scope.json\"\necho ok > \"$RD_OUTPUT_DIR/display.md\"\n" - for name, content := range map[string]string{"manifest.json": manifest, "run.sh": script} { - if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o755); err != nil { - t.Fatal(err) + f, pkg := loadRunPackage(t, manifest, map[string]string{"run.sh": script}, func() []string { return []string{"PATH=" + os.Getenv("PATH")} }) + dir := pkg.dir + ranBeforeValidate := false + srv := newPackageServer(t, map[string]func(http.ResponseWriter, []byte){"/api/v1/packages/validate": func(w http.ResponseWriter, body []byte) { + if _, err := os.Stat(filepath.Join(dir, packages.RunDirName)); err == nil { + ranBeforeValidate = true } - } - scopePath := filepath.Join(t.TempDir(), "scope.json") - if err := os.WriteFile(scopePath, []byte(`{"kind": "class", "id": "h", - "classes": [{"class_hash": "0123456789abcdef0123456789abcdef0123456789abcdef", "class_id": 6}], "assignments": []}`), 0o644); err != nil { - t.Fatal(err) - } - f := packageRunFlags{scopePath: scopePath, environ: func() []string { return []string{"PATH=" + os.Getenv("PATH")} }} - pkg, err := f.load(dir) + answer(http.StatusOK, validated)(w, body) + }}) + out, _ := capture(t) + ref := dataset.Ref{Portal: config.MustPortal("learn.concord.org"), Name: "wildfire"} + t.Chdir(t.TempDir()) + cwd, err := os.Getwd() if err != nil { t.Fatal(err) } - srv := newPackageServer(t, validateAnswering(validated)) - out, _ := capture(t) - ref := dataset.Ref{Portal: config.MustPortal("learn.concord.org"), Name: "wildfire"} - if err := f.run(context.Background(), api.New(srv.URL, "tok"), pkg, ref, "/data/root"); err != nil { + if err := f.run(context.Background(), api.New(srv.URL, "tok"), pkg, ref, "data-root"); err != nil { t.Fatal(err) } + if ranBeforeValidate { + t.Error("the run tree existed before validate answered") + } reqs := srv.requests("/api/v1/packages/validate") if len(reqs) != 1 || reqs[0].contentType != "application/zip" || len(reqs[0].body) == 0 { @@ -566,7 +611,7 @@ func TestPackageRunValidatesThenRunsWithTheRunnersNames(t *testing.T) { if err != nil { t.Fatal(err) } - for _, want := range []string{"RD_DATASET=learn.concord.org/wildfire\n", "CC_DATA_PORTAL=learn.concord.org\n", "CC_DATA_ROOT=/data/root\n", "RD_REPORT_SERVER_URL=" + srv.URL + "\n"} { + for _, want := range []string{"RD_DATASET=learn.concord.org/wildfire\n", "CC_DATA_PORTAL=learn.concord.org\n", "CC_DATA_ROOT=" + filepath.Join(cwd, "data-root") + "\n", "RD_REPORT_SERVER_URL=" + srv.URL + "\n"} { if !strings.Contains(string(env), want) { t.Errorf("environment lacks %q", strings.TrimSpace(want)) } @@ -589,3 +634,51 @@ func TestPackageRunValidatesThenRunsWithTheRunnersNames(t *testing.T) { } } } + +func TestPackageRunPassesAnAppliesFailureThrough(t *testing.T) { + manifest := strings.Replace(testManifest, `"all": []`, `"all": ["*//wildfire.concord.org/*"]`, 1) + f, pkg := loadRunPackage(t, manifest, map[string]string{"run.py": "print(1)\n"}, func() []string { return nil }) + srv := newPackageServer(t, map[string]func(http.ResponseWriter, []byte){ + "/api/v1/packages/validate": answer(http.StatusOK, validated), + "/api/v1/packages/applies": answer(http.StatusServiceUnavailable, `{"error":"SERVICE_UNAVAILABLE","message":"the deriver is unavailable"}`), + }) + capture(t) + ref := dataset.Ref{Portal: config.MustPortal("learn.concord.org"), Name: "wildfire"} + err := f.run(context.Background(), api.New(srv.URL, "tok"), pkg, ref, "/data/root") + if cliErr := asCLIError(t, err); cliErr.Code != "SERVICE_UNAVAILABLE" || cliErr.ExitCode != output.ExitContract { + t.Errorf("err = %+v", cliErr) + } +} + +func TestPackageRunErrorsAreTheRunnersCodesAtExitOne(t *testing.T) { + for _, tc := range []struct { + err error + code string + }{ + {&packages.Refused{Reason: "does not apply"}, "PACKAGE_REFUSED"}, + {&packages.Failed{Reason: "exited 3"}, "PACKAGE_FAILED"}, + {&packages.OutputRefused{Reason: "no display.md"}, "PACKAGE_OUTPUT_REFUSED"}, + } { + got := asCLIError(t, packageRunError(tc.err)) + if got.Code != tc.code || got.ExitCode != output.ExitInternal || got.Message != tc.err.Error() { + t.Errorf("%T: got %s exit %d %q", tc.err, got.Code, got.ExitCode, got.Message) + } + } +} + +func TestPackageRunReachesThisCCData(t *testing.T) { + none := func(string) (string, error) { return "", exec.ErrNotFound } + other := func(string) (string, error) { return "/opt/homebrew/bin/cc-data", nil } + if dir, warning := packageCCData(filepath.Join("/build", "cc-data"), other); dir != "/build" || warning != "" { + t.Errorf("a binary named cc-data: %q, %q", dir, warning) + } + if dir, warning := packageCCData(filepath.Join("/build", "cc-data.exe"), other); dir != "/build" || warning != "" { + t.Errorf("cc-data.exe: %q, %q", dir, warning) + } + if dir, warning := packageCCData("/tmp/go-build/exe/main", other); dir != "" || !strings.Contains(warning, "/opt/homebrew/bin/cc-data") { + t.Errorf("go run: %q, %q", dir, warning) + } + if _, warning := packageCCData("/tmp/go-build/exe/main", none); !strings.Contains(warning, "will fail") { + t.Errorf("no cc-data on PATH: %q", warning) + } +} diff --git a/docs/researcher-guide.md b/docs/researcher-guide.md index 9521844..1156d1f 100644 --- a/docs/researcher-guide.md +++ b/docs/researcher-guide.md @@ -641,7 +641,9 @@ What it cannot reproduce: its own user behind an egress proxy. - **A few of your variables pass through.** `HOME`, `PATH`, your locale, the proxy settings and a handful of others reach the package, so its own `cc-data` - calls find your login. Nothing else from your environment does. + calls find your login. Nothing else from your environment does. The folder of + the `cc-data` running `package run` goes first on the package's `PATH`, so the + package calls that same version rather than an older one installed elsewhere. - **Python.** A `.py` entrypoint runs with `python3.11` when it is on your `PATH`, as on the VM, and otherwise with `python3` after a note saying so. - **CLUE.** A package with `clue_prepull: true` is refused locally, since @@ -672,9 +674,9 @@ configured for " even though the zip itself passed every check. `.cc-data-run/` (the last run's staged copy, inputs, outputs and the package's data directory) and `.cc-data-build/` (built zips) each carry a `.gitignore` -of `*`, so git never commits them. **Both can hold student data**: `display.md` -is drawn from it. `build` never ships either, nor any other path starting with a -dot. +of `*`, so git never commits them. **`.cc-data-run/` holds student data** +(`display.md` is drawn from it), so it is readable only by you, as datasets are. +`build` never ships either folder, nor any other path starting with a dot. ## 8. A few things to keep in mind diff --git a/internal/api/packages.go b/internal/api/packages.go index 24c6caa..ad5591f 100644 --- a/internal/api/packages.go +++ b/internal/api/packages.go @@ -99,8 +99,9 @@ func packageQuery(origin string, official bool) url.Values { return q } -// RouteMissing reports a 404 from a route this cc-data expects and the server does not have -// yet: a report-server older than the validate and applies routes. +// RouteMissing reports a 404 from the validate or applies route: a report-server older than +// those routes. Their contract is never to answer 404 themselves, refusing with 403 or 422 as +// publish does, so a 404 cannot be a refusal. func RouteMissing(err error) bool { var apiErr *APIError return errors.As(err, &apiErr) && apiErr.Status == http.StatusNotFound diff --git a/internal/duck/run_views_test.go b/internal/duck/run_views_test.go index 3ada34f..58ba696 100644 --- a/internal/duck/run_views_test.go +++ b/internal/duck/run_views_test.go @@ -4,6 +4,8 @@ import ( "context" "encoding/json" "fmt" + "os" + "strings" "testing" "time" @@ -93,3 +95,22 @@ func TestRunAnswersRunIDIsBigint(t *testing.T) { } } } + +// The guidance and the init stub each carry the learner count as text; both must stay the +// query the test above checks. +func TestTheDocumentedLearnerCountIsTheTestedOne(t *testing.T) { + flat := func(path string) string { + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + return strings.Join(strings.Fields(string(raw)), " ") + } + if want := "`" + fmt.Sprintf(learnersOfRun, 584) + "`"; !strings.Contains(flat("../guidance/src/core.md"), want) { + t.Errorf("core.md no longer documents %q", want) + } + stub := strings.ReplaceAll(flat("../packages/template/run.py"), " AS n FROM", " FROM") + if want := strings.ReplaceAll(learnersOfRun, "%d", "{run}") + `"""`; !strings.Contains(stub, want) { + t.Errorf("the init stub no longer runs %q", want) + } +} diff --git a/internal/guidance/src/core.md b/internal/guidance/src/core.md index 28625bc..50e7008 100644 --- a/internal/guidance/src/core.md +++ b/internal/guidance/src/core.md @@ -137,9 +137,10 @@ like `wildfire_2026.answers`): fetched. Scope it by the answers' `run_id` only: `student_id_mapping` keeps each learner's row from whichever run was fetched last, so `m.run_id = 584` drops a learner another mapping run of the class also holds. Count by `user_id`, never - by endpoint, since a student has one endpoint per assignment. A Student ID Mapping run is computed live, so re-reading it with - `--refresh` picks up learners who joined since; a Student Answers run is fixed - when its query ran, and a whole-class one fails above three or four assignments. + by endpoint, since a student has one endpoint per assignment. A Student ID + Mapping run is computed live, so re-reading it with `--refresh` picks up + learners who joined since; a Student Answers run is fixed when its query ran, + and a whole-class one fails above three or four assignments. - A run's log freshness is `SELECT count(*) AS logs, max(event_time) AS log_freshness_at FROM logs WHERE run_id = `. - Reports-to-stores join: `reports.res__remote_endpoint = diff --git a/internal/packages/archive.go b/internal/packages/archive.go index 08a2619..df285ee 100644 --- a/internal/packages/archive.go +++ b/internal/packages/archive.go @@ -27,7 +27,6 @@ type Files struct { Dir string Paths []string Modes map[string]os.FileMode - Bytes int64 } // shipMode is the one mode a file keeps through build and unzip: 0755 when any execute bit is @@ -58,11 +57,17 @@ func excluded(rel string, isDir bool) bool { return !isDir && strings.HasSuffix(base, ".pyc") } -// Collect walks dir for the files a package is made of. A link or other non-regular file is -// refused rather than skipped, since the runner refuses an archive holding one. +// Collect walks dir for the files a package is made of. A link or other non-regular file that +// build would ship is refused rather than skipped, since the runner refuses an archive holding +// one; under an excluded name it is skipped like any other file. func Collect(dir string) (Files, error) { + // The package directory itself may be a link; only what is inside it may not. + dir, err := filepath.EvalSymlinks(dir) + if err != nil { + return Files{}, err + } files := Files{Dir: dir, Modes: map[string]os.FileMode{}} - err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error { + err = filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error { if err != nil { return err } @@ -94,7 +99,6 @@ func Collect(dir string) (Files, error) { } files.Paths = append(files.Paths, rel) files.Modes[rel] = shipMode(info.Mode()) - files.Bytes += info.Size() return nil }) sort.Strings(files.Paths) diff --git a/internal/packages/build.go b/internal/packages/build.go index c8d1800..03be3cc 100644 --- a/internal/packages/build.go +++ b/internal/packages/build.go @@ -5,14 +5,20 @@ import ( "os" "path/filepath" "strings" + + "github.com/concord-consortium/cc-data-cli/internal/fsutil" ) // BuildDirName is where build writes by default: inside the package, and skipped by // Collect like every dot path, so one build's zip never reaches the next. const BuildDirName = ".cc-data-build" -// CheckOutsidePackage refuses an explicit build output that a later build would collect. -func CheckOutsidePackage(dir, out string) error { +// CheckBuildOutput refuses an explicit build output that is a directory, that package run would +// empty, or that a later build would collect. +func CheckBuildOutput(dir, out string) error { + if info, err := os.Stat(out); err == nil && info.IsDir() { + return fmt.Errorf("--out %s is a directory; name the zip file to write", out) + } absDir, err := filepath.Abs(dir) if err != nil { return err @@ -25,6 +31,9 @@ func CheckOutsidePackage(dir, out string) error { if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { return nil } + if strings.SplitN(filepath.ToSlash(rel), "/", 2)[0] == RunDirName { + return fmt.Errorf("--out %s is inside %s, which package run empties", out, RunDirName) + } if excluded(filepath.ToSlash(rel), false) || excludedAncestor(filepath.ToSlash(rel)) { return nil } @@ -78,5 +87,5 @@ func replaceFile(path string, data []byte) error { if err := tmp.Close(); err != nil { return err } - return os.Rename(tmp.Name(), path) + return fsutil.RenameAtomic(tmp.Name(), path) } diff --git a/internal/packages/files_test.go b/internal/packages/files_test.go index c7cb4bb..ec513d3 100644 --- a/internal/packages/files_test.go +++ b/internal/packages/files_test.go @@ -224,3 +224,20 @@ func TestResultRefusesALinkedDisplay(t *testing.T) { t.Errorf("err = %v, want OutputRefused", err) } } + +func TestCollectFollowsALinkedPackageDirectory(t *testing.T) { + skipOnWindows(t, "links need privileges on Windows") + real := t.TempDir() + writeTree(t, real, map[string]string{"manifest.json": goodManifest, "run.py": "print(1)"}) + link := filepath.Join(t.TempDir(), "pkg") + if err := os.Symlink(real, link); err != nil { + t.Fatal(err) + } + files, err := Collect(link) + if err != nil { + t.Fatal(err) + } + if got := strings.Join(files.Paths, ","); got != "manifest.json,run.py" { + t.Errorf("collected %q through a linked package directory", got) + } +} diff --git a/internal/packages/init.go b/internal/packages/init.go index 06dfea8..1d2730a 100644 --- a/internal/packages/init.go +++ b/internal/packages/init.go @@ -28,8 +28,12 @@ func Init(dir, name string) ([]string, error) { return nil, fmt.Errorf("--name %q must match ^[a-z0-9][a-z0-9-]{0,62}$", name) } for _, f := range []string{"manifest.json", "run.py"} { - if _, err := os.Lstat(filepath.Join(dir, f)); !errors.Is(err, os.ErrNotExist) { + _, err := os.Lstat(filepath.Join(dir, f)) + switch { + case err == nil: return nil, fmt.Errorf("%s already exists in %s; init writes only into a package that has neither file", f, dir) + case !errors.Is(err, os.ErrNotExist): + return nil, err } } manifest := Manifest{ diff --git a/internal/packages/proc_windows.go b/internal/packages/proc_windows.go index 3b2ad4c..0dead87 100644 --- a/internal/packages/proc_windows.go +++ b/internal/packages/proc_windows.go @@ -4,7 +4,8 @@ package packages import "os/exec" -// killGroup and reapGroup kill only the entrypoint on Windows, which has no release. +// On Windows, which has no release, exec.CommandContext kills only the entrypoint, and nothing +// is killed after it exits. func killGroup(cmd *exec.Cmd) {} func reapGroup(cmd *exec.Cmd) {} diff --git a/internal/packages/result.go b/internal/packages/result.go index a27daaa..decd6a8 100644 --- a/internal/packages/result.go +++ b/internal/packages/result.go @@ -11,8 +11,8 @@ import ( "strings" ) -// DisplayLimitBytes is the runner's cap on display.md in UTF-8 bytes, decided only in -// final-design.md section 10. +// DisplayLimitBytes is the runner's cap on display.md in UTF-8 bytes, decided only in section 10 +// of the Researcher Dashboard design (final-design.md, kept outside this repository). const DisplayLimitBytes = 64 * 1024 // countKeys are the counts.json keys the runner reads; anything else a package writes there @@ -50,8 +50,8 @@ func ReadResult(outDir string) (Result, error) { return res, &OutputRefused{Reason: fmt.Sprintf("the package's display.md is %d bytes, over the %d-byte limit", size, DisplayLimitBytes)} } res.DisplayPath = filepath.Join(outDir, "display.md") - // The runner reads summary.txt and counts.json under the display cap too, truncating them. res.DisplayBytes = size + // The runner reads summary.txt and counts.json under the display cap too, truncating them. if text, _, err := readOwnFile(outDir, "summary.txt", DisplayLimitBytes); err == nil { line := strings.TrimSpace(strings.SplitN(strings.ReplaceAll(text, "\r\n", "\n"), "\n", 2)[0]) if line != "" { diff --git a/internal/packages/run.go b/internal/packages/run.go index b9643a9..58b37d6 100644 --- a/internal/packages/run.go +++ b/internal/packages/run.go @@ -13,6 +13,8 @@ import ( "sort" "strings" "time" + + "github.com/concord-consortium/cc-data-cli/internal/fsutil" ) // TimeoutMargin is what the runner adds to expected_duration_seconds before it kills a run. @@ -23,7 +25,8 @@ const TimeoutMargin = 10 * time.Minute const RunDirName = ".cc-data-run" // passthrough is what a laptop adds so the package's own cc-data calls find the stored -// credential, reach the server through any proxy, and run at all. Matching is case-insensitive. +// credential, reach the server through any proxy, and run at all. These names match in any +// case, so the lower-case proxy spellings pass; the LC_ prefix matches as written. var passthrough = []string{ "HOME", "PATH", "USER", "LOGNAME", "LANG", "TZ", "TMPDIR", "DBUS_SESSION_BUS_ADDRESS", "XDG_RUNTIME_DIR", @@ -61,8 +64,9 @@ type RunOptions struct { Dataset string // the full ref, / Name string // the ref's bare name Portal string // the ref's portal host - DataRoot string // CC_DATA_ROOT as package run resolved it + DataRoot string // CC_DATA_ROOT, absolute ReportSrv string // the stored credential's server, or "" + BinDir string // put first on PATH, so the package calls this cc-data Stderr io.Writer // where the package's own output goes Environ func() []string LookPath func(string) (string, error) @@ -96,7 +100,7 @@ func Run(ctx context.Context, o RunOptions) (Result, error) { return Result{}, err } scopePath := filepath.Join(paths.in, "scope.json") - if err := os.WriteFile(scopePath, b, 0o644); err != nil { + if err := fsutil.WriteFileAtomic0600(scopePath, b); err != nil { return Result{}, err } @@ -128,12 +132,12 @@ func Run(ctx context.Context, o RunOptions) (Result, error) { defer cancel() cmd := exec.CommandContext(runCtx, command, args...) cmd.Dir = paths.pkg - cmd.Env = environment(o.Environ(), env) + cmd.Env = environment(o.Environ(), env, o.BinDir) cmd.Stdout = o.Stderr cmd.Stderr = o.Stderr killGroup(cmd) err = cmd.Run() - // As on the VM, nothing the package started outlives it or writes after it. + // On Unix, whatever is left in the package's process group dies before its output is read. reapGroup(cmd) if err != nil { switch { @@ -149,8 +153,9 @@ func Run(ctx context.Context, o RunOptions) (Result, error) { type runPaths struct{ pkg, in, out, data string } -// prepare empties and rebuilds the run tree. Every directory it removes or creates under the -// package must be a real directory, so a link left there can never aim the removal outside. +// prepare empties and rebuilds the run tree, keeping only data/ from one run to the next. Every +// directory it removes or creates under the package must be a real directory, so a link left +// there can never aim the removal outside. func prepare(run string, files Files) (runPaths, error) { p := runPaths{ pkg: filepath.Join(run, "pkg"), @@ -173,7 +178,7 @@ func prepare(run string, files Files) (runPaths, error) { return p, err } } - if err := os.Mkdir(d, 0o755); err != nil { + if err := os.Mkdir(d, 0o700); err != nil { return p, err } } @@ -200,23 +205,25 @@ func ignoreAll(dir string) error { if info, err := os.Lstat(p); err == nil && !info.Mode().IsRegular() { return fmt.Errorf("refusing to write %s: it is a link or not a file", p) } - return os.WriteFile(p, []byte("*\n"), 0o644) + return fsutil.WriteFileAtomic0600(p, []byte("*\n")) } +// ensureRealDir creates d, or checks that it is a real directory, and keeps it private: what +// this tool writes inside a package can hold student data. func ensureRealDir(d string) error { if _, err := os.Lstat(d); errors.Is(err, os.ErrNotExist) { - return os.Mkdir(d, 0o755) + return os.Mkdir(d, 0o700) } if err := RealDir(d); err != nil { return fmt.Errorf("refusing to use %s: it is a link or not a directory", d) } - return nil + return os.Chmod(d, 0o700) } // copyFile stages one file with the mode build gives it, so the staged package is the one // unzip produces on the VM. func copyFile(src, dst string, mode os.FileMode) error { - if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { + if err := os.MkdirAll(filepath.Dir(dst), 0o700); err != nil { return err } in, err := os.Open(src) @@ -253,7 +260,7 @@ func entrypointCommand(pkgDir, entrypoint string, lookPath func(string) (string, // environment is the passthrough taken from the caller's environment (LC_* included), then // the runner's variables over it; nothing else is inherited. -func environment(from []string, set map[string]string) []string { +func environment(from []string, set map[string]string, binDir string) []string { keep := map[string]string{} for _, kv := range from { k, v, ok := strings.Cut(kv, "=") @@ -267,6 +274,19 @@ func environment(from []string, set map[string]string) []string { for k, v := range set { keep[k] = v } + if binDir != "" { + pathKey := "PATH" + for k := range keep { + if strings.EqualFold(k, "PATH") { + pathKey = k + } + } + if keep[pathKey] == "" { + keep[pathKey] = binDir + } else { + keep[pathKey] = binDir + string(os.PathListSeparator) + keep[pathKey] + } + } out := make([]string, 0, len(keep)) for k, v := range keep { out = append(out, k+"="+v) diff --git a/internal/packages/run_test.go b/internal/packages/run_test.go index 65e357e..5042cf7 100644 --- a/internal/packages/run_test.go +++ b/internal/packages/run_test.go @@ -105,7 +105,9 @@ echo ok > "$RD_OUTPUT_DIR/display.md" `, map[string]string{"local-data/x.csv": "secret"}) var stderr bytes.Buffer environ := []string{"PATH=" + os.Getenv("PATH"), "HOME=/home/r", "LC_ALL=C", "https_proxy=http://proxy:3128", "AWS_SECRET_ACCESS_KEY=leak"} - res, err := Run(context.Background(), runOpts(t, dir, files, environ, &stderr)) + o := runOpts(t, dir, files, environ, &stderr) + o.BinDir = "/opt/cc-data/bin" + res, err := Run(context.Background(), o) if err != nil { t.Fatalf("%v\n%s", err, stderr.String()) } @@ -125,6 +127,8 @@ echo ok > "$RD_OUTPUT_DIR/display.md" "RD_OUTPUT_DIR": filepath.Join(run, "out"), "RD_REPORT_SERVER_URL": "https://report.example", "LC_ALL": "C", + "HOME": "/home/r", + "PATH": "/opt/cc-data/bin" + string(os.PathListSeparator) + os.Getenv("PATH"), "https_proxy": "http://proxy:3128", } { if env[k] != want { @@ -189,6 +193,9 @@ func TestRunRefusesBeforeStarting(t *testing.T) { if _, err := Run(context.Background(), o); !errors.As(err, &refused) { t.Errorf("clue_prepull: err = %v, want Refused", err) } + if _, err := os.Stat(filepath.Join(dir, RunDirName, "started")); err == nil { + t.Error("the entrypoint started for a clue_prepull package") + } } func TestRunWarnsAndRunsWhenApplicabilityIsUnconfirmed(t *testing.T) { @@ -234,21 +241,6 @@ func TestRunKillsTheGroupWhenInterrupted(t *testing.T) { } } -func TestRunLeavesNothingBehind(t *testing.T) { - skipOnWindows(t, "the entrypoint is a shell script") - dir, files := shellPackage(t, `( sleep 1; echo late > "$RD_OUTPUT_DIR/late.txt" ) >/dev/null 2>&1 & -echo ok > "$RD_OUTPUT_DIR/display.md" -`, nil) - var stderr bytes.Buffer - if _, err := Run(context.Background(), runOpts(t, dir, files, []string{"PATH=" + os.Getenv("PATH")}, &stderr)); err != nil { - t.Fatal(err) - } - time.Sleep(1500 * time.Millisecond) - if _, err := os.Stat(filepath.Join(dir, RunDirName, "out", "late.txt")); err == nil { - t.Error("a background process the package left behind wrote after the run") - } -} - func TestRunRefusesALinkedRunDirectory(t *testing.T) { skipOnWindows(t, "links need privileges on Windows") dir, files := shellPackage(t, `echo ok > "$RD_OUTPUT_DIR/display.md"`+"\n", nil) @@ -347,3 +339,45 @@ func TestWriteBuildReplacesALinkedZipRatherThanFollowingIt(t *testing.T) { t.Errorf("build directory holds %v, want the zip and .gitignore only", entries) } } + +func TestRunKeepsItsTreePrivate(t *testing.T) { + skipOnWindows(t, "Windows has no POSIX mode bits") + dir, files := shellPackage(t, `echo ok > "$RD_OUTPUT_DIR/display.md"`+"\n", nil) + run := filepath.Join(dir, RunDirName) + if err := os.Mkdir(run, 0o755); err != nil { + t.Fatal(err) + } + var stderr bytes.Buffer + if _, err := Run(context.Background(), runOpts(t, dir, files, []string{"PATH=" + os.Getenv("PATH")}, &stderr)); err != nil { + t.Fatal(err) + } + for rel, want := range map[string]os.FileMode{ + ".": 0o700, "pkg": 0o700, "in": 0o700, "out": 0o700, "data": 0o700, + "in/scope.json": 0o600, ".gitignore": 0o600, + } { + info, err := os.Stat(filepath.Join(run, rel)) + if err != nil { + t.Fatal(err) + } + if got := info.Mode().Perm(); got != want { + t.Errorf("%s is %o, want %o", rel, got, want) + } + } +} + +func TestRunReportsAFailedPackageAndAMissingDisplay(t *testing.T) { + skipOnWindows(t, "the entrypoint is a shell script") + var stderr bytes.Buffer + dir, files := shellPackage(t, "exit 3\n", nil) + _, err := Run(context.Background(), runOpts(t, dir, files, []string{"PATH=" + os.Getenv("PATH")}, &stderr)) + var failed *Failed + if !errors.As(err, &failed) || !strings.Contains(err.Error(), "exit status 3") { + t.Errorf("a non-zero exit: err = %v, want Failed", err) + } + dir, files = shellPackage(t, "true\n", nil) + _, err = Run(context.Background(), runOpts(t, dir, files, []string{"PATH=" + os.Getenv("PATH")}, &stderr)) + var refused *OutputRefused + if !errors.As(err, &refused) || !strings.Contains(err.Error(), "no display.md") { + t.Errorf("no display.md: err = %v, want OutputRefused", err) + } +} diff --git a/internal/packages/run_unix_test.go b/internal/packages/run_unix_test.go new file mode 100644 index 0000000..efa9047 --- /dev/null +++ b/internal/packages/run_unix_test.go @@ -0,0 +1,56 @@ +//go:build !windows + +package packages + +import ( + "bytes" + "context" + "errors" + "os" + "path/filepath" + "strconv" + "strings" + "syscall" + "testing" + "time" +) + +// alive reports whether pid is a process that has not exited; a zombie awaiting its reaper +// counts as exited. +func alive(pid int) bool { + if err := syscall.Kill(pid, 0); errors.Is(err, syscall.ESRCH) { + return false + } + stat, err := os.ReadFile(filepath.Join("/proc", strconv.Itoa(pid), "stat")) + if err != nil { + return true + } + fields := strings.Fields(string(stat[bytes.LastIndexByte(stat, ')')+1:])) + return len(fields) == 0 || fields[0] != "Z" +} + +func TestRunLeavesNothingBehind(t *testing.T) { + dir, files := shellPackage(t, `sleep 30 >/dev/null 2>&1 & +echo $! > "$RD_OUTPUT_DIR/../bg.pid" +echo ok > "$RD_OUTPUT_DIR/display.md" +`, nil) + var stderr bytes.Buffer + if _, err := Run(context.Background(), runOpts(t, dir, files, []string{"PATH=" + os.Getenv("PATH")}, &stderr)); err != nil { + t.Fatal(err) + } + raw, err := os.ReadFile(filepath.Join(dir, RunDirName, "bg.pid")) + if err != nil { + t.Fatal(err) + } + pid, err := strconv.Atoi(strings.TrimSpace(string(raw))) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { syscall.Kill(pid, syscall.SIGKILL) }) + for deadline := time.Now().Add(5 * time.Second); alive(pid); { + if time.Now().After(deadline) { + t.Fatalf("the package's background process %d outlived the run", pid) + } + time.Sleep(20 * time.Millisecond) + } +} diff --git a/internal/packages/template/run.py b/internal/packages/template/run.py index 5974892..41455f0 100644 --- a/internal/packages/template/run.py +++ b/internal/packages/template/run.py @@ -54,7 +54,7 @@ def main(): try: cc_data("dataset", "create", dataset) except RuntimeError as err: - if "exists" not in str(err).lower(): + if "already exists" not in str(err).lower(): raise run = mapping_run(os.environ["CC_DATA_PORTAL"], class_id) cc_data("get", "report", str(run), "--dataset", dataset, "--refresh") diff --git a/release_test.go b/release_test.go index 8506558..9066749 100644 --- a/release_test.go +++ b/release_test.go @@ -2,7 +2,10 @@ package main import ( "os" + "os/exec" + "path/filepath" "regexp" + "runtime" "strings" "testing" ) @@ -127,7 +130,7 @@ func readFile(t *testing.T, path string) string { func TestReleaseWorkflowKeepsPreReleasesBack(t *testing.T) { rel := readFile(t, ".github/workflows/release.yml") for _, want := range []string{ - `if [[ "${GITHUB_REF_NAME}" == *-* ]]; then`, + "--prerelease=${{ steps.tag.outputs.prerelease }}", "if: steps.tag.outputs.prerelease == 'false'", } { @@ -136,3 +139,28 @@ func TestReleaseWorkflowKeepsPreReleasesBack(t *testing.T) { } } } + +// The tag classification is shell, so it is run here against the tags that matter: a +// pre-release, a release, and build metadata whose hyphen is not a pre-release. +func TestReleaseWorkflowClassifiesTags(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("the step is bash") + } + rel := readFile(t, ".github/workflows/release.yml") + _, after, ok := strings.Cut(rel, "id: tag\n run: |\n") + if !ok { + t.Fatal("release workflow has no tag-classification step") + } + script, _, _ := strings.Cut(after, "\n\n") + for tag, want := range map[string]string{"v0.3.0-pre.1": "true", "v0.3.0": "false", "v1.2.3+ci-build": "false"} { + out := filepath.Join(t.TempDir(), "output") + cmd := exec.Command("bash", "-c", script) + cmd.Env = append(os.Environ(), "GITHUB_REF_NAME="+tag, "GITHUB_OUTPUT="+out) + if b, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("%s: %v\n%s", tag, err, b) + } + if got := strings.TrimSpace(readFile(t, out)); got != "prerelease="+want { + t.Errorf("%s: %s, want prerelease=%s", tag, got, want) + } + } +} diff --git a/specs/REPORT-146-package-commands-and-dataset-views.md b/specs/REPORT-146-package-commands-and-dataset-views.md index 1a23a5e..257c952 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views.md +++ b/specs/REPORT-146-package-commands-and-dataset-views.md @@ -55,7 +55,7 @@ Researchers get four `cc-data package` commands (`init`, `run`, `build`, `publis - `--scope` names a local scope file. It holds the four keys rigse supplies on the VM: `kind`, `id`, `classes` (`[{class_hash, class_id}]`) and `assignments` (`[{offering_id, runnable_id, name, url}]`). The shape is checked; any other key is refused. - `package init` does not write a scope file, so the file comes from the author. - **R11. The package sees the runner's layout.** Before the entrypoint runs: - - **A fresh run directory** under `dir/.cc-data-run/`, with `in/` and `out/`, both emptied. `.cc-data-run` and everything `package run` empties or creates under it must be a real directory, never a link. A link is refused before anything is removed, so a stray link can never point the emptying at a directory outside the package, and a `.gitignore` that is a link is refused rather than written through. + - **A fresh run directory** under `dir/.cc-data-run/`, with `in/` and `out/`, both emptied. `.cc-data-run` and everything `package run` empties or creates under it must be a real directory, never a link. A link is refused before anything is removed, so a stray link can never point the emptying at a directory outside the package, and a `.gitignore` that is a link is refused rather than written through. Like every other folder cc-data writes student data into, `.cc-data-run` and the folders `package run` creates in it are 0700, tightened if an earlier run left them wider, and the files it writes there (`scope.json`, `.gitignore`) are 0600. - **A `.gitignore` containing `*`** written into `dir/.cc-data-run/`. A package directory is usually a git checkout (cc-data-studies ignores only `local-data/`, `__pycache__/` and `*.pyc`), and `out/display.md` holds counts or text drawn from student data. - **`in/scope.json`** in the runner's exact shape: the scope file's four keys; `clue_source` `"firebase"`; `dataset` as the ref's bare name, as the runner writes `pkg-`; and `output_dir` as the absolute `out/` path. - **The runner's environment:** @@ -63,10 +63,10 @@ Researchers get four `cc-data package` commands (`init`, `run`, `build`, `publis - `RD_SCOPE_FILE`. - `RD_OUTPUT_DIR`. - `RD_DATA_DIR` and `CC_DATA_LOCAL`: `dir/.cc-data-run/data/`. - - `CC_DATA_ROOT`: the data root `package run` itself resolved. + - `CC_DATA_ROOT`: the data root `package run` itself resolved, made absolute, since the package runs from its staged copy, where a relative root would name another folder. - `CC_DATA_PORTAL`: the ref's portal. - `RD_REPORT_SERVER_URL`: the portal's stored credential's server, when one is stored. - - **A short passthrough list a laptop needs.** It covers `HOME`, `PATH`, `USER`, `LOGNAME`, `LANG`, `LC_*`, `TZ`, `TMPDIR`, `DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR` and the proxy variables `HTTPS_PROXY`, `HTTP_PROXY` and `NO_PROXY` in either case (the runner sets the first two when it has a proxy, and `NO_PROXY` keeps a laptop's exemptions with them), plus Windows's `SYSTEMROOT`, `USERPROFILE`, `APPDATA`, `LOCALAPPDATA`, `PATHEXT` and `COMSPEC`. There is no Windows release, but CI builds and tests on `windows-2022`, and these variables are absent on Linux and macOS. These are the variables the package's own `cc-data` calls need to find the researcher's stored credential (the keychain or the credentials file), to reach the server, and to run at all. Nothing else is inherited. + - **A short passthrough list a laptop needs.** It covers `HOME`, `PATH`, `USER`, `LOGNAME`, `LANG`, `LC_*`, `TZ`, `TMPDIR`, `DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR` and the proxy variables `HTTPS_PROXY`, `HTTP_PROXY` and `NO_PROXY` in either case (the runner sets the first two when it has a proxy, and `NO_PROXY` keeps a laptop's exemptions with them), plus Windows's `SYSTEMROOT`, `USERPROFILE`, `APPDATA`, `LOCALAPPDATA`, `PATHEXT` and `COMSPEC`. There is no Windows release, but CI builds and tests on `windows-2022`, and these variables are absent on Linux and macOS. These are the variables the package's own `cc-data` calls need to find the researcher's stored credential (the keychain or the credentials file), to reach the server, and to run at all. Nothing else is inherited. When the running binary is named `cc-data`, its folder goes first on the package's `PATH`, so the package's own calls reach the same cc-data rather than an older install. Under any other name (`go run`, a test binary), `run` warns which `cc-data` the package will reach. - **No storage variables.** `RD_BUCKET` and `RD_STORAGE_PREFIX` are never set. - **R12. Applicability before the entrypoint.** - **The question asked.** When the manifest declares any pattern, `package run` sends the patterns and the scope's assignment URLs to `POST /api/v1/packages/applies`. report-server derives the interactive URLs inside those assignments, so the URLs matched are the ones the runner matches. @@ -110,7 +110,7 @@ Researchers get four `cc-data package` commands (`init`, `run`, `build`, `publis - **R19. `build` refuses what `publish` would refuse,** by asking report-server (R6). It takes `--portal` (default: the configured portal), `--origin projects/` and `--official` so the check is made as the publish will be. A refused archive is not written. When validate answers `already_published`, `build` still writes the zip and warns " is already published; publish will refuse it until the version changes". When `publishing_unavailable` names a reason, `build` warns "this package cannot be published yet: " and still writes the zip. `run` ignores both, since running a package needs neither a new version nor a bucket. - **R20. Output.** - The default output path is `dir/.cc-data-build/-.zip`. It sits inside an excluded dot directory, which must be a real directory as R11's are, with the same `.gitignore` as R11's, so building from inside the package directory never feeds one build's zip into the next. A name or version holding a path separator is `INVALID_MANIFEST` rather than a path, since it is unchecked until validate answers, and for good on a server without the route. - - An explicit `--out` inside `dir` is refused unless R17 would exclude that path anyway. + - An explicit `--out` is refused when it is a directory (the package folder included), when it is inside `.cc-data-run`, which `package run` empties, or when it is inside `dir` where R17 would collect it. - stdout carries one JSON line: the path, `sha256:`, the size and the file count. ### `cc-data package publish` @@ -470,3 +470,19 @@ R17 states the rule. **Decision**: The default build zip is written to a temporary file and renamed into place, so a link already at that path is replaced rather than followed. The scope file must hold one JSON object and nothing after it. `summary.txt` and `counts.json` stay read under the display cap, since the runner reads them that way too, and the code now says so. +--- + +### The PR review's findings (Ethan McElroy, PR #18) +**Context**: Raised in Ethan McElroy's review of PR #18, which requested changes for one blocker. + +**Decision**: All were fixed, each with a test that fails when the fix is reverted. +- **Blocker, a private run folder.** `.cc-data-run` and everything `package run` creates in it are 0700, tightened if an earlier run left them wider, and `scope.json` and `.gitignore` are 0600, as every other student-data folder in cc-data is. +- **A relative data root** is made absolute before it reaches the package. +- **`--out`** refuses a directory, including the package folder, and any path inside `.cc-data-run`. +- **A 404 from validate or applies** still means the route is not deployed. REPORT-167's contract already says neither route ever answers 404, and `RouteMissing`'s comment now states that agreement. Matching the catch-all's body would not help, since a real `not_found` sends the same body. +- **A package folder that is itself a link** is resolved before it is walked. Links inside it are still refused. +- **The package's own `cc-data`** is the running binary when that is named `cc-data`; otherwise `run` warns which one the package will reach. +- **`init`** reports the real error when it cannot check the folder, and **`build`** renames its zip through `fsutil.RenameAtomic`, which retries on Windows. +- **Tests added** for the three package error codes, a package that fails or writes no `display.md`, the request count when publish's connection drops, an applies 503 through `run`, validate running before the run, `HOME`, and the run folder's modes. The leftover-process test now checks that the process is dead rather than racing a timed write. A new test keeps the learner query identical in the test, the guidance and the stub. The release workflow's tag classification is now run against sample tags, and it ignores build metadata after `+`. +- **Comments and help text** corrected where they overstated behavior. `Files.Bytes`, which nothing read, is removed, and the stub ignores only an "already exists" error from `dataset create`. + From 8020516418257468f7bf2e00f79cc3a66a785ee9 Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Fri, 9 Oct 2026 09:37:43 -0400 Subject: [PATCH 15/17] test: build the expected cc-data folder with filepath [REPORT-146] TestPackageRunReachesThisCCData compared filepath.Dir's answer with a literal "/build", which Windows spells "\build". The expected folder is now joined with the platform's separator. --- cmd/package_test.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/cmd/package_test.go b/cmd/package_test.go index 2f432fe..1805a1e 100644 --- a/cmd/package_test.go +++ b/cmd/package_test.go @@ -669,10 +669,11 @@ func TestPackageRunErrorsAreTheRunnersCodesAtExitOne(t *testing.T) { func TestPackageRunReachesThisCCData(t *testing.T) { none := func(string) (string, error) { return "", exec.ErrNotFound } other := func(string) (string, error) { return "/opt/homebrew/bin/cc-data", nil } - if dir, warning := packageCCData(filepath.Join("/build", "cc-data"), other); dir != "/build" || warning != "" { + build := filepath.Join(string(filepath.Separator), "build") + if dir, warning := packageCCData(filepath.Join(build, "cc-data"), other); dir != build || warning != "" { t.Errorf("a binary named cc-data: %q, %q", dir, warning) } - if dir, warning := packageCCData(filepath.Join("/build", "cc-data.exe"), other); dir != "/build" || warning != "" { + if dir, warning := packageCCData(filepath.Join(build, "cc-data.exe"), other); dir != build || warning != "" { t.Errorf("cc-data.exe: %q, %q", dir, warning) } if dir, warning := packageCCData("/tmp/go-build/exe/main", other); dir != "" || !strings.Contains(warning, "/opt/homebrew/bin/cc-data") { From 7ffaeb9c85facc7108d4fa35770a094469ed96cc Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Fri, 9 Oct 2026 09:51:49 -0400 Subject: [PATCH 16/17] fix: refuse --out inside .cc-data-run in any letter case [REPORT-146] On a case-insensitive file system, the default on macOS and Windows, --out .CC-DATA-RUN/x.zip lands inside .cc-data-run, where the next package run deletes it, but passed the case-sensitive comparison. The first path component is now compared without case on every platform, and the --out test covers the upper-case spelling. --- cmd/package_test.go | 2 +- internal/packages/build.go | 3 ++- specs/REPORT-146-package-commands-and-dataset-views.md | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/cmd/package_test.go b/cmd/package_test.go index 1805a1e..7726bd0 100644 --- a/cmd/package_test.go +++ b/cmd/package_test.go @@ -321,7 +321,7 @@ func TestPackageBuildRefusesAnOutputItWouldCollect(t *testing.T) { if cliErr := asCLIError(t, err); cliErr.ExitCode != output.ExitUsage { t.Errorf("err = %+v", cliErr) } - for _, out := range []string{dir, filepath.Join(dir, packages.RunDirName, "pkg", "p.zip")} { + for _, out := range []string{dir, filepath.Join(dir, packages.RunDirName, "pkg", "p.zip"), filepath.Join(dir, strings.ToUpper(packages.RunDirName), "p.zip")} { err := (packageBuildFlags{out: out}).run(context.Background(), api.New(srv.URL, "tok"), dir) if cliErr := asCLIError(t, err); cliErr.ExitCode != output.ExitUsage { t.Errorf("--out %s: err = %+v, want a usage error", out, cliErr) diff --git a/internal/packages/build.go b/internal/packages/build.go index 03be3cc..db2805c 100644 --- a/internal/packages/build.go +++ b/internal/packages/build.go @@ -31,7 +31,8 @@ func CheckBuildOutput(dir, out string) error { if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { return nil } - if strings.SplitN(filepath.ToSlash(rel), "/", 2)[0] == RunDirName { + // Compared without case, since macOS and Windows file systems ignore it by default. + if strings.EqualFold(strings.SplitN(filepath.ToSlash(rel), "/", 2)[0], RunDirName) { return fmt.Errorf("--out %s is inside %s, which package run empties", out, RunDirName) } if excluded(filepath.ToSlash(rel), false) || excludedAncestor(filepath.ToSlash(rel)) { diff --git a/specs/REPORT-146-package-commands-and-dataset-views.md b/specs/REPORT-146-package-commands-and-dataset-views.md index 257c952..73a90c3 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views.md +++ b/specs/REPORT-146-package-commands-and-dataset-views.md @@ -110,7 +110,7 @@ Researchers get four `cc-data package` commands (`init`, `run`, `build`, `publis - **R19. `build` refuses what `publish` would refuse,** by asking report-server (R6). It takes `--portal` (default: the configured portal), `--origin projects/` and `--official` so the check is made as the publish will be. A refused archive is not written. When validate answers `already_published`, `build` still writes the zip and warns " is already published; publish will refuse it until the version changes". When `publishing_unavailable` names a reason, `build` warns "this package cannot be published yet: " and still writes the zip. `run` ignores both, since running a package needs neither a new version nor a bucket. - **R20. Output.** - The default output path is `dir/.cc-data-build/-.zip`. It sits inside an excluded dot directory, which must be a real directory as R11's are, with the same `.gitignore` as R11's, so building from inside the package directory never feeds one build's zip into the next. A name or version holding a path separator is `INVALID_MANIFEST` rather than a path, since it is unchecked until validate answers, and for good on a server without the route. - - An explicit `--out` is refused when it is a directory (the package folder included), when it is inside `.cc-data-run`, which `package run` empties, or when it is inside `dir` where R17 would collect it. + - An explicit `--out` is refused when it is a directory (the package folder included), when it is inside `.cc-data-run` (in any letter case, since macOS and Windows file systems ignore case by default), which `package run` empties, or when it is inside `dir` where R17 would collect it. - stdout carries one JSON line: the path, `sha256:`, the size and the file count. ### `cc-data package publish` From e8cb060c1a9593e4f5dc9fb0af4a907ac1a56098 Mon Sep 17 00:00:00 2001 From: Doug Martin Date: Fri, 9 Oct 2026 10:58:48 -0400 Subject: [PATCH 17/17] fix: take Ethan McElroy's approving review notes [REPORT-146] - package run no longer puts the running binary's own folder first on the package's PATH, which moved every other tool in a shared bin ahead of the user's PATH too. It links .cc-data-run/bin/cc-data to the running binary, whatever its name, and puts only that folder first. go run therefore needs no warning; Windows, which needs a privilege to create links, keeps the binary's folder when it is named cc-data and warns otherwise. - build's --out check resolves links in the package folder and in --out before comparing them, so the same place reached two ways compares equal. - A blank line left in a release_test.go list is removed, and the flag the run test shares with its fake server's handler is an atomic.Bool. --- cmd/package.go | 20 +-------------- cmd/package_test.go | 25 +++---------------- docs/researcher-guide.md | 7 +++--- internal/packages/build.go | 25 +++++++++++++++++-- internal/packages/files_test.go | 21 ++++++++++++++++ internal/packages/proc_unix.go | 15 +++++++++++ internal/packages/proc_windows.go | 21 +++++++++++++++- internal/packages/proc_windows_test.go | 25 +++++++++++++++++++ internal/packages/run.go | 18 +++++++++---- internal/packages/run_test.go | 12 ++++++--- release_test.go | 1 - ...-146-package-commands-and-dataset-views.md | 14 +++++++++-- 12 files changed, 146 insertions(+), 58 deletions(-) create mode 100644 internal/packages/proc_windows_test.go diff --git a/cmd/package.go b/cmd/package.go index 73c5c07..55a1e05 100644 --- a/cmd/package.go +++ b/cmd/package.go @@ -137,15 +137,11 @@ func (f packageRunFlags) run(ctx context.Context, client *api.Client, pkg loaded return err } exe, _ := os.Executable() - binDir, warning := packageCCData(exe, exec.LookPath) - if warning != "" { - output.Warnf("%s", warning) - } start := time.Now() res, err := packages.Run(ctx, packages.RunOptions{ Dir: pkg.dir, Manifest: pkg.manifest, Files: pkg.files, Scope: pkg.scope, Applies: appliesFor(client, pkg.scope), Dataset: ref.String(), Name: ref.Name, Portal: ref.Portal.Host(), DataRoot: dataRoot, - ReportSrv: client.BaseURL, BinDir: binDir, Stderr: output.Stderr(), Environ: f.environ, LookPath: exec.LookPath, + ReportSrv: client.BaseURL, CCData: exe, Stderr: output.Stderr(), Environ: f.environ, LookPath: exec.LookPath, }) if err != nil { return packageRunError(err) @@ -156,20 +152,6 @@ func (f packageRunFlags) run(ctx context.Context, client *api.Client, pkg loaded }) } -// packageCCData picks the cc-data a package's own calls reach: this binary, by putting its -// folder first on the package's PATH, when it is named cc-data. A binary with another name (go -// run, a test) cannot be reached by name, so the warning names the cc-data the package will call. -func packageCCData(exe string, lookPath func(string) (string, error)) (binDir, warning string) { - if exe != "" && strings.TrimSuffix(filepath.Base(exe), ".exe") == "cc-data" { - return filepath.Dir(exe), "" - } - found, err := lookPath("cc-data") - if err != nil { - return "", "this binary is not named cc-data and no cc-data is on PATH, so the package's own cc-data calls will fail" - } - return "", fmt.Sprintf("this binary is not named cc-data, so the package's own cc-data calls reach %s", found) -} - func newPackageRunCmd() *cobra.Command { f := packageRunFlags{environ: os.Environ} cmd := &cobra.Command{ diff --git a/cmd/package_test.go b/cmd/package_test.go index 7726bd0..e07f1d8 100644 --- a/cmd/package_test.go +++ b/cmd/package_test.go @@ -10,7 +10,6 @@ import ( "net/http" "net/http/httptest" "os" - "os/exec" "path/filepath" "runtime" "strings" @@ -581,10 +580,10 @@ func TestPackageRunValidatesThenRunsWithTheRunnersNames(t *testing.T) { script := "#!/bin/sh\nenv > \"$RD_OUTPUT_DIR/../env.txt\"\ncp \"$RD_SCOPE_FILE\" \"$RD_OUTPUT_DIR/../scope.json\"\necho ok > \"$RD_OUTPUT_DIR/display.md\"\n" f, pkg := loadRunPackage(t, manifest, map[string]string{"run.sh": script}, func() []string { return []string{"PATH=" + os.Getenv("PATH")} }) dir := pkg.dir - ranBeforeValidate := false + var ranBeforeValidate atomic.Bool srv := newPackageServer(t, map[string]func(http.ResponseWriter, []byte){"/api/v1/packages/validate": func(w http.ResponseWriter, body []byte) { if _, err := os.Stat(filepath.Join(dir, packages.RunDirName)); err == nil { - ranBeforeValidate = true + ranBeforeValidate.Store(true) } answer(http.StatusOK, validated)(w, body) }}) @@ -598,7 +597,7 @@ func TestPackageRunValidatesThenRunsWithTheRunnersNames(t *testing.T) { if err := f.run(context.Background(), api.New(srv.URL, "tok"), pkg, ref, "data-root"); err != nil { t.Fatal(err) } - if ranBeforeValidate { + if ranBeforeValidate.Load() { t.Error("the run tree existed before validate answered") } @@ -665,21 +664,3 @@ func TestPackageRunErrorsAreTheRunnersCodesAtExitOne(t *testing.T) { } } } - -func TestPackageRunReachesThisCCData(t *testing.T) { - none := func(string) (string, error) { return "", exec.ErrNotFound } - other := func(string) (string, error) { return "/opt/homebrew/bin/cc-data", nil } - build := filepath.Join(string(filepath.Separator), "build") - if dir, warning := packageCCData(filepath.Join(build, "cc-data"), other); dir != build || warning != "" { - t.Errorf("a binary named cc-data: %q, %q", dir, warning) - } - if dir, warning := packageCCData(filepath.Join(build, "cc-data.exe"), other); dir != build || warning != "" { - t.Errorf("cc-data.exe: %q, %q", dir, warning) - } - if dir, warning := packageCCData("/tmp/go-build/exe/main", other); dir != "" || !strings.Contains(warning, "/opt/homebrew/bin/cc-data") { - t.Errorf("go run: %q, %q", dir, warning) - } - if _, warning := packageCCData("/tmp/go-build/exe/main", none); !strings.Contains(warning, "will fail") { - t.Errorf("no cc-data on PATH: %q", warning) - } -} diff --git a/docs/researcher-guide.md b/docs/researcher-guide.md index 1156d1f..f4deef8 100644 --- a/docs/researcher-guide.md +++ b/docs/researcher-guide.md @@ -641,9 +641,10 @@ What it cannot reproduce: its own user behind an egress proxy. - **A few of your variables pass through.** `HOME`, `PATH`, your locale, the proxy settings and a handful of others reach the package, so its own `cc-data` - calls find your login. Nothing else from your environment does. The folder of - the `cc-data` running `package run` goes first on the package's `PATH`, so the - package calls that same version rather than an older one installed elsewhere. + calls find your login. Nothing else from your environment does. A private folder + holding only a link to the `cc-data` running `package run` goes first on the + package's `PATH`, so the package calls that same version rather than an older + one installed elsewhere, and nothing else moves ahead of your `PATH`. - **Python.** A `.py` entrypoint runs with `python3.11` when it is on your `PATH`, as on the VM, and otherwise with `python3` after a note saying so. - **CLUE.** A package with `clue_prepull: true` is refused locally, since diff --git a/internal/packages/build.go b/internal/packages/build.go index db2805c..20d2121 100644 --- a/internal/packages/build.go +++ b/internal/packages/build.go @@ -19,11 +19,11 @@ func CheckBuildOutput(dir, out string) error { if info, err := os.Stat(out); err == nil && info.IsDir() { return fmt.Errorf("--out %s is a directory; name the zip file to write", out) } - absDir, err := filepath.Abs(dir) + absDir, err := resolvedPath(dir) if err != nil { return err } - absOut, err := filepath.Abs(out) + absOut, err := resolvedPath(out) if err != nil { return err } @@ -41,6 +41,27 @@ func CheckBuildOutput(dir, out string) error { return fmt.Errorf("--out %s is inside the package, where the next build would include it", out) } +// resolvedPath is p made absolute with every link in its existing part resolved, so the same +// place reached through a link and by its real path compares equal. +func resolvedPath(p string) (string, error) { + abs, err := filepath.Abs(p) + if err != nil { + return "", err + } + rest := "" + for cur := abs; ; { + if real, err := filepath.EvalSymlinks(cur); err == nil { + return filepath.Join(real, rest), nil + } + parent := filepath.Dir(cur) + if parent == cur { + return abs, nil + } + rest = filepath.Join(filepath.Base(cur), rest) + cur = parent + } +} + func excludedAncestor(rel string) bool { parts := strings.Split(rel, "/") for i := 1; i < len(parts); i++ { diff --git a/internal/packages/files_test.go b/internal/packages/files_test.go index ec513d3..0a58177 100644 --- a/internal/packages/files_test.go +++ b/internal/packages/files_test.go @@ -241,3 +241,24 @@ func TestCollectFollowsALinkedPackageDirectory(t *testing.T) { t.Errorf("collected %q through a linked package directory", got) } } + +func TestCheckBuildOutputSeesThroughLinks(t *testing.T) { + skipOnWindows(t, "links need privileges on Windows") + real := t.TempDir() + link := filepath.Join(t.TempDir(), "pkg") + if err := os.Symlink(real, link); err != nil { + t.Fatal(err) + } + for _, tc := range []struct{ dir, out string }{ + {link, filepath.Join(real, RunDirName, "x.zip")}, + {real, filepath.Join(link, RunDirName, "x.zip")}, + {link, filepath.Join(real, "p.zip")}, + } { + if err := CheckBuildOutput(tc.dir, tc.out); err == nil { + t.Errorf("--out %s for package %s was accepted", tc.out, tc.dir) + } + } + if err := CheckBuildOutput(link, filepath.Join(t.TempDir(), "p.zip")); err != nil { + t.Errorf("an --out outside the package was refused: %v", err) + } +} diff --git a/internal/packages/proc_unix.go b/internal/packages/proc_unix.go index d425bf2..0375c19 100644 --- a/internal/packages/proc_unix.go +++ b/internal/packages/proc_unix.go @@ -3,7 +3,9 @@ package packages import ( + "os" "os/exec" + "path/filepath" "syscall" ) @@ -21,3 +23,16 @@ func reapGroup(cmd *exec.Cmd) { syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) } } + +// exposeCCData links bin/cc-data to the running binary, whatever its name, and returns bin to put +// first on the package's PATH. Unlike the binary's own folder, which may be a shared bin, it puts +// nothing else ahead of the user's PATH. +func exposeCCData(bin, exe string, _ func(string) (string, error)) (string, string, error) { + if exe == "" { + return "", "", nil + } + if err := os.Symlink(exe, filepath.Join(bin, "cc-data")); err != nil { + return "", "", err + } + return bin, "", nil +} diff --git a/internal/packages/proc_windows.go b/internal/packages/proc_windows.go index 0dead87..a6ff887 100644 --- a/internal/packages/proc_windows.go +++ b/internal/packages/proc_windows.go @@ -2,10 +2,29 @@ package packages -import "os/exec" +import ( + "fmt" + "os/exec" + "path/filepath" + "strings" +) // On Windows, which has no release, exec.CommandContext kills only the entrypoint, and nothing // is killed after it exits. func killGroup(cmd *exec.Cmd) {} func reapGroup(cmd *exec.Cmd) {} + +// exposeCCData puts the running binary's own folder first on the package's PATH when it is named +// cc-data, since creating a link needs a privilege on Windows. Under any other name the package's +// cc-data calls reach whatever is on PATH, and the warning names it. +func exposeCCData(_, exe string, lookPath func(string) (string, error)) (string, string, error) { + if exe != "" && strings.EqualFold(strings.TrimSuffix(filepath.Base(exe), ".exe"), "cc-data") { + return filepath.Dir(exe), "", nil + } + found, err := lookPath("cc-data") + if err != nil { + return "", "this binary is not named cc-data and no cc-data is on PATH, so the package's own cc-data calls will fail", nil + } + return "", fmt.Sprintf("this binary is not named cc-data, so the package's own cc-data calls reach %s", found), nil +} diff --git a/internal/packages/proc_windows_test.go b/internal/packages/proc_windows_test.go new file mode 100644 index 0000000..2172234 --- /dev/null +++ b/internal/packages/proc_windows_test.go @@ -0,0 +1,25 @@ +//go:build windows + +package packages + +import ( + "os/exec" + "path/filepath" + "strings" + "testing" +) + +func TestExposeCCDataOnWindowsUsesTheBinarysFolderByName(t *testing.T) { + other := func(string) (string, error) { return `C:\tools\cc-data.exe`, nil } + none := func(string) (string, error) { return "", exec.ErrNotFound } + build := filepath.Join(`C:\`, "build") + if dir, warning, err := exposeCCData("", filepath.Join(build, "cc-data.exe"), other); err != nil || dir != build || warning != "" { + t.Errorf("cc-data.exe: %q, %q, %v", dir, warning, err) + } + if dir, warning, _ := exposeCCData("", filepath.Join(build, "main.exe"), other); dir != "" || !strings.Contains(warning, `C:\tools\cc-data.exe`) { + t.Errorf("go run: %q, %q", dir, warning) + } + if _, warning, _ := exposeCCData("", filepath.Join(build, "main.exe"), none); !strings.Contains(warning, "will fail") { + t.Errorf("no cc-data on PATH: %q", warning) + } +} diff --git a/internal/packages/run.go b/internal/packages/run.go index 58b37d6..9d04a20 100644 --- a/internal/packages/run.go +++ b/internal/packages/run.go @@ -66,7 +66,7 @@ type RunOptions struct { Portal string // the ref's portal host DataRoot string // CC_DATA_ROOT, absolute ReportSrv string // the stored credential's server, or "" - BinDir string // put first on PATH, so the package calls this cc-data + CCData string // the running cc-data, which the package's own cc-data calls reach Stderr io.Writer // where the package's own output goes Environ func() []string LookPath func(string) (string, error) @@ -94,6 +94,13 @@ func Run(ctx context.Context, o RunOptions) (Result, error) { if err != nil { return Result{}, err } + binDir, warning, err := exposeCCData(paths.bin, o.CCData, o.LookPath) + if err != nil { + return Result{}, err + } + if warning != "" { + fmt.Fprintln(o.Stderr, "warning: "+warning) + } scope := ScopeFile{LocalScope: o.Scope, ClueSource: "firebase", Dataset: o.Name, OutputDir: paths.out} b, err := json.MarshalIndent(scope, "", " ") if err != nil { @@ -132,7 +139,7 @@ func Run(ctx context.Context, o RunOptions) (Result, error) { defer cancel() cmd := exec.CommandContext(runCtx, command, args...) cmd.Dir = paths.pkg - cmd.Env = environment(o.Environ(), env, o.BinDir) + cmd.Env = environment(o.Environ(), env, binDir) cmd.Stdout = o.Stderr cmd.Stderr = o.Stderr killGroup(cmd) @@ -151,7 +158,7 @@ func Run(ctx context.Context, o RunOptions) (Result, error) { return ReadResult(paths.out) } -type runPaths struct{ pkg, in, out, data string } +type runPaths struct{ pkg, in, out, bin, data string } // prepare empties and rebuilds the run tree, keeping only data/ from one run to the next. Every // directory it removes or creates under the package must be a real directory, so a link left @@ -161,6 +168,7 @@ func prepare(run string, files Files) (runPaths, error) { pkg: filepath.Join(run, "pkg"), in: filepath.Join(run, "in"), out: filepath.Join(run, "out"), + bin: filepath.Join(run, "bin"), data: filepath.Join(run, "data"), } if err := ensureRealDir(run); err != nil { @@ -169,7 +177,7 @@ func prepare(run string, files Files) (runPaths, error) { if err := ignoreAll(run); err != nil { return p, err } - for _, d := range []string{p.pkg, p.in, p.out} { + for _, d := range []string{p.pkg, p.in, p.out, p.bin} { if _, err := os.Lstat(d); err == nil { if err := RealDir(d); err != nil { return p, fmt.Errorf("refusing to empty %s: it is a link or not a directory", d) @@ -189,7 +197,7 @@ func prepare(run string, files Files) (runPaths, error) { if err != nil { return p, err } - p = runPaths{filepath.Join(abs, "pkg"), filepath.Join(abs, "in"), filepath.Join(abs, "out"), filepath.Join(abs, "data")} + p = runPaths{filepath.Join(abs, "pkg"), filepath.Join(abs, "in"), filepath.Join(abs, "out"), filepath.Join(abs, "bin"), filepath.Join(abs, "data")} for _, rel := range files.Paths { if err := copyFile(filepath.Join(files.Dir, filepath.FromSlash(rel)), filepath.Join(p.pkg, filepath.FromSlash(rel)), files.Modes[rel]); err != nil { return p, err diff --git a/internal/packages/run_test.go b/internal/packages/run_test.go index 5042cf7..5da878f 100644 --- a/internal/packages/run_test.go +++ b/internal/packages/run_test.go @@ -106,7 +106,7 @@ echo ok > "$RD_OUTPUT_DIR/display.md" var stderr bytes.Buffer environ := []string{"PATH=" + os.Getenv("PATH"), "HOME=/home/r", "LC_ALL=C", "https_proxy=http://proxy:3128", "AWS_SECRET_ACCESS_KEY=leak"} o := runOpts(t, dir, files, environ, &stderr) - o.BinDir = "/opt/cc-data/bin" + o.CCData = "/opt/shared/bin/cc-data" res, err := Run(context.Background(), o) if err != nil { t.Fatalf("%v\n%s", err, stderr.String()) @@ -128,13 +128,19 @@ echo ok > "$RD_OUTPUT_DIR/display.md" "RD_REPORT_SERVER_URL": "https://report.example", "LC_ALL": "C", "HOME": "/home/r", - "PATH": "/opt/cc-data/bin" + string(os.PathListSeparator) + os.Getenv("PATH"), + "PATH": filepath.Join(run, "bin") + string(os.PathListSeparator) + os.Getenv("PATH"), "https_proxy": "http://proxy:3128", } { if env[k] != want { t.Errorf("%s = %q, want %q", k, env[k], want) } } + if target, err := os.Readlink(filepath.Join(run, "bin", "cc-data")); err != nil || target != "/opt/shared/bin/cc-data" { + t.Errorf("bin/cc-data links to %q (%v), want the running binary", target, err) + } + if entries, _ := os.ReadDir(filepath.Join(run, "bin")); len(entries) != 1 { + t.Errorf("bin holds %v, want only the cc-data link", entries) + } if _, ok := env["AWS_SECRET_ACCESS_KEY"]; ok { t.Error("a variable outside the passthrough reached the package") } @@ -352,7 +358,7 @@ func TestRunKeepsItsTreePrivate(t *testing.T) { t.Fatal(err) } for rel, want := range map[string]os.FileMode{ - ".": 0o700, "pkg": 0o700, "in": 0o700, "out": 0o700, "data": 0o700, + ".": 0o700, "pkg": 0o700, "in": 0o700, "out": 0o700, "bin": 0o700, "data": 0o700, "in/scope.json": 0o600, ".gitignore": 0o600, } { info, err := os.Stat(filepath.Join(run, rel)) diff --git a/release_test.go b/release_test.go index 9066749..a951881 100644 --- a/release_test.go +++ b/release_test.go @@ -130,7 +130,6 @@ func readFile(t *testing.T, path string) string { func TestReleaseWorkflowKeepsPreReleasesBack(t *testing.T) { rel := readFile(t, ".github/workflows/release.yml") for _, want := range []string{ - "--prerelease=${{ steps.tag.outputs.prerelease }}", "if: steps.tag.outputs.prerelease == 'false'", } { diff --git a/specs/REPORT-146-package-commands-and-dataset-views.md b/specs/REPORT-146-package-commands-and-dataset-views.md index 73a90c3..70665bc 100644 --- a/specs/REPORT-146-package-commands-and-dataset-views.md +++ b/specs/REPORT-146-package-commands-and-dataset-views.md @@ -66,7 +66,7 @@ Researchers get four `cc-data package` commands (`init`, `run`, `build`, `publis - `CC_DATA_ROOT`: the data root `package run` itself resolved, made absolute, since the package runs from its staged copy, where a relative root would name another folder. - `CC_DATA_PORTAL`: the ref's portal. - `RD_REPORT_SERVER_URL`: the portal's stored credential's server, when one is stored. - - **A short passthrough list a laptop needs.** It covers `HOME`, `PATH`, `USER`, `LOGNAME`, `LANG`, `LC_*`, `TZ`, `TMPDIR`, `DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR` and the proxy variables `HTTPS_PROXY`, `HTTP_PROXY` and `NO_PROXY` in either case (the runner sets the first two when it has a proxy, and `NO_PROXY` keeps a laptop's exemptions with them), plus Windows's `SYSTEMROOT`, `USERPROFILE`, `APPDATA`, `LOCALAPPDATA`, `PATHEXT` and `COMSPEC`. There is no Windows release, but CI builds and tests on `windows-2022`, and these variables are absent on Linux and macOS. These are the variables the package's own `cc-data` calls need to find the researcher's stored credential (the keychain or the credentials file), to reach the server, and to run at all. Nothing else is inherited. When the running binary is named `cc-data`, its folder goes first on the package's `PATH`, so the package's own calls reach the same cc-data rather than an older install. Under any other name (`go run`, a test binary), `run` warns which `cc-data` the package will reach. + - **A short passthrough list a laptop needs.** It covers `HOME`, `PATH`, `USER`, `LOGNAME`, `LANG`, `LC_*`, `TZ`, `TMPDIR`, `DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR` and the proxy variables `HTTPS_PROXY`, `HTTP_PROXY` and `NO_PROXY` in either case (the runner sets the first two when it has a proxy, and `NO_PROXY` keeps a laptop's exemptions with them), plus Windows's `SYSTEMROOT`, `USERPROFILE`, `APPDATA`, `LOCALAPPDATA`, `PATHEXT` and `COMSPEC`. There is no Windows release, but CI builds and tests on `windows-2022`, and these variables are absent on Linux and macOS. These are the variables the package's own `cc-data` calls need to find the researcher's stored credential (the keychain or the credentials file), to reach the server, and to run at all. Nothing else is inherited. `.cc-data-run/bin/` holds only a link named `cc-data` to the running binary, whatever its name (`go run` included), and goes first on the package's `PATH`. The package's own calls therefore reach the same cc-data rather than an older install, and nothing else in the binary's folder moves ahead of the user's `PATH`. On Windows, which has no release and needs a privilege to create links, the binary's own folder goes first when it is named `cc-data`, and otherwise `run` warns which `cc-data` the package will reach. - **No storage variables.** `RD_BUCKET` and `RD_STORAGE_PREFIX` are never set. - **R12. Applicability before the entrypoint.** - **The question asked.** When the manifest declares any pattern, `package run` sends the patterns and the scope's assignment URLs to `POST /api/v1/packages/applies`. report-server derives the interactive URLs inside those assignments, so the URLs matched are the ones the runner matches. @@ -481,8 +481,18 @@ R17 states the rule. - **`--out`** refuses a directory, including the package folder, and any path inside `.cc-data-run`. - **A 404 from validate or applies** still means the route is not deployed. REPORT-167's contract already says neither route ever answers 404, and `RouteMissing`'s comment now states that agreement. Matching the catch-all's body would not help, since a real `not_found` sends the same body. - **A package folder that is itself a link** is resolved before it is walked. Links inside it are still refused. -- **The package's own `cc-data`** is the running binary when that is named `cc-data`; otherwise `run` warns which one the package will reach. +- **The package's own `cc-data`** is the running binary, reached through a link in `.cc-data-run/bin/`, the only folder put ahead of the user's `PATH`. On Windows the binary's own folder is used when it is named `cc-data`. - **`init`** reports the real error when it cannot check the folder, and **`build`** renames its zip through `fsutil.RenameAtomic`, which retries on Windows. - **Tests added** for the three package error codes, a package that fails or writes no `display.md`, the request count when publish's connection drops, an applies 503 through `run`, validate running before the run, `HOME`, and the run folder's modes. The leftover-process test now checks that the process is dead rather than racing a timed write. A new test keeps the learner query identical in the test, the guidance and the stub. The release workflow's tag classification is now run against sample tags, and it ignores build metadata after `+`. - **Comments and help text** corrected where they overstated behavior. `Files.Bytes`, which nothing read, is removed, and the stub ignores only an "already exists" error from `dataset create`. +--- + +### The approving review's notes (Ethan McElroy, PR #18) +**Context**: Ethan McElroy approved PR #18 with four minor notes left to the author's discretion. + +**Decision**: All four were taken. +- **The `PATH` side effect.** Putting the binary's own folder first also put every other tool in a shared `bin` ahead of the user's `PATH`. The package now gets `.cc-data-run/bin/`, holding only a link to the running binary, so nothing else moves. This also drops the "not named cc-data" warning, since a link can carry the name, except on Windows. +- **`--out` through links.** The package folder and `--out` are both resolved through links before they are compared, so the same place reached two ways compares equal. +- **A blank line** in `release_test.go` and **a flag shared with a test server's handler** without synchronization, now an `atomic.Bool`, are fixed. +