refactor foundry local packaging pipeline

.pipelines/v2/sdk_v2-js-pipeline-plan.md → .pipelines/docs/sdk_v2-js-pipeline-plan.md

@@ -99,13 +99,13 @@ Build and test on the `onnxruntime-linux-ARM64-CPU-2019` pool
 
 - `NodeTool@0` for Node 20.
 - Drop `js-prebuild-<rid>` contents into `sdk_v2/js/prebuilds/`.
-- Check out `test-data-shared` via `checkout-steps.yml`.
-- On macOS, `brew install git-lfs` before checkout (same as Python).
+- Fetch the model cache via `fetch-models-from-artifacts-feed.yml`.
+- On macOS, ensure the test data directory is present and writable before the run.
 - `npm ci --no-audit --no-fund` (no `--ignore-scripts` — install-native
   is allowed to run on the test side; it's a no-op when the prebuild is
   already in place).
 - `npm run build:ts`, then `npx vitest run --reporter=verbose` with
-  `FOUNDRY_TEST_DATA_DIR=$(testDataSharedDir)`.
+  `FOUNDRY_TEST_DATA_DIR=$(Build.SourcesDirectory)/test-data-shared`.
 
 ## Pack stage
 
@@ -140,21 +140,21 @@ signing.
 | `js_pack`                 | none                                                                          | n/a           | `.tgz` not signed by npm convention |
 
 Windows signing block is a near-copy of the SDK DLL signing step in
-[`steps-build-cs.yml`](templates/steps-build-cs.yml) — same
+[`steps-build-cs.yml`](../templates/steps-build-cs.yml) — same
 `ConnectedServiceName`, ESRP variable group
 (`FoundryLocal-ESRP-Signing`), and `signConfigType: inlineSignParams`
 shape.
 
 ## Files added / modified
 
 **New:**
-- [`.pipelines/v2/templates/stages-js.yml`](templates/stages-js.yml)
-- [`.pipelines/v2/templates/steps-build-js.yml`](templates/steps-build-js.yml)
-- [`.pipelines/v2/templates/steps-test-js.yml`](templates/steps-test-js.yml)
-- [`.pipelines/v2/templates/steps-pack-js.yml`](templates/steps-pack-js.yml)
+- [`.pipelines/templates/stages-js.yml`](../templates/stages-js.yml)
+- [`.pipelines/templates/steps-build-js.yml`](../templates/steps-build-js.yml)
+- [`.pipelines/templates/steps-test-js.yml`](../templates/steps-test-js.yml)
+- [`.pipelines/templates/steps-pack-js.yml`](../templates/steps-pack-js.yml)
 
 **Modified:**
-- [`.pipelines/v2/templates/stages-sdk-v2.yml`](templates/stages-sdk-v2.yml)
+- [`.pipelines/templates/stages-sdk-v2.yml`](../templates/stages-sdk-v2.yml)
   — appends `stages-js.yml`.
 - [`sdk_v2/js/script/gyp/print-import-lib-dir.mjs`](../../sdk_v2/js/script/gyp/print-import-lib-dir.mjs)
   — honors `FOUNDRY_LOCAL_LIB_DIR`.

.pipelines/v2/sdk_v2-pipeline-plan.md → .pipelines/docs/sdk_v2-pipeline-plan.md

@@ -21,10 +21,9 @@ JS is documented separately in `sdk_v2-js-pipeline-plan.md`. A standalone
 
 There is **no dedicated sdk_v2 top-level pipeline**. The sdk_v2 stage graph
 is emitted by a single coordinator template,
-`.pipelines/v2/templates/stages-sdk-v2.yml`, which is included by the
+`.pipelines/templates/stages-sdk-v2.yml`, which is included by the
 existing `.pipelines/foundry-local-packaging.yml`. sdk_v2 is built
-unconditionally on every run (there is no v2 gate); the v1 legacy stages
-are gated separately via `.pipelines/v1/templates/stages-sdk-v1.yml`.
+unconditionally on every run.
 
 ## Supported platforms
 
@@ -54,10 +53,10 @@ are gated separately via `.pipelines/v1/templates/stages-sdk-v1.yml`.
    ORT and GenAI native libraries are **not** bundled in the wheel; they are
    resolved at install time from public PyPI (see decision 9). The wheel
    only contains `foundry_local` plus its non-ORT runtime closure.
-4. **Test stages set `FOUNDRY_TEST_DATA_DIR`** to the checked-out
-   `test-data-shared` path for all three languages. C++ stages set it
-   directly; Python's `conftest.py` reads it; C# `Utils.cs` prefers it
-   over `appsettings.Test.json`.
+4. **Test stages set `FOUNDRY_TEST_DATA_DIR`** to the local model-cache path
+   for all three languages. C++ stages set it directly; Python's
+   `conftest.py` reads it; C# `Utils.cs` prefers it over
+   `appsettings.Test.json`.
 5. **Parameter naming.** sdk_v2 templates use `flNugetDir` (not `flcNugetDir`).
    `flc` was Foundry Local Core, which we are replacing.
 6. **Shared `compute_version` stage.** Defined inline in
@@ -129,7 +128,9 @@ are gated separately via `.pipelines/v1/templates/stages-sdk-v1.yml`.
 ## Template layout
 
 ```
-.pipelines/v2/
+.pipelines/
+├── docker/
+├── docs/
 └── templates/
     ├── stages-sdk-v2.yml             # Coordinator: native + C# + Python + JS + Rust
     ├── stages-build-native.yml       # 5 native build stages + C++/NuGet pack stages
@@ -149,9 +150,10 @@ are gated separately via `.pipelines/v1/templates/stages-sdk-v1.yml`.
     └── steps-pack-nuget.yml          # Runs sdk_v2/cpp/nuget/pack.py
 ```
 
-The repo-shared `.pipelines/templates/checkout-steps.yml` is reused for
-`test-data-shared` (LFS) checkout via the `FoundryLocalCore-SP` Azure CLI
-service connection.
+The repo-shared `.pipelines/templates/fetch-models-from-artifacts-feed.yml`
+pre-populates the local model cache from the AIFoundryLocal public artifacts
+feed, so every test stage can point `FOUNDRY_TEST_DATA_DIR` at the same
+staged cache without a Git checkout dependency.
 
 ## Stage dependency graph
 
@@ -231,8 +233,8 @@ hand off, so:
   matrices.
 * Integration tests run with `--test-threads=1`: the native core enforces a
   single `FoundryLocalManager` per process, so tests that each create one must
-  not run concurrently. They consume the same `test-data-shared` models as the
-  C# / Python suites (`FOUNDRY_TEST_DATA_DIR`).
+  not run concurrently. They consume the same model cache as the C# / Python
+  suites (`FOUNDRY_TEST_DATA_DIR`).
 
 ## Versioning
 
@@ -268,11 +270,9 @@ Pipeline parameters allow override:
 ## Dependency versions
 
 The pipeline pins ORT / GenAI / WinML versions and downloads the NuGet
-packages from **public nuget.org** (`https://www.nuget.org/api/v2/package`)
-before invoking CMake. This matches Foundry Local Core's own
-[nuget.config](../templates/build-core-steps.yml), which maps everything
-except `Microsoft.Telemetry*` to nuget.org. Pre-fetching serves two
-purposes:
+packages through `AIFoundryLocal_PublicPackages` before invoking CMake.
+The feed provides the approved upstreams while preventing pipeline jobs from
+accessing public package feeds directly. Pre-fetching serves two purposes:
 
 1. **Version pinning** — the `KEY=PATH` pairs are passed via
    `--cmake_extra_defines` (`ORT_FETCH_URL`, `GENAI_FETCH_URL`,
@@ -297,17 +297,17 @@ a PowerShell and a bash implementation behind a `shell` parameter. The
 PowerShell build steps consume its `cmakeFetchDefines` variable; Linux
 container builds use the downloaded package paths directly.
 
-WinML downloads `Microsoft.Windows.AI.MachineLearning` directly from
-nuget.org as a single self-contained reg-free package — no transitive
-WinAppSDK Foundation resolution needed. The bash branch fails fast if
-`includeWinml=true` is ever passed — WinML is Windows-only.
+WinML downloads `Microsoft.Windows.AI.MachineLearning` through
+`AIFoundryLocal_PublicPackages` as a single self-contained reg-free package,
+so no transitive WinAppSDK Foundation resolution is needed. The bash branch
+fails fast if `includeWinml=true` is ever passed because WinML is Windows-only.
 
 ## Test data
 
-All three SDKs honor `FOUNDRY_TEST_DATA_DIR` (env var) pointing at a
-checked-out `test-data-shared` working tree. CI test stages check out
-`test-data-shared` (LFS) via `.pipelines/templates/checkout-steps.yml` and
-set the env var. SDK *build* stages do not need test-data-shared.
+All three SDKs honor `FOUNDRY_TEST_DATA_DIR` (env var) pointing at a local
+model cache directory. CI test stages fetch models via
+`.pipelines/templates/fetch-models-from-artifacts-feed.yml` and set the env
+var. SDK *build* stages do not need the model cache.
 
 Test data is fetched on every stage that runs tests:
 
@@ -361,7 +361,7 @@ Build output directories follow `build.py`'s convention:
 ## Things explicitly out of scope
 
 - No `foundry-local-runtime` standalone wheel.
-- No multi-repo `Foundry-Local`/`test-data-shared` path-juggling logic in
+- No multi-repo `Foundry-Local`/shared-cache path-juggling logic in
   the sdk_v2 templates — sdk_v2 paths are repo-relative.
 - No private Azure DevOps feed dependency for the Python wheel install
   path — ORT/GenAI come from public PyPI (decision 9).
@@ -381,11 +381,11 @@ Build output directories follow `build.py`'s convention:
 
 ## Files
 
-* Coordinator: [stages-sdk-v2.yml](templates/stages-sdk-v2.yml)
-* Native build/pack: [templates/stages-build-native.yml](templates/stages-build-native.yml)
-* C# stages: [templates/stages-cs.yml](templates/stages-cs.yml)
-* Python stages: [templates/stages-python.yml](templates/stages-python.yml)
-* Rust stages: [templates/stages-rust.yml](templates/stages-rust.yml), [templates/steps-build-rust.yml](templates/steps-build-rust.yml)
-* Pre-fetch: [templates/steps-prefetch-nuget.yml](templates/steps-prefetch-nuget.yml)
+* Coordinator: [stages-sdk-v2.yml](../templates/stages-sdk-v2.yml)
+* Native build/pack: [templates/stages-build-native.yml](../templates/stages-build-native.yml)
+* C# stages: [templates/stages-cs.yml](../templates/stages-cs.yml)
+* Python stages: [templates/stages-python.yml](../templates/stages-python.yml)
+* Rust stages: [templates/stages-rust.yml](../templates/stages-rust.yml), [templates/steps-build-rust.yml](../templates/steps-build-rust.yml)
+* Pre-fetch: [templates/steps-prefetch-nuget.yml](../templates/steps-prefetch-nuget.yml)
 * Pack tool: [sdk_v2/cpp/nuget/pack.py](../../sdk_v2/cpp/nuget/pack.py)
 * Top-level pipeline: [foundry-local-packaging.yml](../foundry-local-packaging.yml)

.pipelines/foundry-local-packaging.yml

@@ -1,11 +1,8 @@
 # Foundry Local Packaging Pipeline
 #
-# Builds Foundry Local Core from neutron-server (windows.ai.toolkit project),
-# then packages the C# and JS SDKs from this repo using the built Core.
+# Builds the sdk_v2 native runtime and packages the C#, JS, Python, and Rust SDKs.
 #
-# Produces artifacts: flc-nuget, flc-nuget-winml, flc-wheels, flc-wheels-winml,
-#   cs-sdk, cs-sdk-winml, js-sdk, js-sdk-winml, python-sdk, python-sdk-winml,
-#   rust-sdk, rust-sdk-winml
+# Produces the sdk_v2 native runtime, C#, JS, Python, and Rust artifacts.
 
 # CI: only auto-run on pushes to main / release branches. Without an explicit
 # trigger block, ADO defaults to "every commit on every branch", which fires
@@ -30,10 +27,6 @@ parameters:
   displayName: 'Package version (sdk_v2/...)'
   type: string
   default: '2.0.0'
-- name: versionV1
-  displayName: 'Package version (sdk/...)'
-  type: string
-  default: '1.3.0'
 - name: prereleaseId
   displayName: 'Pre-release identifier (e.g. rc1, beta).'
   type: string
@@ -42,14 +35,6 @@ parameters:
   displayName: 'Release build'
   type: boolean
   default: false
-- name: buildV1
-  displayName: 'Force build v1 sdk. If unchecked, auto-detected from PR changes.'
-  type: boolean
-  default: false
-- name: neutronServerBranch
-  displayName: 'Foundry Local Core branch (windows.ai.toolkit/neutron-server)'
-  type: string
-  default: 'dev/FoundryLocalCore/main'
 
 variables:
 - group: FoundryLocal-ESRP-Signing
@@ -85,93 +70,13 @@ extends:
         break: false
         scanOutputDirectoryOnly: true
     stages:
-    # ── Detect Changed Paths ──
-    # Decides whether the legacy sdk/ branch should run. sdk_v2 always builds,
-    # so there is no buildV2 flag here. Emits one output variable on the
-    # `setflags` job: `buildV1` ("true"/"false").
-    #
-    # Rules:
-    #   * `buildV1` parameter set true => buildV1 = true.
-    #   * Otherwise => buildV1 = true iff this branch has changes under sdk/
-    #     relative to its base. Base is the PR target branch when available,
-    #     else origin/main. If that range is empty (e.g. post-merge CI build
-    #     of main itself), fall back to the latest commit's diff so a merge
-    #     that touched sdk/ still triggers v1. Branches forked off something
-    #     other than main and built outside a PR will over-trigger; that's
-    #     acceptable — release branches aren't expected to touch sdk/.
-    - stage: detect_changes
-      displayName: 'Detect Changed Paths'
-      dependsOn: []
-      jobs:
-      - job: setflags
-        displayName: 'Compute build flags'
-        pool:
-          name: onnxruntime-Win-CPU-2022
-          os: windows
-        steps:
-        - checkout: self
-          fetchDepth: 0
-        - task: PowerShell@2
-          name: setflags
-          displayName: 'Detect changed paths and set buildV1'
-          inputs:
-            targetType: inline
-            pwsh: true
-            script: |
-              $forceV1 = "${{ parameters.buildV1 }}" -eq "True"
-
-              if ($forceV1) {
-                $buildV1 = $true
-                Write-Host "Manual override: buildV1=$buildV1"
-              } else {
-                # Read via env var so PowerShell doesn't mis-parse the literal
-                # '$(System.PullRequest.TargetBranch)' as a subexpression when
-                # ADO leaves it unresolved on non-PR runs. Env var is empty
-                # (not undefined) when the variable isn't set.
-                $prTarget = $env:SYSTEM_PULLREQUEST_TARGETBRANCH
-                if (-not [string]::IsNullOrEmpty($prTarget)) {
-                  # Typically "refs/heads/<branch>"; normalize to a short name for git fetch + origin/<branch> refs.
-                  $baseBranch = $prTarget -replace '^refs/heads/', ''
-                  Write-Host "Base branch (from PR target): $baseBranch"
-                } else {
-                  $baseBranch = "main"
-                  Write-Host "Base branch (fallback): $baseBranch"
-                }
-
-                git fetch --no-tags --depth=50 origin $baseBranch 2>&1 | Out-Null
-                $diffRange = "origin/$baseBranch...HEAD"
-                $changed = @(git diff --name-only $diffRange)
-                Write-Host "Changed files vs $diffRange ($($changed.Count)):"
-                $changed | ForEach-Object { Write-Host "  $_" }
-
-                # On a build of the base branch itself (e.g. post-merge CI on main),
-                # the range above is empty. Fall back to the latest commit's diff so
-                # a merge that touched sdk/ still triggers v1.
-                if ($changed.Count -eq 0) {
-                  $diffRange = "HEAD~1...HEAD"
-                  $changed = @(git diff --name-only $diffRange)
-                  Write-Host "Empty range vs base; falling back to $diffRange ($($changed.Count)):"
-                  $changed | ForEach-Object { Write-Host "  $_" }
-                }
-
-                $buildV1 = @($changed | Where-Object { $_ -like 'sdk/*' }).Count -gt 0
-                Write-Host "Auto-detected: buildV1=$buildV1"
-              }
-
-              $v1 = $buildV1.ToString().ToLower()
-              Write-Host "##vso[task.setvariable variable=buildV1;isOutput=true]$v1"
-
     # ── Compute Version ──
     # A single version string is computed once and shared across all stages.
     # This prevents timestamp drift between standard and WinML builds.
     # Outputs (written to the `version-info` pipeline artifact):
-    #   sdk_v2 (no suffix; FLC is legacy-only and not emitted here):
+    #   sdk_v2:
     #     sdkVersion.txt    – semver for C#       (e.g. X.Y.Z-dev.202604061234)
     #     pyVersion.txt     – PEP 440 for Python  (e.g. X.Y.Z.dev202604061234)
-    #   Legacy sdk/ (.v1. suffix):
-    #     sdkVersion.v1.txt – semver for JS, C#, Rust
-    #     pyVersion.v1.txt  – PEP 440 for Python
-    #     flcVersion.v1.txt – NuGet/FLC style
     - stage: compute_version
       displayName: 'Compute Version'
       dependsOn: []
@@ -222,46 +127,19 @@ extends:
                 }
               }
 
-              $v2 = Compute-Versions "${{ parameters.version }}"   $false
-              $v1 = Compute-Versions "${{ parameters.versionV1 }}" $true
+              $v2 = Compute-Versions "${{ parameters.version }}" $false
 
               $outDir = "$(Build.ArtifactStagingDirectory)/version-info"
               New-Item -ItemType Directory -Path $outDir -Force | Out-Null
 
-              # sdk_v2 versions (canonical, no suffix). No flcVersion — v2
-              # does not consume Foundry Local Core.
               Set-Content -Path "$outDir/sdkVersion.txt" -Value $v2.sdk -NoNewline
               Set-Content -Path "$outDir/pyVersion.txt"  -Value $v2.py  -NoNewline
 
-              # Legacy sdk/ versions (.v1. suffix).
-              Set-Content -Path "$outDir/sdkVersion.v1.txt" -Value $v1.sdk -NoNewline
-              Set-Content -Path "$outDir/pyVersion.v1.txt"  -Value $v1.py  -NoNewline
-              Set-Content -Path "$outDir/flcVersion.v1.txt" -Value $v1.flc -NoNewline
-
               Write-Host "v2 SDK version:    $($v2.sdk)"
               Write-Host "v2 Python version: $($v2.py)"
-              Write-Host "v1 SDK version:    $($v1.sdk)"
-              Write-Host "v1 Python version: $($v1.py)"
-              Write-Host "v1 FLC version:    $($v1.flc)"
-
-    # -- sdk_v1 (sdk/) C++ Core (FLC) + C# / JS / Python / Rust SDKs --
-    # Entire subgraph is gated by a single artificial root stage (`v1_gate`)
-    # defined inside stages-sdk-v1.yml. When detect_changes.buildV1 is false,
-    # v1_gate is skipped and every downstream v1 stage skips via the default
-    # `succeeded()` cascade.
-    - template: v1/templates/stages-sdk-v1.yml
-      parameters:
-        gateCondition: eq(dependencies.detect_changes.outputs['setflags.setflags.buildV1'], 'true')
-        gateDependsOn: [detect_changes]
-        version: ${{ parameters.versionV1 }}
-        prereleaseId: ${{ parameters.prereleaseId }}
-        isRelease: ${{ parameters.isRelease }}
-        neutronServerBranch: ${{ parameters.neutronServerBranch }}
-
-
     # ── sdk_v2 (C++ native + C# SDK + Python SDK) ──
     # Built unconditionally on every run; no detect_changes gate.
-    - template: v2/templates/stages-sdk-v2.yml
+    - template: templates/stages-sdk-v2.yml
       parameters:
         buildConfig: $(cppBuildConfig)
         ortVersion: $(cppOrtVersion)