diff --git a/.pipelines/v2/docker/manylinux/Dockerfile b/.pipelines/docker/manylinux/Dockerfile similarity index 100% rename from .pipelines/v2/docker/manylinux/Dockerfile rename to .pipelines/docker/manylinux/Dockerfile diff --git a/.pipelines/v2/sdk_v2-js-pipeline-plan.md b/.pipelines/docs/sdk_v2-js-pipeline-plan.md similarity index 94% rename from .pipelines/v2/sdk_v2-js-pipeline-plan.md rename to .pipelines/docs/sdk_v2-js-pipeline-plan.md index 1e3a04903..ba417bd66 100644 --- a/.pipelines/v2/sdk_v2-js-pipeline-plan.md +++ b/.pipelines/docs/sdk_v2-js-pipeline-plan.md @@ -99,13 +99,13 @@ Build and test on the `onnxruntime-linux-ARM64-CPU-2019` pool - `NodeTool@0` for Node 20. - Drop `js-prebuild-` contents into `sdk_v2/js/prebuilds/`. -- Check out `test-data-shared` via `checkout-steps.yml`. -- On macOS, `brew install git-lfs` before checkout (same as Python). +- Fetch the model cache via `fetch-models-from-artifacts-feed.yml`. +- On macOS, ensure the test data directory is present and writable before the run. - `npm ci --no-audit --no-fund` (no `--ignore-scripts` — install-native is allowed to run on the test side; it's a no-op when the prebuild is already in place). - `npm run build:ts`, then `npx vitest run --reporter=verbose` with - `FOUNDRY_TEST_DATA_DIR=$(testDataSharedDir)`. + `FOUNDRY_TEST_DATA_DIR=$(Build.SourcesDirectory)/test-data-shared`. ## Pack stage @@ -140,7 +140,7 @@ signing. | `js_pack` | none | n/a | `.tgz` not signed by npm convention | Windows signing block is a near-copy of the SDK DLL signing step in -[`steps-build-cs.yml`](templates/steps-build-cs.yml) — same +[`steps-build-cs.yml`](../templates/steps-build-cs.yml) — same `ConnectedServiceName`, ESRP variable group (`FoundryLocal-ESRP-Signing`), and `signConfigType: inlineSignParams` shape. @@ -148,13 +148,13 @@ shape. ## Files added / modified **New:** -- [`.pipelines/v2/templates/stages-js.yml`](templates/stages-js.yml) -- [`.pipelines/v2/templates/steps-build-js.yml`](templates/steps-build-js.yml) -- [`.pipelines/v2/templates/steps-test-js.yml`](templates/steps-test-js.yml) -- [`.pipelines/v2/templates/steps-pack-js.yml`](templates/steps-pack-js.yml) +- [`.pipelines/templates/stages-js.yml`](../templates/stages-js.yml) +- [`.pipelines/templates/steps-build-js.yml`](../templates/steps-build-js.yml) +- [`.pipelines/templates/steps-test-js.yml`](../templates/steps-test-js.yml) +- [`.pipelines/templates/steps-pack-js.yml`](../templates/steps-pack-js.yml) **Modified:** -- [`.pipelines/v2/templates/stages-sdk-v2.yml`](templates/stages-sdk-v2.yml) +- [`.pipelines/templates/stages-sdk-v2.yml`](../templates/stages-sdk-v2.yml) — appends `stages-js.yml`. - [`sdk_v2/js/script/gyp/print-import-lib-dir.mjs`](../../sdk_v2/js/script/gyp/print-import-lib-dir.mjs) — honors `FOUNDRY_LOCAL_LIB_DIR`. diff --git a/.pipelines/v2/sdk_v2-pipeline-plan.md b/.pipelines/docs/sdk_v2-pipeline-plan.md similarity index 89% rename from .pipelines/v2/sdk_v2-pipeline-plan.md rename to .pipelines/docs/sdk_v2-pipeline-plan.md index c3809c649..70239ea78 100644 --- a/.pipelines/v2/sdk_v2-pipeline-plan.md +++ b/.pipelines/docs/sdk_v2-pipeline-plan.md @@ -21,10 +21,9 @@ JS is documented separately in `sdk_v2-js-pipeline-plan.md`. A standalone There is **no dedicated sdk_v2 top-level pipeline**. The sdk_v2 stage graph is emitted by a single coordinator template, -`.pipelines/v2/templates/stages-sdk-v2.yml`, which is included by the +`.pipelines/templates/stages-sdk-v2.yml`, which is included by the existing `.pipelines/foundry-local-packaging.yml`. sdk_v2 is built -unconditionally on every run (there is no v2 gate); the v1 legacy stages -are gated separately via `.pipelines/v1/templates/stages-sdk-v1.yml`. +unconditionally on every run. ## Supported platforms @@ -54,10 +53,10 @@ are gated separately via `.pipelines/v1/templates/stages-sdk-v1.yml`. ORT and GenAI native libraries are **not** bundled in the wheel; they are resolved at install time from public PyPI (see decision 9). The wheel only contains `foundry_local` plus its non-ORT runtime closure. -4. **Test stages set `FOUNDRY_TEST_DATA_DIR`** to the checked-out - `test-data-shared` path for all three languages. C++ stages set it - directly; Python's `conftest.py` reads it; C# `Utils.cs` prefers it - over `appsettings.Test.json`. +4. **Test stages set `FOUNDRY_TEST_DATA_DIR`** to the local model-cache path + for all three languages. C++ stages set it directly; Python's + `conftest.py` reads it; C# `Utils.cs` prefers it over + `appsettings.Test.json`. 5. **Parameter naming.** sdk_v2 templates use `flNugetDir` (not `flcNugetDir`). `flc` was Foundry Local Core, which we are replacing. 6. **Shared `compute_version` stage.** Defined inline in @@ -129,7 +128,9 @@ are gated separately via `.pipelines/v1/templates/stages-sdk-v1.yml`. ## Template layout ``` -.pipelines/v2/ +.pipelines/ +├── docker/ +├── docs/ └── templates/ ├── stages-sdk-v2.yml # Coordinator: native + C# + Python + JS + Rust ├── stages-build-native.yml # 5 native build stages + C++/NuGet pack stages @@ -149,9 +150,10 @@ are gated separately via `.pipelines/v1/templates/stages-sdk-v1.yml`. └── steps-pack-nuget.yml # Runs sdk_v2/cpp/nuget/pack.py ``` -The repo-shared `.pipelines/templates/checkout-steps.yml` is reused for -`test-data-shared` (LFS) checkout via the `FoundryLocalCore-SP` Azure CLI -service connection. +The repo-shared `.pipelines/templates/fetch-models-from-artifacts-feed.yml` +pre-populates the local model cache from the AIFoundryLocal public artifacts +feed, so every test stage can point `FOUNDRY_TEST_DATA_DIR` at the same +staged cache without a Git checkout dependency. ## Stage dependency graph @@ -231,8 +233,8 @@ hand off, so: matrices. * Integration tests run with `--test-threads=1`: the native core enforces a single `FoundryLocalManager` per process, so tests that each create one must - not run concurrently. They consume the same `test-data-shared` models as the - C# / Python suites (`FOUNDRY_TEST_DATA_DIR`). + not run concurrently. They consume the same model cache as the C# / Python + suites (`FOUNDRY_TEST_DATA_DIR`). ## Versioning @@ -268,11 +270,9 @@ Pipeline parameters allow override: ## Dependency versions The pipeline pins ORT / GenAI / WinML versions and downloads the NuGet -packages from **public nuget.org** (`https://www.nuget.org/api/v2/package`) -before invoking CMake. This matches Foundry Local Core's own -[nuget.config](../templates/build-core-steps.yml), which maps everything -except `Microsoft.Telemetry*` to nuget.org. Pre-fetching serves two -purposes: +packages through `AIFoundryLocal_PublicPackages` before invoking CMake. +The feed provides the approved upstreams while preventing pipeline jobs from +accessing public package feeds directly. Pre-fetching serves two purposes: 1. **Version pinning** — the `KEY=PATH` pairs are passed via `--cmake_extra_defines` (`ORT_FETCH_URL`, `GENAI_FETCH_URL`, @@ -297,17 +297,17 @@ a PowerShell and a bash implementation behind a `shell` parameter. The PowerShell build steps consume its `cmakeFetchDefines` variable; Linux container builds use the downloaded package paths directly. -WinML downloads `Microsoft.Windows.AI.MachineLearning` directly from -nuget.org as a single self-contained reg-free package — no transitive -WinAppSDK Foundation resolution needed. The bash branch fails fast if -`includeWinml=true` is ever passed — WinML is Windows-only. +WinML downloads `Microsoft.Windows.AI.MachineLearning` through +`AIFoundryLocal_PublicPackages` as a single self-contained reg-free package, +so no transitive WinAppSDK Foundation resolution is needed. The bash branch +fails fast if `includeWinml=true` is ever passed because WinML is Windows-only. ## Test data -All three SDKs honor `FOUNDRY_TEST_DATA_DIR` (env var) pointing at a -checked-out `test-data-shared` working tree. CI test stages check out -`test-data-shared` (LFS) via `.pipelines/templates/checkout-steps.yml` and -set the env var. SDK *build* stages do not need test-data-shared. +All three SDKs honor `FOUNDRY_TEST_DATA_DIR` (env var) pointing at a local +model cache directory. CI test stages fetch models via +`.pipelines/templates/fetch-models-from-artifacts-feed.yml` and set the env +var. SDK *build* stages do not need the model cache. Test data is fetched on every stage that runs tests: @@ -361,7 +361,7 @@ Build output directories follow `build.py`'s convention: ## Things explicitly out of scope - No `foundry-local-runtime` standalone wheel. -- No multi-repo `Foundry-Local`/`test-data-shared` path-juggling logic in +- No multi-repo `Foundry-Local`/shared-cache path-juggling logic in the sdk_v2 templates — sdk_v2 paths are repo-relative. - No private Azure DevOps feed dependency for the Python wheel install path — ORT/GenAI come from public PyPI (decision 9). @@ -381,11 +381,11 @@ Build output directories follow `build.py`'s convention: ## Files -* Coordinator: [stages-sdk-v2.yml](templates/stages-sdk-v2.yml) -* Native build/pack: [templates/stages-build-native.yml](templates/stages-build-native.yml) -* C# stages: [templates/stages-cs.yml](templates/stages-cs.yml) -* Python stages: [templates/stages-python.yml](templates/stages-python.yml) -* Rust stages: [templates/stages-rust.yml](templates/stages-rust.yml), [templates/steps-build-rust.yml](templates/steps-build-rust.yml) -* Pre-fetch: [templates/steps-prefetch-nuget.yml](templates/steps-prefetch-nuget.yml) +* Coordinator: [stages-sdk-v2.yml](../templates/stages-sdk-v2.yml) +* Native build/pack: [templates/stages-build-native.yml](../templates/stages-build-native.yml) +* C# stages: [templates/stages-cs.yml](../templates/stages-cs.yml) +* Python stages: [templates/stages-python.yml](../templates/stages-python.yml) +* Rust stages: [templates/stages-rust.yml](../templates/stages-rust.yml), [templates/steps-build-rust.yml](../templates/steps-build-rust.yml) +* Pre-fetch: [templates/steps-prefetch-nuget.yml](../templates/steps-prefetch-nuget.yml) * Pack tool: [sdk_v2/cpp/nuget/pack.py](../../sdk_v2/cpp/nuget/pack.py) * Top-level pipeline: [foundry-local-packaging.yml](../foundry-local-packaging.yml) diff --git a/.pipelines/foundry-local-packaging.yml b/.pipelines/foundry-local-packaging.yml index 51d9b2365..705d8ce22 100644 --- a/.pipelines/foundry-local-packaging.yml +++ b/.pipelines/foundry-local-packaging.yml @@ -1,11 +1,8 @@ # Foundry Local Packaging Pipeline # -# Builds Foundry Local Core from neutron-server (windows.ai.toolkit project), -# then packages the C# and JS SDKs from this repo using the built Core. +# Builds the sdk_v2 native runtime and packages the C#, JS, Python, and Rust SDKs. # -# Produces artifacts: flc-nuget, flc-nuget-winml, flc-wheels, flc-wheels-winml, -# cs-sdk, cs-sdk-winml, js-sdk, js-sdk-winml, python-sdk, python-sdk-winml, -# rust-sdk, rust-sdk-winml +# Produces the sdk_v2 native runtime, C#, JS, Python, and Rust artifacts. # CI: only auto-run on pushes to main / release branches. Without an explicit # trigger block, ADO defaults to "every commit on every branch", which fires @@ -30,10 +27,6 @@ parameters: displayName: 'Package version (sdk_v2/...)' type: string default: '2.0.0' -- name: versionV1 - displayName: 'Package version (sdk/...)' - type: string - default: '1.3.0' - name: prereleaseId displayName: 'Pre-release identifier (e.g. rc1, beta).' type: string @@ -42,14 +35,6 @@ parameters: displayName: 'Release build' type: boolean default: false -- name: buildV1 - displayName: 'Force build v1 sdk. If unchecked, auto-detected from PR changes.' - type: boolean - default: false -- name: neutronServerBranch - displayName: 'Foundry Local Core branch (windows.ai.toolkit/neutron-server)' - type: string - default: 'dev/FoundryLocalCore/main' variables: - group: FoundryLocal-ESRP-Signing @@ -85,93 +70,13 @@ extends: break: false scanOutputDirectoryOnly: true stages: - # ── Detect Changed Paths ── - # Decides whether the legacy sdk/ branch should run. sdk_v2 always builds, - # so there is no buildV2 flag here. Emits one output variable on the - # `setflags` job: `buildV1` ("true"/"false"). - # - # Rules: - # * `buildV1` parameter set true => buildV1 = true. - # * Otherwise => buildV1 = true iff this branch has changes under sdk/ - # relative to its base. Base is the PR target branch when available, - # else origin/main. If that range is empty (e.g. post-merge CI build - # of main itself), fall back to the latest commit's diff so a merge - # that touched sdk/ still triggers v1. Branches forked off something - # other than main and built outside a PR will over-trigger; that's - # acceptable — release branches aren't expected to touch sdk/. - - stage: detect_changes - displayName: 'Detect Changed Paths' - dependsOn: [] - jobs: - - job: setflags - displayName: 'Compute build flags' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - steps: - - checkout: self - fetchDepth: 0 - - task: PowerShell@2 - name: setflags - displayName: 'Detect changed paths and set buildV1' - inputs: - targetType: inline - pwsh: true - script: | - $forceV1 = "${{ parameters.buildV1 }}" -eq "True" - - if ($forceV1) { - $buildV1 = $true - Write-Host "Manual override: buildV1=$buildV1" - } else { - # Read via env var so PowerShell doesn't mis-parse the literal - # '$(System.PullRequest.TargetBranch)' as a subexpression when - # ADO leaves it unresolved on non-PR runs. Env var is empty - # (not undefined) when the variable isn't set. - $prTarget = $env:SYSTEM_PULLREQUEST_TARGETBRANCH - if (-not [string]::IsNullOrEmpty($prTarget)) { - # Typically "refs/heads/"; normalize to a short name for git fetch + origin/ refs. - $baseBranch = $prTarget -replace '^refs/heads/', '' - Write-Host "Base branch (from PR target): $baseBranch" - } else { - $baseBranch = "main" - Write-Host "Base branch (fallback): $baseBranch" - } - - git fetch --no-tags --depth=50 origin $baseBranch 2>&1 | Out-Null - $diffRange = "origin/$baseBranch...HEAD" - $changed = @(git diff --name-only $diffRange) - Write-Host "Changed files vs $diffRange ($($changed.Count)):" - $changed | ForEach-Object { Write-Host " $_" } - - # On a build of the base branch itself (e.g. post-merge CI on main), - # the range above is empty. Fall back to the latest commit's diff so - # a merge that touched sdk/ still triggers v1. - if ($changed.Count -eq 0) { - $diffRange = "HEAD~1...HEAD" - $changed = @(git diff --name-only $diffRange) - Write-Host "Empty range vs base; falling back to $diffRange ($($changed.Count)):" - $changed | ForEach-Object { Write-Host " $_" } - } - - $buildV1 = @($changed | Where-Object { $_ -like 'sdk/*' }).Count -gt 0 - Write-Host "Auto-detected: buildV1=$buildV1" - } - - $v1 = $buildV1.ToString().ToLower() - Write-Host "##vso[task.setvariable variable=buildV1;isOutput=true]$v1" - # ── Compute Version ── # A single version string is computed once and shared across all stages. # This prevents timestamp drift between standard and WinML builds. # Outputs (written to the `version-info` pipeline artifact): - # sdk_v2 (no suffix; FLC is legacy-only and not emitted here): + # sdk_v2: # sdkVersion.txt – semver for C# (e.g. X.Y.Z-dev.202604061234) # pyVersion.txt – PEP 440 for Python (e.g. X.Y.Z.dev202604061234) - # Legacy sdk/ (.v1. suffix): - # sdkVersion.v1.txt – semver for JS, C#, Rust - # pyVersion.v1.txt – PEP 440 for Python - # flcVersion.v1.txt – NuGet/FLC style - stage: compute_version displayName: 'Compute Version' dependsOn: [] @@ -222,46 +127,19 @@ extends: } } - $v2 = Compute-Versions "${{ parameters.version }}" $false - $v1 = Compute-Versions "${{ parameters.versionV1 }}" $true + $v2 = Compute-Versions "${{ parameters.version }}" $false $outDir = "$(Build.ArtifactStagingDirectory)/version-info" New-Item -ItemType Directory -Path $outDir -Force | Out-Null - # sdk_v2 versions (canonical, no suffix). No flcVersion — v2 - # does not consume Foundry Local Core. Set-Content -Path "$outDir/sdkVersion.txt" -Value $v2.sdk -NoNewline Set-Content -Path "$outDir/pyVersion.txt" -Value $v2.py -NoNewline - # Legacy sdk/ versions (.v1. suffix). - Set-Content -Path "$outDir/sdkVersion.v1.txt" -Value $v1.sdk -NoNewline - Set-Content -Path "$outDir/pyVersion.v1.txt" -Value $v1.py -NoNewline - Set-Content -Path "$outDir/flcVersion.v1.txt" -Value $v1.flc -NoNewline - Write-Host "v2 SDK version: $($v2.sdk)" Write-Host "v2 Python version: $($v2.py)" - Write-Host "v1 SDK version: $($v1.sdk)" - Write-Host "v1 Python version: $($v1.py)" - Write-Host "v1 FLC version: $($v1.flc)" - - # -- sdk_v1 (sdk/) C++ Core (FLC) + C# / JS / Python / Rust SDKs -- - # Entire subgraph is gated by a single artificial root stage (`v1_gate`) - # defined inside stages-sdk-v1.yml. When detect_changes.buildV1 is false, - # v1_gate is skipped and every downstream v1 stage skips via the default - # `succeeded()` cascade. - - template: v1/templates/stages-sdk-v1.yml - parameters: - gateCondition: eq(dependencies.detect_changes.outputs['setflags.setflags.buildV1'], 'true') - gateDependsOn: [detect_changes] - version: ${{ parameters.versionV1 }} - prereleaseId: ${{ parameters.prereleaseId }} - isRelease: ${{ parameters.isRelease }} - neutronServerBranch: ${{ parameters.neutronServerBranch }} - - # ── sdk_v2 (C++ native + C# SDK + Python SDK) ── # Built unconditionally on every run; no detect_changes gate. - - template: v2/templates/stages-sdk-v2.yml + - template: templates/stages-sdk-v2.yml parameters: buildConfig: $(cppBuildConfig) ortVersion: $(cppOrtVersion) diff --git a/.pipelines/templates/checkout-steps.yml b/.pipelines/templates/checkout-steps.yml deleted file mode 100644 index 601eacf56..000000000 --- a/.pipelines/templates/checkout-steps.yml +++ /dev/null @@ -1,74 +0,0 @@ -# Clones an Azure DevOps Git repo using a service connection. -# Replaces `checkout: ` for repos that cannot use PAT-based endpoints. -# -# The repo is cloned into $(Build.SourcesDirectory)/ to match the -# directory layout that multi-checkout produces. -parameters: -- name: repoName - type: string - displayName: 'Repository name (also used as the checkout directory name)' -- name: ref - type: string - default: 'refs/heads/main' - displayName: 'Git ref to checkout (branch, tag, or commit SHA)' -- name: basePath - type: string - default: '$(Build.SourcesDirectory)' - displayName: 'Base directory to clone into (repo will be at basePath/repoName)' - -steps: -- task: AzureCLI@2 - displayName: 'Checkout ${{ parameters.repoName }}' - inputs: - azureSubscription: 'FoundryLocalCore-SP' - scriptType: pscore - scriptLocation: inlineScript - inlineScript: | - $ErrorActionPreference = 'Stop' - - $repoDir = "${{ parameters.basePath }}/${{ parameters.repoName }}" - $ref = "${{ parameters.ref }}" - - # Obtain bearer token scoped to Azure DevOps - $token = az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query accessToken -o tsv - if ($LASTEXITCODE -ne 0 -or -not $token) { throw "Failed to obtain access token" } - - # Embed the token directly in the remote URL for reliable auth - $repoUrl = "https://oauth:${token}@dev.azure.com/microsoft/windows.ai.toolkit/_git/${{ parameters.repoName }}" - - # Always clean to avoid conflicts with stale state from agent reuse - if (Test-Path $repoDir) { - Write-Host "Cleaning existing directory: $repoDir" - Remove-Item -Recurse -Force $repoDir - } - - # Create and init - New-Item -ItemType Directory -Path $repoDir -Force | Out-Null - Push-Location $repoDir - git init - git remote add origin $repoUrl - - # Fetch — use shallow clone for test-data-shared (no NBGV), full clone for others - $branch = $ref -replace '^refs/heads/', '' - $repoName = '${{ parameters.repoName }}' - if ($repoName -eq 'test-data-shared') { - git fetch origin $branch --depth=1 - } else { - git fetch origin $branch - } - if ($LASTEXITCODE -ne 0) { throw "git fetch failed for ${{ parameters.repoName }}" } - - # Checkout - git checkout "origin/$branch" -B $branch - if ($LASTEXITCODE -ne 0) { throw "git checkout failed for ${{ parameters.repoName }} at ref $ref" } - - # LFS — enable for test-data-shared (contains LFS-tracked test data) - if ($repoName -eq 'test-data-shared') { - Write-Host "Pulling LFS objects..." - git lfs install --local - git lfs pull - if ($LASTEXITCODE -ne 0) { throw "git lfs pull failed for ${{ parameters.repoName }}" } - } - - Pop-Location - Write-Host "Checked out ${{ parameters.repoName }} at $(git -C $repoDir rev-parse HEAD)" diff --git a/.pipelines/v2/templates/stages-build-native.yml b/.pipelines/templates/stages-build-native.yml similarity index 100% rename from .pipelines/v2/templates/stages-build-native.yml rename to .pipelines/templates/stages-build-native.yml diff --git a/.pipelines/v2/templates/stages-cs.yml b/.pipelines/templates/stages-cs.yml similarity index 92% rename from .pipelines/v2/templates/stages-cs.yml rename to .pipelines/templates/stages-cs.yml index 395145890..55224b6df 100644 --- a/.pipelines/v2/templates/stages-cs.yml +++ b/.pipelines/templates/stages-cs.yml @@ -67,13 +67,12 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml parameters: scriptType: ps - template: steps-test-cs.yml parameters: flNugetDir: '$(Pipeline.Workspace)/cpp-nuget' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' - stage: cs_test_win_arm64 displayName: 'C# SDK: Test Windows ARM64' @@ -96,13 +95,12 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml parameters: scriptType: ps - template: steps-test-cs.yml parameters: flNugetDir: '$(Pipeline.Workspace)/cpp-nuget' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' # ==================================================================== # Test — Linux x64 @@ -127,11 +125,10 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-test-cs.yml parameters: flNugetDir: '$(Pipeline.Workspace)/cpp-nuget' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' # ==================================================================== # Test — Linux ARM64 @@ -157,11 +154,10 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-test-cs.yml parameters: flNugetDir: '$(Pipeline.Workspace)/cpp-nuget' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' # ==================================================================== # Test — macOS ARM64 @@ -188,11 +184,10 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-test-cs.yml parameters: flNugetDir: '$(Pipeline.Workspace)/cpp-nuget' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' additionalTestArgs: '--settings $(Build.SourcesDirectory)/sdk_v2/cs/test/FoundryLocal.Tests/sequential.runsettings' # ==================================================================== diff --git a/.pipelines/v2/templates/stages-js.yml b/.pipelines/templates/stages-js.yml similarity index 90% rename from .pipelines/v2/templates/stages-js.yml rename to .pipelines/templates/stages-js.yml index c054ecc5c..6d538bf1f 100644 --- a/.pipelines/v2/templates/stages-js.yml +++ b/.pipelines/templates/stages-js.yml @@ -208,12 +208,11 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-test-js.yml parameters: rid: 'win-x64' prebuildArtifactDir: '$(Pipeline.Workspace)/js-prebuild-win-x64' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' - stage: js_test_win_arm64 displayName: 'JS SDK: Test Windows ARM64' @@ -233,14 +232,13 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml parameters: scriptType: ps - template: steps-test-js.yml parameters: rid: 'win-arm64' prebuildArtifactDir: '$(Pipeline.Workspace)/js-prebuild-win-arm64' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' - stage: js_test_linux_x64 displayName: 'JS SDK: Test Linux x64' @@ -259,12 +257,11 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-test-js.yml parameters: rid: 'linux-x64' prebuildArtifactDir: '$(Pipeline.Workspace)/js-prebuild-linux-x64' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' - stage: js_test_linux_arm64 displayName: 'JS SDK: Test Linux ARM64' @@ -284,12 +281,11 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-test-js.yml parameters: rid: 'linux-arm64' prebuildArtifactDir: '$(Pipeline.Workspace)/js-prebuild-linux-arm64' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' - stage: js_test_osx_arm64 displayName: 'JS SDK: Test macOS ARM64' @@ -310,12 +306,11 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-test-js.yml parameters: rid: 'osx-arm64' prebuildArtifactDir: '$(Pipeline.Workspace)/js-prebuild-osx-arm64' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' # ===================================================================== # Pack stage — runs after all five builds. Pure packaging on Windows. @@ -390,6 +385,23 @@ stages: displayName: 'Use Node.js 20' inputs: versionSpec: '20.x' + - task: UseDotNet@2 + displayName: 'Use .NET 10 SDK for native package restore' + inputs: + packageType: sdk + version: '10.0.x' + - task: NuGetAuthenticate@1 + displayName: 'Authenticate NuGet feeds' + - task: PowerShell@2 + displayName: 'Configure AIFoundryLocal_PublicPackages npm registry' + inputs: + targetType: inline + script: | + Set-Content '$(Agent.TempDirectory)/js-sdk-package-test.npmrc' 'registry=https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/npm/registry/' + - task: npmAuthenticate@0 + displayName: 'Authenticate AIFoundryLocal_PublicPackages for npm' + inputs: + workingFile: '$(Agent.TempDirectory)/js-sdk-package-test.npmrc' - task: PowerShell@2 displayName: 'Run packaged JS basic_chat sample' inputs: @@ -420,3 +432,6 @@ stages: if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } env: FOUNDRY_LOCAL_SAMPLE_CACHE_DIR: '$(Agent.TempDirectory)/js-package-model-cache' + FOUNDRY_LOCAL_NUGET_MODE: 'dotnet' + FOUNDRY_LOCAL_NUGET_FEEDS: 'https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/nuget/v3/index.json' + NPM_CONFIG_USERCONFIG: '$(Agent.TempDirectory)/js-sdk-package-test.npmrc' diff --git a/.pipelines/v2/templates/stages-python.yml b/.pipelines/templates/stages-python.yml similarity index 94% rename from .pipelines/v2/templates/stages-python.yml rename to .pipelines/templates/stages-python.yml index e01f9eb1e..57622de4c 100644 --- a/.pipelines/v2/templates/stages-python.yml +++ b/.pipelines/templates/stages-python.yml @@ -198,13 +198,12 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/checkout-steps.yml@self + - template: fetch-models-from-artifacts-feed.yml parameters: - repoName: test-data-shared + scriptType: ps - template: steps-test-python.yml parameters: wheelDir: '$(Pipeline.Workspace)/python-sdk-win-x64' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' - stage: python_test_win_arm64 displayName: 'Python SDK: Test Windows ARM64' @@ -224,13 +223,12 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml parameters: scriptType: ps - template: steps-test-python.yml parameters: wheelDir: '$(Pipeline.Workspace)/python-sdk-win-arm64' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' pythonArchitecture: 'arm64' - stage: python_test_linux_x64 @@ -250,11 +248,10 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-test-python.yml parameters: wheelDir: '$(Pipeline.Workspace)/python-sdk-linux-x64' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' useSystemPython: true - stage: python_test_linux_arm64 @@ -275,11 +272,10 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-test-python.yml parameters: wheelDir: '$(Pipeline.Workspace)/python-sdk-linux-arm64' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' useSystemPython: true - stage: python_test_osx_arm64 @@ -301,11 +297,10 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-test-python.yml parameters: wheelDir: '$(Pipeline.Workspace)/python-sdk-osx-arm64' - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' pythonArchitecture: 'arm64' # ================================================================ @@ -414,6 +409,10 @@ stages: versionSpec: '3.12' addToPath: true architecture: 'x64' + - task: PipAuthenticate@1 + displayName: 'Authenticate AIFoundryLocal_PublicPackages for pip' + inputs: + artifactFeeds: 'AIFoundryLocal/AIFoundryLocal_PublicPackages' - task: PowerShell@2 displayName: 'Run packaged Python chat_completion sample' inputs: diff --git a/.pipelines/v2/templates/stages-rust.yml b/.pipelines/templates/stages-rust.yml similarity index 85% rename from .pipelines/v2/templates/stages-rust.yml rename to .pipelines/templates/stages-rust.yml index c8ee1aa9b..42bb5acc6 100644 --- a/.pipelines/v2/templates/stages-rust.yml +++ b/.pipelines/templates/stages-rust.yml @@ -42,13 +42,12 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-build-rust.yml parameters: rid: 'win-x64' nativeArtifactDir: '$(Pipeline.Workspace)/cpp-native-win-x64' runTests: true - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' - stage: rust_build_win_arm64 displayName: 'Rust SDK: Build Windows ARM64 (cross)' @@ -99,13 +98,12 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-build-rust.yml parameters: rid: 'linux-x64' nativeArtifactDir: '$(Pipeline.Workspace)/cpp-native-linux-x64' runTests: true - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' - stage: rust_build_linux_arm64 displayName: 'Rust SDK: Build + Test Linux ARM64' @@ -130,13 +128,12 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-build-rust.yml parameters: rid: 'linux-arm64' nativeArtifactDir: '$(Pipeline.Workspace)/cpp-native-linux-arm64' runTests: true - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' - stage: rust_build_osx_arm64 displayName: 'Rust SDK: Build + Test macOS ARM64' @@ -162,13 +159,12 @@ stages: steps: - checkout: self clean: true - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml - template: steps-build-rust.yml parameters: rid: 'osx-arm64' nativeArtifactDir: '$(Pipeline.Workspace)/cpp-native-osx-arm64' runTests: true - testDataSharedDir: '$(Build.SourcesDirectory)/test-data-shared' # ================================================================ # Pack the platform-independent crate (rust-sdk-v2). Runs only if @@ -220,6 +216,36 @@ stages: $cargoBin = Join-Path $env:USERPROFILE '.cargo\bin' } Write-Host "##vso[task.prependpath]$cargoBin" + $env:PATH = "$cargoBin$([IO.Path]::PathSeparator)$env:PATH" + + $cargoHome = if ($env:CARGO_HOME) { + $env:CARGO_HOME + } elseif ($env:HOME) { + Join-Path $env:HOME '.cargo' + } else { + Join-Path $env:USERPROFILE '.cargo' + } + New-Item -ItemType Directory -Force -Path $cargoHome | Out-Null + @" + [source.crates-io] + replace-with = "AIFoundryLocal_PublicPackages" + + [source.AIFoundryLocal_PublicPackages] + registry = "sparse+https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/Cargo/index/" + + [registries.AIFoundryLocal_PublicPackages] + index = "sparse+https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/Cargo/index/" + + [registry] + global-credential-providers = ["cargo:token"] + "@ | Set-Content (Join-Path $cargoHome 'config.toml') + + if (-not $env:SYSTEM_ACCESSTOKEN) { throw 'System.AccessToken is required for Cargo feed authentication.' } + $credential = 'Basic ' + [Convert]::ToBase64String( + [Text.Encoding]::UTF8.GetBytes("build:$($env:SYSTEM_ACCESSTOKEN)")) + $credential | cargo login --registry AIFoundryLocal_PublicPackages + env: + SYSTEM_ACCESSTOKEN: $(System.AccessToken) - task: PowerShell@2 displayName: 'Set crate version from sdkVersion.txt' inputs: diff --git a/.pipelines/v2/templates/stages-sdk-v2.yml b/.pipelines/templates/stages-sdk-v2.yml similarity index 100% rename from .pipelines/v2/templates/stages-sdk-v2.yml rename to .pipelines/templates/stages-sdk-v2.yml diff --git a/.pipelines/v2/templates/stages-test-engine.yml b/.pipelines/templates/stages-test-engine.yml similarity index 100% rename from .pipelines/v2/templates/stages-test-engine.yml rename to .pipelines/templates/stages-test-engine.yml diff --git a/.pipelines/v2/templates/steps-build-cs.yml b/.pipelines/templates/steps-build-cs.yml similarity index 98% rename from .pipelines/v2/templates/steps-build-cs.yml rename to .pipelines/templates/steps-build-cs.yml index ea236dda5..1044b88cc 100644 --- a/.pipelines/v2/templates/steps-build-cs.yml +++ b/.pipelines/templates/steps-build-cs.yml @@ -66,7 +66,6 @@ steps: - diff --git a/.pipelines/v2/templates/steps-build-js.yml b/.pipelines/templates/steps-build-js.yml similarity index 97% rename from .pipelines/v2/templates/steps-build-js.yml rename to .pipelines/templates/steps-build-js.yml index 27d85453f..95513fd7d 100644 --- a/.pipelines/v2/templates/steps-build-js.yml +++ b/.pipelines/templates/steps-build-js.yml @@ -41,6 +41,15 @@ steps: inputs: versionSpec: '20.x' +- powershell: | + Set-Content '$(Build.SourcesDirectory)/sdk_v2/js/.npmrc' 'registry=https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/npm/registry/' + displayName: 'Configure AIFoundryLocal_PublicPackages npm registry' + +- task: npmAuthenticate@0 + displayName: 'Authenticate AIFoundryLocal_PublicPackages for npm' + inputs: + workingFile: '$(Build.SourcesDirectory)/sdk_v2/js/.npmrc' + - ${{ if eq(parameters.rid, 'win-x64') }}: - task: UsePythonVersion@0 displayName: 'Use Python 3.12 (x64)' diff --git a/.pipelines/v2/templates/steps-build-linux.yml b/.pipelines/templates/steps-build-linux.yml similarity index 93% rename from .pipelines/v2/templates/steps-build-linux.yml rename to .pipelines/templates/steps-build-linux.yml index b345bd521..4e0365525 100644 --- a/.pipelines/v2/templates/steps-build-linux.yml +++ b/.pipelines/templates/steps-build-linux.yml @@ -22,6 +22,11 @@ parameters: steps: +- task: PipAuthenticate@1 + displayName: 'Authenticate AIFoundryLocal_PublicPackages for pip' + inputs: + artifactFeeds: 'AIFoundryLocal/AIFoundryLocal_PublicPackages' + - bash: | set -euo pipefail case '${{ parameters.arch }}' in @@ -39,10 +44,10 @@ steps: image_id_file="$(Build.BinariesDirectory)/manylinux-${{ parameters.arch }}.iid" docker build \ --build-arg "BASE_IMAGE=$image" \ - --file "$(Build.SourcesDirectory)/.pipelines/v2/docker/manylinux/Dockerfile" \ + --file "$(Build.SourcesDirectory)/.pipelines/docker/manylinux/Dockerfile" \ --iidfile "$image_id_file" \ --tag "$build_image" \ - "$(Build.SourcesDirectory)/.pipelines/v2/docker/manylinux" + "$(Build.SourcesDirectory)/.pipelines/docker/manylinux" IFS= read -r image_id < "$image_id_file" || [ -n "$image_id" ] echo "##vso[task.setvariable variable=manylinuxImage]$image_id" displayName: 'Prepare manylinux 2.28 build image' @@ -55,7 +60,7 @@ steps: includeWinml: false - ${{ if eq(parameters.runTests, true) }}: - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml parameters: destinationPath: '$(Agent.BuildDirectory)/test-data-shared' @@ -64,6 +69,7 @@ steps: version=$(tr -d '[:space:]' < "$(Pipeline.Workspace)/version-info/sdkVersion.txt") home_dir="$(Build.BinariesDirectory)/manylinux-home" mkdir -p "$home_dir" + : "${PIP_INDEX_URL:?PipAuthenticate did not set PIP_INDEX_URL}" user_id=$(id -u) group_id=$(id -g) @@ -75,6 +81,8 @@ steps: --env "VCPKG_ROOT=$(Build.BinariesDirectory)/vcpkg" \ --env "NUGET_CACHE=$(Build.BinariesDirectory)/nuget_packages" \ --env "SDK_VERSION=$version" \ + --env PIP_INDEX_URL \ + --env PIP_NO_INPUT=1 \ --workdir "$(Build.SourcesDirectory)/sdk_v2/cpp" \ "$(manylinuxImage)" \ bash -c ' diff --git a/.pipelines/v2/templates/steps-build-macos.yml b/.pipelines/templates/steps-build-macos.yml similarity index 98% rename from .pipelines/v2/templates/steps-build-macos.yml rename to .pipelines/templates/steps-build-macos.yml index ebcfcbea5..0a834b05a 100644 --- a/.pipelines/v2/templates/steps-build-macos.yml +++ b/.pipelines/templates/steps-build-macos.yml @@ -50,7 +50,7 @@ steps: displayName: 'Append version define' - ${{ if eq(parameters.runTests, true) }}: - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml parameters: destinationPath: '$(Agent.BuildDirectory)/test-data-shared' diff --git a/.pipelines/v2/templates/steps-build-python.yml b/.pipelines/templates/steps-build-python.yml similarity index 98% rename from .pipelines/v2/templates/steps-build-python.yml rename to .pipelines/templates/steps-build-python.yml index f87a0e383..34ffc2bc9 100644 --- a/.pipelines/v2/templates/steps-build-python.yml +++ b/.pipelines/templates/steps-build-python.yml @@ -60,6 +60,11 @@ steps: Write-Host "##vso[task.prependpath]$venvBin" Write-Host "Using system Python through build venv: $venvDir" +- task: PipAuthenticate@1 + displayName: 'Authenticate AIFoundryLocal_PublicPackages for pip' + inputs: + artifactFeeds: 'AIFoundryLocal/AIFoundryLocal_PublicPackages' + - task: PowerShell@2 displayName: 'Set package version from pyVersion.txt' inputs: @@ -270,6 +275,7 @@ steps: out_dir="${{ parameters.outputDir }}" home_dir="$(Agent.TempDirectory)/manylinux-python-home" mkdir -p "$raw_dir" "$out_dir" "$home_dir" + : "${PIP_INDEX_URL:?PipAuthenticate did not set PIP_INDEX_URL}" user_id=$(id -u) group_id=$(id -g) docker version @@ -285,6 +291,8 @@ steps: --env "RAW_WHEEL_DIR=$raw_dir" \ --env "WHEEL_DIR=$out_dir" \ --env "AUDITWHEEL_PLAT=$platform_tag" \ + --env PIP_INDEX_URL \ + --env PIP_NO_INPUT=1 \ --workdir "$(Build.SourcesDirectory)/sdk_v2/python" \ "$image" \ bash -c ' diff --git a/.pipelines/v2/templates/steps-build-rust.yml b/.pipelines/templates/steps-build-rust.yml similarity index 86% rename from .pipelines/v2/templates/steps-build-rust.yml rename to .pipelines/templates/steps-build-rust.yml index 9d8bc9e82..8c4e6170f 100644 --- a/.pipelines/v2/templates/steps-build-rust.yml +++ b/.pipelines/templates/steps-build-rust.yml @@ -38,12 +38,15 @@ parameters: type: string default: '' # e.g. 'aarch64-pc-windows-msvc' for the win-arm64 cross build; '' for native. -- name: testDataSharedDir - type: string - default: '' steps: +- powershell: | + $feed = 'https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/nuget/v3/index.json' + Write-Host "##vso[task.setvariable variable=FOUNDRY_LOCAL_NUGET_FEEDS]$feed" + $env:FOUNDRY_LOCAL_NUGET_FEEDS = $feed + displayName: 'Configure Rust native package feed' + # ── 1. Rust toolchain ──────────────────────────────────────────────────────── - task: PowerShell@2 displayName: 'Install Rust toolchain (stable + clippy + rustfmt)' @@ -86,9 +89,38 @@ steps: rustup target add $cross } + $cargoHome = if ($env:CARGO_HOME) { + $env:CARGO_HOME + } elseif ($env:HOME) { + Join-Path $env:HOME '.cargo' + } else { + Join-Path $env:USERPROFILE '.cargo' + } + New-Item -ItemType Directory -Force -Path $cargoHome | Out-Null + @" + [source.crates-io] + replace-with = "AIFoundryLocal_PublicPackages" + + [source.AIFoundryLocal_PublicPackages] + registry = "sparse+https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/Cargo/index/" + + [registries.AIFoundryLocal_PublicPackages] + index = "sparse+https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/Cargo/index/" + + [registry] + global-credential-providers = ["cargo:token"] + "@ | Set-Content (Join-Path $cargoHome 'config.toml') + + if (-not $env:SYSTEM_ACCESSTOKEN) { throw 'System.AccessToken is required for Cargo feed authentication.' } + $credential = 'Basic ' + [Convert]::ToBase64String( + [Text.Encoding]::UTF8.GetBytes("build:$($env:SYSTEM_ACCESSTOKEN)")) + $credential | cargo login --registry AIFoundryLocal_PublicPackages + rustup show cargo --version rustc --version + env: + SYSTEM_ACCESSTOKEN: $(System.AccessToken) # ── 2. Assemble the native runtime directory ───────────────────────────────── - task: PowerShell@2 @@ -235,6 +267,7 @@ steps: env: FOUNDRY_LOCAL_NATIVE_BIN_DIR: $(flNativeDir) FOUNDRY_LOCAL_LIB_DIR: $(flNativeDir) + FOUNDRY_TEST_DATA_DIR: '$(Build.SourcesDirectory)/test-data-shared' - task: PowerShell@2 displayName: 'cargo test --tests (integration)' @@ -251,5 +284,5 @@ steps: env: FOUNDRY_LOCAL_NATIVE_BIN_DIR: $(flNativeDir) FOUNDRY_LOCAL_LIB_DIR: $(flNativeDir) - FOUNDRY_TEST_DATA_DIR: ${{ parameters.testDataSharedDir }} + FOUNDRY_TEST_DATA_DIR: '$(Build.SourcesDirectory)/test-data-shared' TF_BUILD: 'true' diff --git a/.pipelines/v2/templates/steps-build-windows.yml b/.pipelines/templates/steps-build-windows.yml similarity index 98% rename from .pipelines/v2/templates/steps-build-windows.yml rename to .pipelines/templates/steps-build-windows.yml index 69802dad6..0d5b4da58 100644 --- a/.pipelines/v2/templates/steps-build-windows.yml +++ b/.pipelines/templates/steps-build-windows.yml @@ -77,7 +77,7 @@ steps: # Tests need shared model files. - ${{ if eq(parameters.runTests, true) }}: - - template: ../../templates/fetch-models-from-artifacts-feed.yml@self + - template: fetch-models-from-artifacts-feed.yml parameters: destinationPath: '$(Agent.BuildDirectory)/test-data-shared' scriptType: ps diff --git a/.pipelines/v2/templates/steps-pack-cpp-sdk.yml b/.pipelines/templates/steps-pack-cpp-sdk.yml similarity index 100% rename from .pipelines/v2/templates/steps-pack-cpp-sdk.yml rename to .pipelines/templates/steps-pack-cpp-sdk.yml diff --git a/.pipelines/v2/templates/steps-pack-js.yml b/.pipelines/templates/steps-pack-js.yml similarity index 88% rename from .pipelines/v2/templates/steps-pack-js.yml rename to .pipelines/templates/steps-pack-js.yml index 4f7481c54..b56e8c736 100644 --- a/.pipelines/v2/templates/steps-pack-js.yml +++ b/.pipelines/templates/steps-pack-js.yml @@ -28,6 +28,15 @@ steps: inputs: versionSpec: '20.x' +- powershell: | + Set-Content '$(Build.SourcesDirectory)/sdk_v2/js/.npmrc' 'registry=https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/npm/registry/' + displayName: 'Configure AIFoundryLocal_PublicPackages npm registry' + +- task: npmAuthenticate@0 + displayName: 'Authenticate AIFoundryLocal_PublicPackages for npm' + inputs: + workingFile: '$(Build.SourcesDirectory)/sdk_v2/js/.npmrc' + - pwsh: | $dst = "$(Build.SourcesDirectory)/sdk_v2/js/prebuilds" if (Test-Path $dst) { Remove-Item -Recurse -Force $dst } diff --git a/.pipelines/v2/templates/steps-pack-nuget.yml b/.pipelines/templates/steps-pack-nuget.yml similarity index 100% rename from .pipelines/v2/templates/steps-pack-nuget.yml rename to .pipelines/templates/steps-pack-nuget.yml diff --git a/.pipelines/v2/templates/steps-prefetch-nuget.yml b/.pipelines/templates/steps-prefetch-nuget.yml similarity index 100% rename from .pipelines/v2/templates/steps-prefetch-nuget.yml rename to .pipelines/templates/steps-prefetch-nuget.yml diff --git a/.pipelines/v2/templates/steps-test-cs.yml b/.pipelines/templates/steps-test-cs.yml similarity index 96% rename from .pipelines/v2/templates/steps-test-cs.yml rename to .pipelines/templates/steps-test-cs.yml index ef3f7321b..068536d6f 100644 --- a/.pipelines/v2/templates/steps-test-cs.yml +++ b/.pipelines/templates/steps-test-cs.yml @@ -2,15 +2,12 @@ # (FoundryLocal.Tests). The caller is responsible for: # - placing the Foundry Local Runtime nupkg in `flNugetDir` # - downloading the `version-info` pipeline artifact -# - populating the model cache directory at `testDataSharedDir` +# - pre-populating the model cache via fetch-models-from-artifacts-feed.yml parameters: - name: flNugetDir type: string displayName: 'Path to directory containing the Foundry Local Runtime .nupkg' -- name: testDataSharedDir - type: string - displayName: 'Absolute path to the model cache directory used by tests' - name: additionalTestArgs type: string default: '' @@ -103,7 +100,6 @@ steps: - @@ -225,4 +221,4 @@ steps: } env: TF_BUILD: 'true' - FOUNDRY_TEST_DATA_DIR: ${{ parameters.testDataSharedDir }} + FOUNDRY_TEST_DATA_DIR: '$(Build.SourcesDirectory)/test-data-shared' diff --git a/.pipelines/v2/templates/steps-test-js.yml b/.pipelines/templates/steps-test-js.yml similarity index 81% rename from .pipelines/v2/templates/steps-test-js.yml rename to .pipelines/templates/steps-test-js.yml index d75c22fa0..dd819486b 100644 --- a/.pipelines/v2/templates/steps-test-js.yml +++ b/.pipelines/templates/steps-test-js.yml @@ -3,6 +3,7 @@ # The caller is responsible for placing the `js-prebuild-` artifact at # `prebuildArtifactDir`. We drop its contents into sdk_v2/js/prebuilds/-/ # so the runtime addon-loader in dist/native/load.js finds it, then run vitest. +# Model cache is pre-populated via fetch-models-from-artifacts-feed.yml. parameters: - name: rid @@ -12,9 +13,6 @@ parameters: - name: prebuildArtifactDir type: string displayName: 'Path to the downloaded js-prebuild- artifact directory (root with prebuilds/ inside)' -- name: testDataSharedDir - type: string - displayName: 'Path to the model cache directory used by tests' steps: @@ -23,6 +21,24 @@ steps: inputs: versionSpec: '20.x' +- task: UseDotNet@2 + displayName: 'Use .NET 10 SDK for native package restore' + inputs: + packageType: sdk + version: '10.0.x' + +- task: NuGetAuthenticate@1 + displayName: 'Authenticate NuGet feeds' + +- powershell: | + Set-Content '$(Build.SourcesDirectory)/sdk_v2/js/.npmrc' 'registry=https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/npm/registry/' + displayName: 'Configure AIFoundryLocal_PublicPackages npm registry' + +- task: npmAuthenticate@0 + displayName: 'Authenticate AIFoundryLocal_PublicPackages for npm' + inputs: + workingFile: '$(Build.SourcesDirectory)/sdk_v2/js/.npmrc' + - ${{ if or(eq(parameters.rid, 'win-x64'), eq(parameters.rid, 'win-arm64')) }}: - task: PowerShell@2 displayName: 'Drop prebuild into sdk_v2/js/prebuilds' @@ -96,6 +112,9 @@ steps: command: custom workingDir: '$(Build.SourcesDirectory)/sdk_v2/js' customCommand: 'ci --no-audit --no-fund' + env: + FOUNDRY_LOCAL_NUGET_MODE: 'dotnet' + FOUNDRY_LOCAL_NUGET_FEEDS: 'https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/nuget/v3/index.json' - task: PowerShell@2 displayName: 'Build TypeScript' @@ -120,7 +139,7 @@ steps: npx --no-install vitest run --reporter=verbose if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } env: - FOUNDRY_TEST_DATA_DIR: '${{ parameters.testDataSharedDir }}' + FOUNDRY_TEST_DATA_DIR: '$(Build.SourcesDirectory)/test-data-shared' - ${{ if or(eq(parameters.rid, 'linux-x64'), eq(parameters.rid, 'linux-arm64'), eq(parameters.rid, 'osx-arm64')) }}: - bash: | @@ -128,6 +147,9 @@ steps: cd "$(Build.SourcesDirectory)/sdk_v2/js" npm ci --no-audit --no-fund displayName: 'npm ci (runs install-native postinstall)' + env: + FOUNDRY_LOCAL_NUGET_MODE: 'dotnet' + FOUNDRY_LOCAL_NUGET_FEEDS: 'https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/nuget/v3/index.json' - bash: | set -euo pipefail @@ -138,10 +160,7 @@ steps: - bash: | set -euo pipefail cd "$(Build.SourcesDirectory)/sdk_v2/js" - echo "FOUNDRY_TEST_DATA_DIR=${FOUNDRY_TEST_DATA_DIR}" - test -d "${FOUNDRY_TEST_DATA_DIR}" - ls -la "${FOUNDRY_TEST_DATA_DIR}" npx --no-install vitest run --reporter=verbose displayName: 'Run vitest' env: - FOUNDRY_TEST_DATA_DIR: '${{ parameters.testDataSharedDir }}' + FOUNDRY_TEST_DATA_DIR: '$(Build.SourcesDirectory)/test-data-shared' diff --git a/.pipelines/v2/templates/steps-test-python.yml b/.pipelines/templates/steps-test-python.yml similarity index 95% rename from .pipelines/v2/templates/steps-test-python.yml rename to .pipelines/templates/steps-test-python.yml index 96f4bbf73..1debcc629 100644 --- a/.pipelines/v2/templates/steps-test-python.yml +++ b/.pipelines/templates/steps-test-python.yml @@ -3,15 +3,12 @@ # # The caller is responsible for: # * placing the built wheel (.whl) under `wheelDir` -# * populating the model cache directory at `testDataSharedDir` +# * pre-populating the model cache via fetch-models-from-artifacts-feed.yml parameters: - name: wheelDir type: string displayName: 'Path to directory containing the built foundry_local_sdk wheel' -- name: testDataSharedDir - type: string - displayName: 'Absolute path to a checked-out test-data-shared working tree' - name: pythonArchitecture type: string default: 'x64' @@ -51,6 +48,11 @@ steps: Write-Host "##vso[task.prependpath]$userBin" Write-Host "Using system Python: $python3Path" +- task: PipAuthenticate@1 + displayName: 'Authenticate AIFoundryLocal_PublicPackages for pip' + inputs: + artifactFeeds: 'AIFoundryLocal/AIFoundryLocal_PublicPackages' + - task: PowerShell@2 displayName: 'Bootstrap VC++ ARM64 runtime' condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'), eq(variables['Agent.OSArchitecture'], 'ARM64')) @@ -158,4 +160,4 @@ steps: } env: TF_BUILD: 'true' - FOUNDRY_TEST_DATA_DIR: ${{ parameters.testDataSharedDir }} + FOUNDRY_TEST_DATA_DIR: '$(Build.SourcesDirectory)/test-data-shared' diff --git a/.pipelines/v1/templates/build-core-steps.yml b/.pipelines/v1/templates/build-core-steps.yml deleted file mode 100644 index e2f13aed3..000000000 --- a/.pipelines/v1/templates/build-core-steps.yml +++ /dev/null @@ -1,177 +0,0 @@ -# Steps to build a single Foundry Local Core native AOT binary. -# Parameterized by flavor (RID) and platform (arch). -# The parent job must checkout 'neutron-server'. -parameters: -- name: flavor - type: string # e.g. win-x64, linux-x64, linux-arm64, osx-arm64 -- name: platform - type: string # e.g. x64, arm64 -- name: isWinML - type: boolean - default: false - -steps: -- task: PowerShell@2 - displayName: 'Set source paths' - inputs: - targetType: inline - script: | - $nsRoot = "$(Build.SourcesDirectory)/neutron-server" - Write-Host "##vso[task.setvariable variable=nsRoot]$nsRoot" - -- task: UseDotNet@2 - displayName: 'Use .NET SDK from global.json' - inputs: - packageType: sdk - useGlobalJson: true - workingDirectory: '$(nsRoot)' - -- task: PowerShell@2 - displayName: 'Override nuget.config' - inputs: - targetType: inline - script: | - $nugetConfig = @" - - - - - - - - - - - - - - - - - "@ - Set-Content -Path "$(nsRoot)/nuget.config" -Value $nugetConfig - Write-Host "Updated nuget.config to use nuget.org, AIFoundryLocal_PublicPackages, and Neutron with mappings" - -- ${{ if eq(parameters.isWinML, true) }}: - - task: DotNetCoreCLI@2 - displayName: 'Restore FLC Core ${{ parameters.flavor }} (WinML)' - inputs: - command: restore - projects: '$(nsRoot)/src/FoundryLocalCore/Core/Core.csproj' - restoreArguments: '-r ${{ parameters.flavor }} /p:Platform=${{ parameters.platform }} /p:IncludeWebService=true /p:Configuration=Release /p:NetTargetFramework=net9.0-windows10.0.18362.0 /p:UseWinML=true' - feedsToUse: config - nugetConfigPath: '$(nsRoot)/nuget.config' - - - task: DotNetCoreCLI@2 - displayName: 'Build FLC Core ${{ parameters.flavor }} (WinML)' - inputs: - command: build - projects: '$(nsRoot)/src/FoundryLocalCore/Core/Core.csproj' - arguments: '--no-restore -r ${{ parameters.flavor }} -f net9.0-windows10.0.18362.0 /p:Platform=${{ parameters.platform }} /p:IncludeWebService=true /p:Configuration=Release /p:NetTargetFramework=net9.0-windows10.0.18362.0 /p:UseWinML=true' - - - task: DotNetCoreCLI@2 - displayName: 'Publish FLC AOT ${{ parameters.flavor }} (WinML)' - inputs: - command: publish - projects: '$(nsRoot)/src/FoundryLocalCore/Core/Core.csproj' - arguments: '--no-restore --no-build -r ${{ parameters.flavor }} -f net9.0-windows10.0.18362.0 /p:Platform=${{ parameters.platform }} /p:Configuration=Release /p:PublishAot=true /p:NetTargetFramework=net9.0-windows10.0.18362.0 /p:UseWinML=true' - publishWebProjects: false - zipAfterPublish: false - - - ${{ if eq(parameters.flavor, 'win-x64') }}: - - task: DotNetCoreCLI@2 - displayName: 'Restore FLC Tests ${{ parameters.flavor }} (WinML)' - inputs: - command: restore - projects: '$(nsRoot)/test/FoundryLocalCore/Core/FoundryLocalCore.Tests.csproj' - restoreArguments: '-r ${{ parameters.flavor }} /p:Platform=${{ parameters.platform }} /p:IncludeWebService=true /p:Configuration=Release /p:NetTargetFramework=net9.0-windows10.0.18362.0 /p:UseWinML=true' - feedsToUse: config - nugetConfigPath: '$(nsRoot)/nuget.config' - - - task: DotNetCoreCLI@2 - displayName: 'Build FLC Tests ${{ parameters.flavor }} (WinML)' - inputs: - command: build - projects: '$(nsRoot)/test/FoundryLocalCore/Core/FoundryLocalCore.Tests.csproj' - arguments: '--no-restore -r ${{ parameters.flavor }} /p:Platform=${{ parameters.platform }} /p:IncludeWebService=true /p:Configuration=Release /p:NetTargetFramework=net9.0-windows10.0.18362.0 /p:UseWinML=true' - - - task: DotNetCoreCLI@2 - displayName: 'Test FLC ${{ parameters.flavor }} (WinML)' - inputs: - command: test - projects: '$(nsRoot)/test/FoundryLocalCore/Core/FoundryLocalCore.Tests.csproj' - arguments: '--no-build --configuration Release -r ${{ parameters.flavor }} /p:Platform=${{ parameters.platform }}' - -- ${{ if eq(parameters.isWinML, false) }}: - - task: DotNetCoreCLI@2 - displayName: 'Restore FLC Core ${{ parameters.flavor }}' - inputs: - command: restore - projects: '$(nsRoot)/src/FoundryLocalCore/Core/Core.csproj' - restoreArguments: '-r ${{ parameters.flavor }} /p:Platform=${{ parameters.platform }} /p:IncludeWebService=true /p:Configuration=Release /p:TargetFramework=net9.0' - feedsToUse: config - nugetConfigPath: '$(nsRoot)/nuget.config' - - - task: DotNetCoreCLI@2 - displayName: 'Build FLC Core ${{ parameters.flavor }}' - inputs: - command: build - projects: '$(nsRoot)/src/FoundryLocalCore/Core/Core.csproj' - arguments: '--no-restore -r ${{ parameters.flavor }} /p:Platform=${{ parameters.platform }} /p:IncludeWebService=true /p:Configuration=Release' - - # FLC tests on osx-arm64 are flaky in CI, will investigate separately. Skip for now since the main goal of this job is to produce the AOT binary. - - ${{ if or(eq(parameters.flavor, 'win-x64'), eq(parameters.flavor, 'linux-x64'), eq(parameters.flavor, 'linux-arm64')) }}: - - task: DotNetCoreCLI@2 - displayName: 'Restore FLC Tests ${{ parameters.flavor }}' - inputs: - command: restore - projects: '$(nsRoot)/test/FoundryLocalCore/Core/FoundryLocalCore.Tests.csproj' - restoreArguments: '-r ${{ parameters.flavor }} /p:Platform=${{ parameters.platform }} /p:IncludeWebService=true /p:Configuration=Release /p:TargetFramework=net9.0' - feedsToUse: config - nugetConfigPath: '$(nsRoot)/nuget.config' - - - task: DotNetCoreCLI@2 - displayName: 'Build FLC Tests ${{ parameters.flavor }}' - inputs: - command: build - projects: '$(nsRoot)/test/FoundryLocalCore/Core/FoundryLocalCore.Tests.csproj' - arguments: '--no-restore -r ${{ parameters.flavor }} /p:Platform=${{ parameters.platform }} /p:IncludeWebService=true /p:Configuration=Release' - - - task: DotNetCoreCLI@2 - displayName: 'Test FLC ${{ parameters.flavor }}' - inputs: - command: test - projects: '$(nsRoot)/test/FoundryLocalCore/Core/FoundryLocalCore.Tests.csproj' - arguments: '--no-build --configuration Release -r ${{ parameters.flavor }} /p:Platform=${{ parameters.platform }}' - - - task: DotNetCoreCLI@2 - displayName: 'Publish FLC AOT ${{ parameters.flavor }}' - inputs: - command: publish - projects: '$(nsRoot)/src/FoundryLocalCore/Core/Core.csproj' - arguments: '--no-restore --no-build -r ${{ parameters.flavor }} /p:Platform=${{ parameters.platform }} /p:Configuration=Release /p:PublishAot=true /p:TargetFramework=net9.0' - publishWebProjects: false - zipAfterPublish: false - -# Cleanup non-binary files -- task: PowerShell@2 - displayName: 'Cleanup publish artifacts' - inputs: - targetType: inline - script: | - Get-ChildItem "$(nsRoot)/artifacts/publish" -Recurse -Include "*.json", "*.xml" | - Remove-Item -Force - -# Stage the native binary for the artifact -- task: PowerShell@2 - displayName: 'Stage ${{ parameters.flavor }} binary' - inputs: - targetType: inline - script: | - $destDir = "$(Build.ArtifactStagingDirectory)/native" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Get-ChildItem "$(nsRoot)/artifacts/publish" -Recurse -File | - Where-Object { $_.Name -like "Microsoft.AI.Foundry.Local.Core.*" } | - Copy-Item -Destination $destDir -Force - Write-Host "Staged binaries:" - Get-ChildItem $destDir | ForEach-Object { Write-Host " $($_.Name)" } diff --git a/.pipelines/v1/templates/build-cs-steps.yml b/.pipelines/v1/templates/build-cs-steps.yml deleted file mode 100644 index b9bfd962a..000000000 --- a/.pipelines/v1/templates/build-cs-steps.yml +++ /dev/null @@ -1,199 +0,0 @@ -# Steps to build, sign, and pack the C# SDK NuGet package. -# When test-data-shared is checked out alongside self, ADO places repos under -# $(Build.SourcesDirectory)/. The self repo is 'Foundry-Local'. -parameters: -- name: version - type: string -- name: isRelease - type: boolean - default: false -- name: isWinML - type: boolean - default: false -- name: flcNugetDir - type: string - displayName: 'Path to directory containing the FLC .nupkg' -- name: outputDir - type: string - default: '$(Build.ArtifactStagingDirectory)/cs-sdk' - displayName: 'Path to directory for the packed SDK' -- name: prereleaseId - type: string - default: '' -- name: depsVersionsDir - type: string - default: '' - displayName: 'Path to deps-versions artifact directory' -steps: -# Set paths for multi-repo checkout -- task: PowerShell@2 - displayName: 'Set source paths' - inputs: - targetType: inline - script: | - $multiCheckout = "$(Build.SourcesDirectory)/Foundry-Local" - if (Test-Path $multiCheckout) { - $repoRoot = $multiCheckout - } else { - $repoRoot = "$(Build.SourcesDirectory)" - } - $testDataDir = "$(Build.SourcesDirectory)/test-data-shared" - Write-Host "##vso[task.setvariable variable=repoRoot]$repoRoot" - Write-Host "##vso[task.setvariable variable=testDataDir]$testDataDir" - -# Using the latest SDK does not prevent us from targeting other platforms. -# It ensures that we've got the latest tooling, fixes, and performance enhancements. -- task: UseDotNet@2 - displayName: 'Use .NET 10 SDK' - inputs: - packageType: sdk - version: '10.0.x' - -# Read version from the version-info artifact produced by compute_version stage. -# Legacy SDKs read the `.v1.txt` flavor (sdk_v2 reads `.txt`). -- task: PowerShell@2 - displayName: 'Set package version' - inputs: - targetType: inline - script: | - $v = (Get-Content "$(Pipeline.Workspace)/version-info/sdkVersion.v1.txt" -Raw).Trim() - Write-Host "Package version: $v" - Write-Host "##vso[task.setvariable variable=packageVersion]$v" - -# Load dependency versions from deps_versions.json -- template: update-deps-versions-steps.yml - parameters: - repoRoot: $(repoRoot) - artifactDir: ${{ parameters.depsVersionsDir }} - isWinML: ${{ parameters.isWinML }} - -# List downloaded artifact for debugging -- task: PowerShell@2 - displayName: 'List downloaded FLC artifact' - inputs: - targetType: inline - script: | - Write-Host "Contents of ${{ parameters.flcNugetDir }}:" - Get-ChildItem "${{ parameters.flcNugetDir }}" -Recurse | ForEach-Object { Write-Host $_.FullName } - -# Create a temporary NuGet.config that includes the local FLC feed -- task: PowerShell@2 - displayName: 'Create NuGet.config with local FLC feed' - inputs: - targetType: inline - script: | - $nugetConfig = @" - - - - - - - - - - "@ - # Point the local NuGet feed at the directory that actually contains the .nupkg - $nupkg = Get-ChildItem "${{ parameters.flcNugetDir }}" -Recurse -Filter "Microsoft.AI.Foundry.Local.Core*.nupkg" -Exclude "*.snupkg" | Select-Object -First 1 - if (-not $nupkg) { throw "No FLC .nupkg found in ${{ parameters.flcNugetDir }}" } - $flcFeedDir = $nupkg.DirectoryName - $nugetConfig = $nugetConfig -replace [regex]::Escape("${{ parameters.flcNugetDir }}"), $flcFeedDir - $configPath = "$(Build.ArtifactStagingDirectory)/NuGet.config" - Set-Content -Path $configPath -Value $nugetConfig - Write-Host "##vso[task.setvariable variable=customNugetConfig]$configPath" - Write-Host "Local FLC feed directory: $flcFeedDir" - -- task: NuGetAuthenticate@1 - displayName: 'Authenticate NuGet feeds' - -- task: PowerShell@2 - displayName: 'Restore SDK' - inputs: - targetType: inline - script: | - $proj = "$(repoRoot)/sdk/cs/src/Microsoft.AI.Foundry.Local.csproj" - if (-not (Test-Path $proj)) { throw "Project not found: $proj" } - dotnet restore $proj ` - --configfile "$(customNugetConfig)" ` - /p:UseWinML=${{ parameters.isWinML }} - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -- task: PowerShell@2 - displayName: 'Build SDK' - inputs: - targetType: inline - script: | - dotnet build "$(repoRoot)/sdk/cs/src/Microsoft.AI.Foundry.Local.csproj" ` - --no-restore --configuration Release ` - /p:UseWinML=${{ parameters.isWinML }} - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -# Discover target framework directory -- task: PowerShell@2 - displayName: 'Find target framework' - inputs: - targetType: inline - script: | - $base = "$(repoRoot)/sdk/cs/src/bin/Release" - # The SDK targets netstandard2.0 and any compatible runtime. - # Find whichever TFM directory was produced by the build. - $tfmDir = Get-ChildItem $base -Directory | Select-Object -First 1 - if (-not $tfmDir) { throw "No target framework directory found under $base" } - Write-Host "##vso[task.setvariable variable=TargetFramework]$($tfmDir.Name)" - Write-Host "Target framework: $($tfmDir.Name)" - -# Sign DLLs -- task: SFP.build-tasks.custom-build-task-1.EsrpCodeSigning@5 - displayName: 'Sign SDK DLLs' - inputs: - ConnectedServiceName: 'OnnxrunTimeCodeSign_20240611' - UseMSIAuthentication: true - AppRegistrationClientId: '$(esrpClientId)' - AppRegistrationTenantId: '$(esrpTenantId)' - EsrpClientId: '$(esrpClientId)' - AuthAKVName: '$(esrpAkvName)' - AuthSignCertName: '$(esrpSignCertName)' - FolderPath: '$(repoRoot)/sdk/cs/src/bin/Release/$(TargetFramework)' - Pattern: '*.dll' - SessionTimeout: 90 - ServiceEndpointUrl: 'https://api.esrp.microsoft.com/api/v2' - MaxConcurrency: 25 - signConfigType: inlineSignParams - inlineOperation: | - [{"keyCode":"CP-230012","operationSetCode":"SigntoolSign","parameters":[{"parameterName":"OpusName","parameterValue":"Microsoft"},{"parameterName":"OpusInfo","parameterValue":"http://www.microsoft.com"},{"parameterName":"PageHash","parameterValue":"/NPH"},{"parameterName":"FileDigest","parameterValue":"/fd sha256"},{"parameterName":"TimeStamp","parameterValue":"/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256"}],"toolName":"signtool.exe","toolVersion":"6.2.9304.0"}] - -# Pack NuGet -- task: PowerShell@2 - displayName: 'Pack NuGet' - inputs: - targetType: inline - script: | - dotnet pack "$(repoRoot)/sdk/cs/src/Microsoft.AI.Foundry.Local.csproj" ` - --no-build --no-restore --configuration Release ` - --output "${{ parameters.outputDir }}" ` - /p:PackageVersion=$(packageVersion) ` - /p:UseWinML=${{ parameters.isWinML }} ` - /p:IncludeSymbols=true ` - /p:SymbolPackageFormat=snupkg - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -# Sign NuGet package -- task: SFP.build-tasks.custom-build-task-1.EsrpCodeSigning@5 - displayName: 'Sign SDK NuGet package' - inputs: - ConnectedServiceName: 'OnnxrunTimeCodeSign_20240611' - UseMSIAuthentication: true - AppRegistrationClientId: '$(esrpClientId)' - AppRegistrationTenantId: '$(esrpTenantId)' - EsrpClientId: '$(esrpClientId)' - AuthAKVName: '$(esrpAkvName)' - AuthSignCertName: '$(esrpSignCertName)' - FolderPath: '${{ parameters.outputDir }}' - Pattern: '*.nupkg' - SessionTimeout: 90 - ServiceEndpointUrl: 'https://api.esrp.microsoft.com/api/v2' - MaxConcurrency: 25 - signConfigType: inlineSignParams - inlineOperation: | - [{"keyCode":"CP-401405","operationSetCode":"NuGetSign","parameters":[],"toolName":"sign","toolVersion":"6.2.9304.0"},{"keyCode":"CP-401405","operationSetCode":"NuGetVerify","parameters":[],"toolName":"sign","toolVersion":"6.2.9304.0"}] - diff --git a/.pipelines/v1/templates/build-js-addon-steps.yml b/.pipelines/v1/templates/build-js-addon-steps.yml deleted file mode 100644 index dce0f7fce..000000000 --- a/.pipelines/v1/templates/build-js-addon-steps.yml +++ /dev/null @@ -1,45 +0,0 @@ -# Builds the Node-API native addon for the current platform. -# Produces a foundry_local_napi.node binary under prebuilds//. -parameters: -- name: repoRoot - type: string -- name: targetArch - type: string - default: '' - displayName: 'Target architecture for cross-compilation (e.g. arm64). Empty = native.' - -steps: -- task: NodeTool@0 - displayName: 'Use Node.js 20' - inputs: - versionSpec: '20.x' - -- task: PowerShell@2 - displayName: 'Install node-gyp and node-api-headers' - inputs: - targetType: inline - pwsh: true - script: | - Set-Location "${{ parameters.repoRoot }}/sdk/js" - npm install --no-save node-gyp node-api-headers --registry https://registry.npmjs.org - -- task: PowerShell@2 - displayName: 'Build Node-API addon' - inputs: - targetType: inline - pwsh: true - script: | - Set-Location "${{ parameters.repoRoot }}/sdk/js/native" - $archFlag = "${{ parameters.targetArch }}" - if ($archFlag -ne '') { - npx node-gyp rebuild --arch=$archFlag - $platformKey = (node -e 'process.stdout.write(process.platform)') + "-$archFlag" - } else { - npx node-gyp rebuild - $platformKey = node -e 'process.stdout.write(process.platform + "-" + process.arch)' - } - $destDir = "${{ parameters.repoRoot }}/sdk/js/prebuilds/$platformKey" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Copy-Item "build/Release/foundry_local_napi.node" "$destDir/foundry_local_napi.node" -Force - Write-Host "Built addon for $platformKey -> $destDir/foundry_local_napi.node" - Get-Item "$destDir/foundry_local_napi.node" | ForEach-Object { Write-Host " Size: $($_.Length) bytes" } diff --git a/.pipelines/v1/templates/build-js-steps.yml b/.pipelines/v1/templates/build-js-steps.yml deleted file mode 100644 index 1b2a48263..000000000 --- a/.pipelines/v1/templates/build-js-steps.yml +++ /dev/null @@ -1,221 +0,0 @@ -# Steps to build and pack the JS SDK. -# When test-data-shared is checked out alongside self, ADO places repos under -# $(Build.SourcesDirectory)/. The self repo is 'Foundry-Local'. -parameters: -- name: version - type: string -- name: isRelease - type: boolean - default: false -- name: isWinML - type: boolean - default: false -- name: flcNugetDir - type: string - default: '' - displayName: 'Path to directory containing the FLC .nupkg (for tests)' -- name: prereleaseId - type: string - default: '' -- name: depsVersionsDir - type: string - default: '' - displayName: 'Path to deps-versions artifact directory' -steps: -# Set paths for multi-repo checkout -- task: PowerShell@2 - displayName: 'Set source paths' - inputs: - targetType: inline - script: | - $multiCheckout = "$(Build.SourcesDirectory)/Foundry-Local" - if (Test-Path $multiCheckout) { - $repoRoot = $multiCheckout - } else { - $repoRoot = "$(Build.SourcesDirectory)" - } - $testDataDir = "$(Build.SourcesDirectory)/test-data-shared" - Write-Host "##vso[task.setvariable variable=repoRoot]$repoRoot" - Write-Host "##vso[task.setvariable variable=testDataDir]$testDataDir" - Write-Host "Repo root: $repoRoot" - Write-Host "Test data: $testDataDir" - -- task: PowerShell@2 - displayName: 'List downloaded FLC artifact' - condition: and(succeeded(), ne('${{ parameters.flcNugetDir }}', '')) - inputs: - targetType: inline - script: | - Write-Host "Contents of ${{ parameters.flcNugetDir }}:" - Get-ChildItem "${{ parameters.flcNugetDir }}" -Recurse | ForEach-Object { Write-Host $_.FullName } - -- task: NodeTool@0 - displayName: 'Use Node.js 20' - inputs: - versionSpec: '20.x' - -# Load dependency versions from deps_versions.json -- template: update-deps-versions-steps.yml - parameters: - repoRoot: $(repoRoot) - artifactDir: ${{ parameters.depsVersionsDir }} - isWinML: ${{ parameters.isWinML }} - -# Compute version -- task: PowerShell@2 - displayName: 'Set package version' - inputs: - targetType: inline - script: | - $v = (Get-Content "$(Pipeline.Workspace)/version-info/sdkVersion.v1.txt" -Raw).Trim() - Write-Host "Package version: $v" - Write-Host "##vso[task.setvariable variable=packageVersion]$v" - -# Install JS dependencies. When a pipeline-built FLC artifact is provided, -# use --ignore-scripts to skip the native binary download (which would 404 -# on the unpublished FLC package), then extract FLC from the local artifact -# and run the install script manually to fetch ORT/GenAI from public feeds. -- task: Npm@1 - displayName: 'npm install (skip native downloads)' - inputs: - command: custom - workingDir: $(repoRoot)/sdk/js - customCommand: 'install --ignore-scripts' - -- task: PowerShell@2 - displayName: 'Extract FLC from pipeline-built artifact' - inputs: - targetType: inline - script: | - $os = 'win32' - $arch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') { 'arm64' } else { 'x64' } - $platformKey = "$os-$arch" - $rid = if ($arch -eq 'arm64') { 'win-arm64' } else { 'win-x64' } - - if ($IsLinux) { - $os = 'linux' - $platformKey = "$os-$arch" - $rid = "linux-$arch" - } elseif ($IsMacOS) { - $os = 'darwin' - $platformKey = "$os-$arch" - $rid = "osx-$arch" - } - - $nupkg = Get-ChildItem "${{ parameters.flcNugetDir }}" -Recurse -Filter "Microsoft.AI.Foundry.Local.Core*.nupkg" -Exclude "*.snupkg" | Select-Object -First 1 - if (-not $nupkg) { throw "No FLC .nupkg found in ${{ parameters.flcNugetDir }}" } - - $extractDir = "$(Build.ArtifactStagingDirectory)/flc-extract" - $zip = [System.IO.Path]::ChangeExtension($nupkg.FullName, ".zip") - Copy-Item $nupkg.FullName $zip -Force - Expand-Archive -Path $zip -DestinationPath $extractDir -Force - - # Place FLC binary so the install script skips downloading it - $destDir = "$(repoRoot)/sdk/js/foundry-local-core/$platformKey" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - $nativeDir = "$extractDir/runtimes/$rid/native" - if (Test-Path $nativeDir) { - Get-ChildItem $nativeDir -File | ForEach-Object { - Copy-Item $_.FullName -Destination "$destDir/$($_.Name)" -Force - Write-Host "Placed $($_.Name) from pipeline artifact" - } - } else { - Write-Warning "No native binaries found at $nativeDir for RID $rid" - } - -- task: PowerShell@2 - displayName: 'Run native binary install (ORT + GenAI)' - inputs: - targetType: inline - script: | - Set-Location "$(repoRoot)/sdk/js" - node script/preinstall.cjs - node script/install-standard.cjs - -# Overwrite the FLC native binary with the one we just built -- task: PowerShell@2 - displayName: 'Overwrite FLC with pipeline-built binary' - condition: and(succeeded(), ne('${{ parameters.flcNugetDir }}', '')) - inputs: - targetType: inline - script: | - $os = 'win32' - $arch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') { 'arm64' } else { 'x64' } - $platformKey = "$os-$arch" - $rid = if ($arch -eq 'arm64') { 'win-arm64' } else { 'win-x64' } - - # Detect macOS/Linux - if ($IsLinux) { - $os = 'linux' - $platformKey = "$os-$arch" - $rid = "linux-$arch" - } elseif ($IsMacOS) { - $os = 'darwin' - $platformKey = "$os-$arch" - $rid = "osx-$arch" - } - - $nupkg = Get-ChildItem "${{ parameters.flcNugetDir }}" -Recurse -Filter "Microsoft.AI.Foundry.Local.Core*.nupkg" -Exclude "*.snupkg" | Select-Object -First 1 - if (-not $nupkg) { throw "No FLC .nupkg found in ${{ parameters.flcNugetDir }}" } - - # Extract the NuGet package (it's a zip) - $extractDir = "$(Build.ArtifactStagingDirectory)/flc-extract" - $zip = [System.IO.Path]::ChangeExtension($nupkg.FullName, ".zip") - Copy-Item $nupkg.FullName $zip -Force - Expand-Archive -Path $zip -DestinationPath $extractDir -Force - - # Overwrite FLC binary in the npm-installed location - $destDir = "$(repoRoot)/sdk/js/foundry-local-core/$platformKey" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - $nativeDir = "$extractDir/runtimes/$rid/native" - if (Test-Path $nativeDir) { - Get-ChildItem $nativeDir -File | ForEach-Object { - Copy-Item $_.FullName -Destination "$destDir/$($_.Name)" -Force - Write-Host "Overwrote $($_.Name) with pipeline-built version" - } - } else { - Write-Warning "No native binaries found at $nativeDir for RID $rid" - } - - Write-Host "Final binaries in $destDir`:" - Get-ChildItem $destDir | ForEach-Object { Write-Host " $($_.Name)" } - -- task: Npm@1 - displayName: 'npm version' - inputs: - command: custom - workingDir: $(repoRoot)/sdk/js - customCommand: 'version $(packageVersion) --no-git-tag-version --allow-same-version' - -- task: Npm@1 - displayName: 'npm build' - inputs: - command: custom - workingDir: $(repoRoot)/sdk/js - customCommand: 'run build' - -- ${{ if eq(parameters.isWinML, true) }}: - - task: Npm@1 - displayName: 'npm run pack:winml' - inputs: - command: custom - workingDir: $(repoRoot)/sdk/js - customCommand: 'run pack:winml' - -- ${{ else }}: - - task: Npm@1 - displayName: 'npm run pack' - inputs: - command: custom - workingDir: $(repoRoot)/sdk/js - customCommand: 'run pack' - -- task: PowerShell@2 - displayName: 'Stage artifact' - inputs: - targetType: inline - script: | - $destDir = "$(Build.ArtifactStagingDirectory)/js-sdk" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Copy-Item "$(repoRoot)/sdk/js/*.tgz" "$destDir/" - diff --git a/.pipelines/v1/templates/build-python-steps.yml b/.pipelines/v1/templates/build-python-steps.yml deleted file mode 100644 index 6309782cc..000000000 --- a/.pipelines/v1/templates/build-python-steps.yml +++ /dev/null @@ -1,227 +0,0 @@ -# Steps to build and pack the Python SDK wheel. -# When test-data-shared is checked out alongside self, ADO places repos under -# $(Build.SourcesDirectory)/. The self repo is 'Foundry-Local'. -parameters: -- name: version - type: string -- name: isRelease - type: boolean - default: false -- name: isWinML - type: boolean - default: false -- name: flcWheelsDir - type: string - displayName: 'Path to directory containing the FLC wheels (for overriding foundry-local-core)' -- name: outputDir - type: string - default: '$(Build.ArtifactStagingDirectory)/python-sdk' - displayName: 'Path to directory for the built wheel' -- name: prereleaseId - type: string - default: '' -- name: depsVersionsDir - type: string - default: '' - displayName: 'Path to deps-versions artifact directory' -steps: -# Set paths for multi-repo checkout -- task: PowerShell@2 - displayName: 'Set source paths' - inputs: - targetType: inline - script: | - $multiCheckout = "$(Build.SourcesDirectory)/Foundry-Local" - if (Test-Path $multiCheckout) { - $repoRoot = $multiCheckout - } else { - $repoRoot = "$(Build.SourcesDirectory)" - } - $testDataDir = "$(Build.SourcesDirectory)/test-data-shared" - Write-Host "##vso[task.setvariable variable=repoRoot]$repoRoot" - Write-Host "##vso[task.setvariable variable=testDataDir]$testDataDir" - -- task: PowerShell@2 - displayName: 'Detect agent architecture' - inputs: - targetType: inline - script: | - # Hosted Linux arm64 image has no python-versions toolcache; skip UsePythonVersion there. - # macOS arm64 has a working toolcache so UsePythonVersion is fine. - $isArm64 = [System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64' - $isLinuxArm64 = $IsLinux -and $isArm64 - Write-Host "##vso[task.setvariable variable=IsLinuxArm64Agent]$($isLinuxArm64.ToString().ToLower())" - -- task: UsePythonVersion@0 - displayName: 'Use Python 3.12' - condition: and(succeeded(), ne(variables['IsLinuxArm64Agent'], 'true')) - inputs: - versionSpec: '3.12' - -- task: PowerShell@2 - displayName: 'Configure system Python for user-mode installs (Linux arm64)' - condition: and(succeeded(), eq(variables['IsLinuxArm64Agent'], 'true')) - inputs: - targetType: inline - script: | - # System python on hosted Linux arm64 cannot write to /usr/lib/python3.12/site-packages. - # The image also only exposes `python3` (no `python` alias), so drop a wrapper - # symlink into the user-bin directory for scripts that invoke `python` directly. - python3 --version - $userBase = & python3 -c "import site; print(site.USER_BASE)" - $userBin = Join-Path $userBase 'bin' - New-Item -ItemType Directory -Force -Path $userBin | Out-Null - $python3Path = (& which python3).Trim() - & ln -sf $python3Path (Join-Path $userBin 'python') - Write-Host "User base: $userBase" - Write-Host "Created symlink: $userBin/python -> $python3Path" - Write-Host "##vso[task.setvariable variable=PIP_USER]1" - Write-Host "##vso[task.prependpath]$userBin" - -# Load dependency versions from deps_versions.json -- template: update-deps-versions-steps.yml - parameters: - repoRoot: $(repoRoot) - artifactDir: ${{ parameters.depsVersionsDir }} - isWinML: ${{ parameters.isWinML }} - -# List downloaded FLC wheels for debugging -- task: PowerShell@2 - displayName: 'List downloaded FLC wheels' - condition: and(succeeded(), ne('${{ parameters.flcWheelsDir }}', '')) - inputs: - targetType: inline - script: | - Write-Host "Contents of ${{ parameters.flcWheelsDir }}:" - Get-ChildItem "${{ parameters.flcWheelsDir }}" -Recurse | ForEach-Object { Write-Host $_.FullName } - -# Read version from the version-info artifact produced by compute_version stage. -# Legacy SDKs read the `.v1.txt` flavor (sdk_v2 reads `.txt`). -- task: PowerShell@2 - displayName: 'Set package version' - inputs: - targetType: inline - script: | - $v = (Get-Content "$(Pipeline.Workspace)/version-info/pyVersion.v1.txt" -Raw).Trim() - Write-Host "Package version: $v" - Write-Host "##vso[task.setvariable variable=packageVersion]$v" - -# Configure pip to use AIFoundryLocal_PublicPackages feed (plus PyPI as fallback) -- task: PipAuthenticate@1 - displayName: 'Authenticate pip with Azure Artifacts' - inputs: - artifactFeeds: 'AIFoundryLocal/AIFoundryLocal_PublicPackages' - -- task: PowerShell@2 - displayName: 'Configure pip index URLs' - inputs: - targetType: inline - script: | - pip config set global.index-url https://pkgs.dev.azure.com/aiinfra/AIFoundryLocal/_packaging/AIFoundryLocal_PublicPackages/pypi/simple/ - pip config set global.extra-index-url https://pypi.org/simple/ - pip config set global.pre true - -# Install the build tool -- script: python -m pip install build - displayName: 'Install build tool' - -# Write version file -- task: PowerShell@2 - displayName: 'Set SDK version' - inputs: - targetType: inline - script: | - Set-Content -Path "$(repoRoot)/sdk/python/src/version.py" -Value '__version__ = "$(packageVersion)"' - -# Install the FLC wheels from the pipeline if provided, so the build -# backend picks up the freshly-built foundry-local-core instead of -# pulling a stale one from the feed. -- task: PowerShell@2 - displayName: 'Pre-install pipeline-built FLC wheel' - condition: and(succeeded(), ne('${{ parameters.flcWheelsDir }}', '')) - inputs: - targetType: inline - script: | - # Determine platform wheel tag for the current machine - $arch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') { 'arm64' } else { 'amd64' } - if ($IsLinux) { - $linuxArch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') { 'aarch64' } else { 'x86_64' } - $platTag = "manylinux*$linuxArch" - } - elseif ($IsMacOS) { $platTag = "macosx*$arch" } - else { $platTag = "win_$arch" } - - $filter = if ("${{ parameters.isWinML }}" -eq "True") { "foundry_local_core_winml*$platTag.whl" } else { "foundry_local_core-*$platTag.whl" } - $wheel = Get-ChildItem "${{ parameters.flcWheelsDir }}" -Recurse -Filter $filter | Select-Object -First 1 - if ($wheel) { - Write-Host "Installing pipeline-built FLC wheel: $($wheel.FullName)" - pip install $($wheel.FullName) - } else { - Write-Warning "No FLC wheel found matching $filter in ${{ parameters.flcWheelsDir }}" - } - -- task: PowerShell@2 - displayName: 'Install ORT native packages' - inputs: - targetType: inline - script: | - $isWinML = "${{ parameters.isWinML }}" -eq "True" - $fileName = if ($isWinML) { "deps_versions_winml.json" } else { "deps_versions.json" } - $deps = Get-Content "$(repoRoot)/sdk/$fileName" -Raw | ConvertFrom-Json - $ortVer = $deps.onnxruntime.version - $genaiVer = $deps.'onnxruntime-genai'.version - # Platform-specific ORT package selection: - # - Linux x86_64: onnxruntime-gpu / onnxruntime-genai-cuda (CUDA wheels). - # - Linux aarch64: onnxruntime / onnxruntime-genai (CUDA variants don't ship aarch64). - # - Other: onnxruntime-core / onnxruntime-genai-core (cross-platform). - $isArm64 = [System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64' - if ($IsLinux -and -not $isArm64) { - Write-Host "Installing onnxruntime-gpu==$ortVer onnxruntime-genai-cuda==$genaiVer (Linux x64)" - pip install "onnxruntime-gpu==$ortVer" "onnxruntime-genai-cuda==$genaiVer" - } elseif ($IsLinux -and $isArm64) { - Write-Host "Installing onnxruntime==$ortVer onnxruntime-genai==$genaiVer (Linux arm64)" - pip install "onnxruntime==$ortVer" "onnxruntime-genai==$genaiVer" - } else { - Write-Host "Installing onnxruntime-core==$ortVer onnxruntime-genai-core==$genaiVer" - pip install "onnxruntime-core==$ortVer" "onnxruntime-genai-core==$genaiVer" - } - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -- script: pip install "pydantic>=2.0.0" "requests>=2.32.4" "openai>=2.24.0" - displayName: 'Install pure python dependencies' - -# Build wheel — standard or WinML variant -# The wheel retains all dependencies in its metadata so end users get -# native packages installed automatically. CI uses --no-deps to avoid -# re-downloading packages that were pre-installed from pipeline builds. -- ${{ if eq(parameters.isWinML, true) }}: - - script: python -m build --wheel -C winml=true --outdir dist/ - displayName: 'Build wheel (WinML)' - workingDirectory: $(repoRoot)/sdk/python - -- ${{ else }}: - - script: python -m build --wheel --outdir dist/ - displayName: 'Build wheel' - workingDirectory: $(repoRoot)/sdk/python - -# Install the built wheel -- task: PowerShell@2 - displayName: 'Install built wheel' - inputs: - targetType: inline - script: | - $wheel = (Get-ChildItem "$(repoRoot)/sdk/python/dist/*.whl" | Select-Object -First 1).FullName - pip install --no-deps $wheel - -# Stage output -- task: PowerShell@2 - displayName: 'Stage wheel artifact' - inputs: - targetType: inline - script: | - $destDir = "${{ parameters.outputDir }}" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Copy-Item "$(repoRoot)/sdk/python/dist/*" "$destDir/" - Write-Host "Staged wheels:" - Get-ChildItem $destDir | ForEach-Object { Write-Host " $($_.Name)" } - diff --git a/.pipelines/v1/templates/build-rust-steps.yml b/.pipelines/v1/templates/build-rust-steps.yml deleted file mode 100644 index 8402b9aac..000000000 --- a/.pipelines/v1/templates/build-rust-steps.yml +++ /dev/null @@ -1,221 +0,0 @@ -# Steps to build and package the Rust SDK crate. -# When test-data-shared is checked out alongside self, ADO places repos under -# $(Build.SourcesDirectory)/. The self repo is 'Foundry-Local'. -parameters: -- name: version - type: string -- name: isRelease - type: boolean - default: false -- name: prereleaseId - type: string - default: '' -- name: isWinML - type: boolean - default: false -- name: flcNugetDir - type: string - displayName: 'Path to directory containing the FLC .nupkg' -- name: outputDir - type: string - default: '$(Build.ArtifactStagingDirectory)/rust-sdk' - displayName: 'Path to directory for the packaged crate' -- name: depsVersionsDir - type: string - default: '' - displayName: 'Path to deps-versions artifact directory' -- name: depsVersionsWinmlDir - type: string - default: '' - displayName: 'Path to WinML deps-versions artifact directory' -steps: -# Set paths for multi-repo checkout -- task: PowerShell@2 - displayName: 'Set source paths' - inputs: - targetType: inline - script: | - $multiCheckout = "$(Build.SourcesDirectory)/Foundry-Local" - if (Test-Path $multiCheckout) { - $repoRoot = $multiCheckout - } else { - $repoRoot = "$(Build.SourcesDirectory)" - } - $testDataDir = "$(Build.SourcesDirectory)/test-data-shared" - Write-Host "##vso[task.setvariable variable=repoRoot]$repoRoot" - Write-Host "##vso[task.setvariable variable=testDataDir]$testDataDir" - -# Read version from the version-info artifact produced by compute_version stage. -# Legacy SDKs read the `.v1.txt` flavor (sdk_v2 reads `.txt`). -- task: PowerShell@2 - displayName: 'Set crate version' - inputs: - targetType: inline - script: | - $v = (Get-Content "$(Pipeline.Workspace)/version-info/sdkVersion.v1.txt" -Raw).Trim() - Write-Host "Crate version: $v" - - # Patch Cargo.toml version field - $cargoPath = "$(repoRoot)/sdk/rust/Cargo.toml" - $content = Get-Content $cargoPath -Raw - $content = $content -replace '(?m)^version\s*=\s*"[^"]+"', "version = `"$v`"" - Set-Content -Path $cargoPath -Value $content - Write-Host "Patched Cargo.toml with version $v" - -# List downloaded FLC artifact for debugging -- task: PowerShell@2 - displayName: 'List downloaded FLC artifact' - inputs: - targetType: inline - script: | - Write-Host "Contents of ${{ parameters.flcNugetDir }}:" - Get-ChildItem "${{ parameters.flcNugetDir }}" -Recurse | ForEach-Object { Write-Host $_.FullName } - -# Load dependency versions from deps_versions.json -- template: update-deps-versions-steps.yml - parameters: - repoRoot: $(repoRoot) - artifactDir: ${{ parameters.depsVersionsDir }} - isWinML: ${{ parameters.isWinML }} - -# Rust publishes a single crate that carries both deps_versions.json and -# deps_versions_winml.json, so always refresh WinML deps as part of packaging. -- template: update-deps-versions-steps.yml - parameters: - repoRoot: $(repoRoot) - artifactDir: ${{ parameters.depsVersionsWinmlDir }} - isWinML: true - -# Copy both deps_versions JSON files into the crate directory so cargo -# package includes them and build.rs can find the right one at build time -# since there is only 1 package for both rust artifacts. -- task: PowerShell@2 - displayName: 'Copy deps_versions for crate packaging' - inputs: - targetType: inline - script: | - Copy-Item "$(repoRoot)/sdk/deps_versions.json" "$(repoRoot)/sdk/rust/deps_versions.json" -Force - Copy-Item "$(repoRoot)/sdk/deps_versions_winml.json" "$(repoRoot)/sdk/rust/deps_versions_winml.json" -Force - -# Extract FLC native binaries from the pipeline-built .nupkg so that -# build.rs finds them already present and skips downloading from the feed. -- task: PowerShell@2 - displayName: 'Extract FLC native binaries for Rust build' - inputs: - targetType: inline - script: | - $nupkg = Get-ChildItem "${{ parameters.flcNugetDir }}" -Recurse -Filter "Microsoft.AI.Foundry.Local.Core*.nupkg" -Exclude "*.snupkg" | Select-Object -First 1 - if (-not $nupkg) { throw "No FLC .nupkg found in ${{ parameters.flcNugetDir }}" } - Write-Host "Found NuGet package: $($nupkg.FullName)" - - $extractDir = "$(Build.ArtifactStagingDirectory)/flc-extract-rust" - $zip = [System.IO.Path]::ChangeExtension($nupkg.FullName, ".zip") - Copy-Item $nupkg.FullName $zip -Force - Expand-Archive -Path $zip -DestinationPath $extractDir -Force - - # Determine RID for this agent - $arch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') { 'arm64' } else { 'x64' } - if ($IsLinux) { - $rid = "linux-$arch" - } elseif ($IsMacOS) { - $rid = "osx-$arch" - } else { - $rid = "win-$arch" - } - - $nativeDir = "$extractDir/runtimes/$rid/native" - if (-not (Test-Path $nativeDir)) { throw "No native binaries found at $nativeDir for RID $rid" } - - # Stage them where build.rs can discover them - $flcNativeDir = "$(Build.ArtifactStagingDirectory)/flc-native-rust" - New-Item -ItemType Directory -Path $flcNativeDir -Force | Out-Null - Get-ChildItem $nativeDir -File | Copy-Item -Destination $flcNativeDir -Force - # Set FOUNDRY_NATIVE_OVERRIDE_DIR so build.rs copies these into OUT_DIR - # instead of trying to download the unpublished FLC Core from the feed. - Write-Host "##vso[task.setvariable variable=FOUNDRY_NATIVE_OVERRIDE_DIR]$flcNativeDir" - Write-Host "Extracted FLC native binaries to $flcNativeDir`:" - Get-ChildItem $flcNativeDir | ForEach-Object { Write-Host " $($_.Name)" } - -# Install Rust toolchain -- task: PowerShell@2 - displayName: 'Install Rust toolchain' - inputs: - targetType: inline - script: | - if ($IsWindows -or (-not $IsLinux -and -not $IsMacOS)) { - Invoke-WebRequest -Uri https://win.rustup.rs/x86_64 -OutFile rustup-init.exe - .\rustup-init.exe -y --default-toolchain stable --profile minimal -c clippy,rustfmt - Remove-Item rustup-init.exe - $cargoPath = "$env:USERPROFILE\.cargo\bin" - } else { - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable --profile minimal -c clippy,rustfmt - $cargoPath = "$env:HOME/.cargo/bin" - } - Write-Host "##vso[task.prependpath]$cargoPath" - -# The .cargo/config.toml redirects crates-io to an Azure Artifacts feed -# for CFS compliance. Remove the redirect in CI so cargo can fetch from -# crates.io directly without Azure DevOps auth. -- task: PowerShell@2 - displayName: 'Use crates.io directly' - inputs: - targetType: inline - script: | - $configPath = "$(repoRoot)/sdk/rust/.cargo/config.toml" - if (Test-Path $configPath) { - Remove-Item $configPath - Write-Host "Removed .cargo/config.toml crates-io redirect" - } - -- task: PowerShell@2 - displayName: 'Check formatting' - inputs: - targetType: inline - script: | - Set-Location "$(repoRoot)/sdk/rust" - cargo fmt --all -- --check - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -- task: PowerShell@2 - displayName: 'Run clippy' - inputs: - targetType: inline - script: | - Set-Location "$(repoRoot)/sdk/rust" - $features = if ("${{ parameters.isWinML }}" -eq "True") { "--features winml" } else { "" } - Invoke-Expression "cargo clippy --all-targets $features -- -D warnings" - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -- task: PowerShell@2 - displayName: 'Build' - inputs: - targetType: inline - script: | - Set-Location "$(repoRoot)/sdk/rust" - $features = if ("${{ parameters.isWinML }}" -eq "True") { "--features winml" } else { "" } - Invoke-Expression "cargo build $features" - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -# --allow-dirty allows packaging with uncommitted changes (build.rs modifies generated files) -- task: PowerShell@2 - displayName: 'Package crate' - inputs: - targetType: inline - script: | - Set-Location "$(repoRoot)/sdk/rust" - $features = if ("${{ parameters.isWinML }}" -eq "True") { "--features winml" } else { "" } - Invoke-Expression "cargo package $features --allow-dirty" - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -# Stage output -- task: PowerShell@2 - displayName: 'Stage crate artifact' - inputs: - targetType: inline - script: | - $destDir = "${{ parameters.outputDir }}" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Copy-Item "$(repoRoot)/sdk/rust/target/package/*.crate" "$destDir/" - Write-Host "Staged crates:" - Get-ChildItem $destDir | ForEach-Object { Write-Host " $($_.Name)" } - diff --git a/.pipelines/v1/templates/package-core-steps.yml b/.pipelines/v1/templates/package-core-steps.yml deleted file mode 100644 index fb9edc514..000000000 --- a/.pipelines/v1/templates/package-core-steps.yml +++ /dev/null @@ -1,345 +0,0 @@ -# Steps to collect per-platform FLC native binaries, organize into NuGet layout, -# pack + sign the NuGet package, and build Python wheels (wheel package name and -# platforms depend on the isWinML parameter). The parent job must download all -# platform artifacts and checkout neutron-server. -parameters: -- name: version - type: string -- name: isRelease - type: boolean - default: false -- name: isWinML - type: boolean - default: false -- name: prereleaseId - type: string - default: '' -- name: platforms - type: object # list of { name, artifactName } - -steps: -- task: PowerShell@2 - displayName: 'Set source paths' - inputs: - targetType: inline - script: | - $nsRoot = "$(Build.SourcesDirectory)/neutron-server" - Write-Host "##vso[task.setvariable variable=nsRoot]$nsRoot" - -- task: PowerShell@2 - displayName: 'Organize native binaries' - inputs: - targetType: inline - script: | - $unifiedPath = "$(Build.ArtifactStagingDirectory)/unified" - New-Item -ItemType Directory -Path $unifiedPath -Force | Out-Null - - $platformsJson = @' - ${{ convertToJson(parameters.platforms) }} - '@ - $platforms = $platformsJson | ConvertFrom-Json - - $isWinML = "${{ parameters.isWinML }}" -eq "True" - - foreach ($p in $platforms) { - $srcDir = "$(Pipeline.Workspace)/$($p.artifactName)" - Write-Host "Looking for artifacts at: $srcDir" - if (-not (Test-Path $srcDir)) { - throw "Artifact directory $srcDir does not exist. All platform artifacts must be present to produce a complete NuGet package." - } - $destDir = "$unifiedPath/runtimes/$($p.name)/native" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Get-ChildItem $srcDir -File | Where-Object { $_.Name -like "Microsoft.AI.Foundry.Local.Core.*" } | - Copy-Item -Destination $destDir -Force - Write-Host "Copied $($p.name) binaries to $destDir" - } - - $nsRoot = "$(nsRoot)" - - if ($isWinML) { - [xml]$propsXml = Get-Content "$nsRoot/Directory.Packages.props" - $winMLVer = [string]$propsXml.Project.PropertyGroup.WinMLVersion - if ([string]::IsNullOrWhiteSpace($winMLVer)) { - throw "Directory.Packages.props is missing WinMLVersion." - } - - $runtimePackageDir = "$(Build.ArtifactStagingDirectory)/winml-runtime-package" - New-Item -ItemType Directory -Path $runtimePackageDir -Force | Out-Null - - $nugetArgs = @( - 'install', 'Microsoft.Windows.AI.MachineLearning', - '-Version', $winMLVer, - '-Source', 'https://api.nuget.org/v3/index.json', - '-OutputDirectory', $runtimePackageDir, - '-ExcludeVersion', - '-NonInteractive', - '-DirectDownload' - ) - Write-Host "Running: nuget $($nugetArgs -join ' ')" - & nuget $nugetArgs - if ($LASTEXITCODE -ne 0) { throw "Failed to download Microsoft.Windows.AI.MachineLearning $winMLVer" } - - $runtimePackageRoot = Get-ChildItem $runtimePackageDir -Directory | - Where-Object { $_.Name -like "Microsoft.Windows.AI.MachineLearning*" } | - Select-Object -First 1 - if (-not $runtimePackageRoot) { - throw "nuget install did not produce a Microsoft.Windows.AI.MachineLearning package directory in $runtimePackageDir" - } - - foreach ($p in $platforms) { - if (-not $p.name.StartsWith("win-")) { - continue - } - - $runtimeDll = @( - "$($runtimePackageRoot.FullName)/runtimes/$($p.name)/Microsoft.Windows.AI.MachineLearning.dll", - "$($runtimePackageRoot.FullName)/runtimes/$($p.name)/native/Microsoft.Windows.AI.MachineLearning.dll" - ) | Where-Object { Test-Path $_ } | Select-Object -First 1 - if ([string]::IsNullOrWhiteSpace($runtimeDll)) { - throw "Microsoft.Windows.AI.MachineLearning $winMLVer does not contain a Microsoft.Windows.AI.MachineLearning.dll for $($p.name)" - } - - $destDir = "$unifiedPath/runtimes/$($p.name)/native" - Copy-Item $runtimeDll -Destination $destDir -Force - Write-Host "Copied WinML runtime DLL for $($p.name) to $destDir" - } - } - - # Copy build integration files from neutron-server - foreach ($dir in @("build", "buildTransitive")) { - $src = "$nsRoot/src/FoundryLocalCore/Core/$dir" - if (Test-Path $src) { - Copy-Item -Path $src -Destination "$unifiedPath/$dir" -Recurse -Force - } - } - $license = "$nsRoot/src/FoundryLocalCore/Core/LICENSE.txt" - if (Test-Path $license) { - Copy-Item $license "$unifiedPath/LICENSE.txt" -Force - } - -# Read version from the version-info artifact produced by compute_version stage. -# Legacy core reads the `.v1.txt` flavor (sdk_v2 reads `.txt`). -- task: PowerShell@2 - displayName: 'Set FLC package version' - inputs: - targetType: inline - script: | - $v = (Get-Content "$(Pipeline.Workspace)/version-info/flcVersion.v1.txt" -Raw).Trim() - Write-Host "FLC version: $v" - Write-Host "##vso[task.setvariable variable=flcVersion]$v" - -# Pack NuGet -- task: PowerShell@2 - displayName: 'Pack FLC NuGet' - inputs: - targetType: inline - script: | - $nsRoot = "$(nsRoot)" - [xml]$propsXml = Get-Content "$nsRoot/Directory.Packages.props" - $pg = $propsXml.Project.PropertyGroup - - $outDir = "$(Build.ArtifactStagingDirectory)/flc-nuget" - New-Item -ItemType Directory -Path $outDir -Force | Out-Null - - if ("${{ parameters.isWinML }}" -eq "True") { - $nuspec = "$nsRoot/src/FoundryLocalCore/Core/WinMLNuget.nuspec" - $id = "Microsoft.AI.Foundry.Local.Core.WinML" - $ortVer = [string]$pg.OnnxRuntimeFoundryVersionForWinML - $genaiVer = [string]$pg.OnnxRuntimeGenAIFoundryVersion - $winMLVer = [string]$pg.WinMLVersion - if ([string]::IsNullOrWhiteSpace($ortVer)) { throw "Directory.Packages.props is missing OnnxRuntimeFoundryVersionForWinML." } - if ([string]::IsNullOrWhiteSpace($genaiVer)) { throw "Directory.Packages.props is missing OnnxRuntimeGenAIFoundryVersion." } - if ([string]::IsNullOrWhiteSpace($winMLVer)) { throw "Directory.Packages.props is missing WinMLVersion." } - - $props = "id=$id;version=$(flcVersion);commitId=$(Build.SourceVersion);OnnxRuntimeFoundryVersionForWinML=$ortVer;OnnxRuntimeGenAIFoundryVersion=$genaiVer;WinMLVersion=$winMLVer" - } else { - $nuspec = "$nsRoot/src/FoundryLocalCore/Core/NativeNuget.nuspec" - $id = "Microsoft.AI.Foundry.Local.Core" - $ortVer = $pg.OnnxRuntimeFoundryVersion - $genaiVer = $pg.OnnxRuntimeGenAIFoundryVersion - $props = "id=$id;version=$(flcVersion);commitId=$(Build.SourceVersion);OnnxRuntimeFoundryVersion=$ortVer;OnnxRuntimeGenAIFoundryVersion=$genaiVer" - } - - $nugetArgs = @( - 'pack', $nuspec, - '-OutputDirectory', $outDir, - '-BasePath', "$(Build.ArtifactStagingDirectory)/unified", - '-Properties', $props, - '-Symbols', '-SymbolPackageFormat', 'snupkg' - ) - Write-Host "Running: nuget $($nugetArgs -join ' ')" - & nuget $nugetArgs - if ($LASTEXITCODE -ne 0) { throw "NuGet pack failed" } - -# Sign NuGet package -- task: SFP.build-tasks.custom-build-task-1.EsrpCodeSigning@5 - displayName: 'Sign FLC NuGet package' - inputs: - ConnectedServiceName: 'OnnxrunTimeCodeSign_20240611' - UseMSIAuthentication: true - AppRegistrationClientId: '$(esrpClientId)' - AppRegistrationTenantId: '$(esrpTenantId)' - EsrpClientId: '$(esrpClientId)' - AuthAKVName: '$(esrpAkvName)' - AuthSignCertName: '$(esrpSignCertName)' - FolderPath: '$(Build.ArtifactStagingDirectory)/flc-nuget' - Pattern: '*.nupkg' - SessionTimeout: 90 - ServiceEndpointUrl: 'https://api.esrp.microsoft.com/api/v2' - MaxConcurrency: 25 - signConfigType: inlineSignParams - inlineOperation: | - [{"keyCode":"CP-401405","operationSetCode":"NuGetSign","parameters":[],"toolName":"sign","toolVersion":"6.2.9304.0"},{"keyCode":"CP-401405","operationSetCode":"NuGetVerify","parameters":[],"toolName":"sign","toolVersion":"6.2.9304.0"}] - -# Build Python wheels from the NuGet package -- task: PowerShell@2 - displayName: 'Build foundry_local_core Python Wheels' - inputs: - targetType: inline - script: | - $stagingDir = "$(Build.ArtifactStagingDirectory)/flc-wheels" - New-Item -ItemType Directory -Path $stagingDir -Force | Out-Null - - $isWinML = "${{ parameters.isWinML }}" -eq "True" - - # Find and extract the NuGet package (.nupkg is a zip archive) - $nupkgFilter = if ($isWinML) { "Microsoft.AI.Foundry.Local.Core.WinML*.nupkg" } else { "Microsoft.AI.Foundry.Local.Core*.nupkg" } - $nupkg = Get-ChildItem "$(Build.ArtifactStagingDirectory)/flc-nuget" -Filter $nupkgFilter | Where-Object { $_.Name -notlike "*.snupkg" } | Select-Object -First 1 - if (-not $nupkg) { throw "No FLC .nupkg found matching $nupkgFilter" } - Write-Host "Found NuGet package: $($nupkg.Name)" - - $extractDir = "$(Build.ArtifactStagingDirectory)/flc-extracted" - $nupkgZip = [System.IO.Path]::ChangeExtension($nupkg.FullName, ".zip") - Copy-Item -Path $nupkg.FullName -Destination $nupkgZip -Force - Expand-Archive -Path $nupkgZip -DestinationPath $extractDir -Force - - # Convert NuGet version to PEP 440 - # NuGet: 0.9.0-dev-202603271723-bb400310 → PEP 440: 0.9.0.dev202603271723 - # The commit hash is dropped because .devN requires N to be a pure integer. - $nupkgVersion = $nupkg.BaseName -replace '^Microsoft\.AI\.Foundry\.Local\.Core(\.WinML)?\.', '' - $parts = $nupkgVersion -split '-' - $pyVersion = if ($parts.Count -ge 3 -and $parts[1] -eq 'dev') { "$($parts[0]).dev$($parts[2])" } - elseif ($parts.Count -eq 2) { "$($parts[0])$($parts[1])" } - else { $parts[0] } - Write-Host "Python package version: $pyVersion" - - $packageName = if ($isWinML) { "foundry_local_core_winml" } else { "foundry_local_core" } - - if ($isWinML) { - $platforms = @( - @{rid="win-x64"; pyKey="bin"; tag="win_amd64"}, - @{rid="win-arm64"; pyKey="bin"; tag="win_arm64"} - ) - } else { - $platforms = @( - @{rid="win-x64"; pyKey="bin"; tag="win_amd64"}, - @{rid="win-arm64"; pyKey="bin"; tag="win_arm64"}, - @{rid="linux-x64"; pyKey="bin"; tag="manylinux_2_28_x86_64"}, - @{rid="linux-arm64"; pyKey="bin"; tag="manylinux_2_28_aarch64"}, - @{rid="osx-arm64"; pyKey="bin"; tag="macosx_11_0_arm64"} - ) - } - - foreach ($p in $platforms) { - $nativeSrc = "$extractDir/runtimes/$($p.rid)/native" - if (-not (Test-Path $nativeSrc)) { - Write-Warning "No native binaries found for $($p.rid) — skipping." - continue - } - - $wheelRoot = "$(Build.ArtifactStagingDirectory)/wheels-build/flc_wheel_$($p.tag)" - $pkgDir = "$wheelRoot/$packageName" - New-Item -ItemType Directory -Path "$pkgDir/$($p.pyKey)" -Force | Out-Null - "" | Set-Content -Encoding ascii "$pkgDir/__init__.py" - Get-ChildItem $nativeSrc -File | Copy-Item -Destination "$pkgDir/$($p.pyKey)" - - $normalizedName = $packageName.Replace('_', '-') - $wheelTag = "py3-none-$($p.tag)" - $distInfoName = "$packageName-$pyVersion" - $wheelName = "$distInfoName-$wheelTag.whl" - $distInfoDir = "$wheelRoot/$distInfoName.dist-info" - New-Item -ItemType Directory -Path $distInfoDir -Force | Out-Null - - $utf8NoBom = [System.Text.UTF8Encoding]::new($false) - - [System.IO.File]::WriteAllText("$distInfoDir/WHEEL", - "Wheel-Version: 1.0`nGenerator: custom`nRoot-Is-Purelib: false`nTag: $wheelTag`n", $utf8NoBom) - - [System.IO.File]::WriteAllText("$distInfoDir/METADATA", - "Metadata-Version: 2.1`nName: $normalizedName`nVersion: $pyVersion`n", $utf8NoBom) - - $recordLines = Get-ChildItem $wheelRoot -Recurse -File | ForEach-Object { - $rel = $_.FullName.Substring($wheelRoot.Length + 1).Replace('\', '/') - $raw = (Get-FileHash $_.FullName -Algorithm SHA256).Hash - $bytes = [byte[]]::new($raw.Length / 2) - for ($i = 0; $i -lt $raw.Length; $i += 2) { $bytes[$i/2] = [Convert]::ToByte($raw.Substring($i, 2), 16) } - $b64 = [Convert]::ToBase64String($bytes) -replace '\+','-' -replace '/','_' -replace '=','' - "$rel,sha256=$b64,$($_.Length)" - } - $recordContent = ($recordLines + "$distInfoName.dist-info/RECORD,,") -join "`n" - [System.IO.File]::WriteAllText("$distInfoDir/RECORD", $recordContent, $utf8NoBom) - - $wheelPath = "$stagingDir/$wheelName" - Add-Type -AssemblyName System.IO.Compression.FileSystem - $zip = [System.IO.Compression.ZipFile]::Open($wheelPath, 'Create') - try { - Get-ChildItem $wheelRoot -Recurse -File | ForEach-Object { - $rel = $_.FullName.Substring($wheelRoot.Length + 1).Replace('\', '/') - [System.IO.Compression.ZipFileExtensions]::CreateEntryFromFile($zip, $_.FullName, $rel) | Out-Null - } - } finally { - $zip.Dispose() - } - Write-Host "Created wheel: $wheelName" - } - - Write-Host "`nAll wheels:" - Get-ChildItem $stagingDir -Filter "*.whl" | ForEach-Object { Write-Host " $($_.Name)" } - -# Write partial deps_versions.json for this variant. The merge_deps_versions -# stage combines standard + WinML partials into a single complete artifact. -- task: PowerShell@2 - displayName: 'Write deps_versions.json artifact' - inputs: - targetType: inline - script: | - $nsRoot = "$(nsRoot)" - [xml]$propsXml = Get-Content "$nsRoot/Directory.Packages.props" - $pg = $propsXml.Project.PropertyGroup - - $isWinML = "${{ parameters.isWinML }}" -eq "True" - - # Compute PEP 440 version from the NuGet flcVersion - $parts = "$(flcVersion)" -split '-' - $pyVer = if ($parts.Count -ge 3 -and $parts[1] -eq 'dev') { "$($parts[0]).dev$($parts[2])" } - elseif ($parts.Count -eq 2) { "$($parts[0])$($parts[1])" } - else { $parts[0] } - - # The pipeline produces separate dependency version artifacts - # (deps-versions-standard / deps-versions-winml), so SDK stages pick the - # right one via isWinML. - if ($isWinML) { - $deps = @{ - 'foundry-local-core' = @{ nuget = "$(flcVersion)"; python = $pyVer } - 'windows-ai-machinelearning' = @{ version = [string]$pg.WinMLVersion } - onnxruntime = @{ version = [string]$pg.OnnxRuntimeFoundryVersionForWinML } - 'onnxruntime-genai' = @{ version = [string]$pg.OnnxRuntimeGenAIFoundryVersion } - } - } else { - $deps = @{ - 'foundry-local-core' = @{ nuget = "$(flcVersion)"; python = $pyVer } - onnxruntime = @{ version = [string]$pg.OnnxRuntimeFoundryVersion } - 'onnxruntime-genai' = @{ version = [string]$pg.OnnxRuntimeGenAIFoundryVersion } - } - } - - # WinML artifact is named deps_versions_winml.json to match repo convention. - $fileName = if ($isWinML) { "deps_versions_winml.json" } else { "deps_versions.json" } - $json = $deps | ConvertTo-Json -Depth 3 - Write-Host "${fileName}:" - Write-Host $json - - $outDir = "$(Build.ArtifactStagingDirectory)/deps-versions" - New-Item -ItemType Directory -Path $outDir -Force | Out-Null - [System.IO.File]::WriteAllText("$outDir/$fileName", $json, [System.Text.UTF8Encoding]::new($false)) - Write-Host "Wrote $fileName to $outDir" diff --git a/.pipelines/v1/templates/stages-sdk-v1.yml b/.pipelines/v1/templates/stages-sdk-v1.yml deleted file mode 100644 index 7bbd1f368..000000000 --- a/.pipelines/v1/templates/stages-sdk-v1.yml +++ /dev/null @@ -1,1523 +0,0 @@ -# sdk_v1 coordinator: emits the full legacy (sdk/) stage graph. -# -# Produces NuGet/wheel/JS artifacts for Foundry Local Core (FLC) plus the -# v1 C# / JS / Python / Rust SDKs in both base and WinML flavors. -# -# Assumes the caller has already emitted: -# * a `compute_version` stage publishing the `version-info` pipeline -# artifact (containing sdkVersion.v1.txt, pyVersion.v1.txt, flcVersion.v1.txt) -# * any stage referenced by `gateDependsOn` (typically `detect_changes`) -# -# The entire subgraph is gated by a single artificial root stage (`v1_gate`) -# whose runtime condition is supplied via `gateCondition`. When v1_gate is -# skipped, every downstream v1 stage skips via ADO's default `succeeded()` -# cascade on dependsOn. See stages-sdk-v2.yml for the rationale behind this -# pattern. - -parameters: -- name: version - type: string -- name: prereleaseId - type: string - default: 'none' -- name: isRelease - type: boolean - default: false -- name: neutronServerBranch - type: string -- name: gateCondition - type: string - default: 'succeeded()' -- name: gateDependsOn - type: object - default: [] - -stages: - -# -- Artificial root: gates the entire v1 subgraph -- -- stage: v1_gate - displayName: 'sdk_v1 build gate' - condition: ${{ parameters.gateCondition }} - dependsOn: ${{ parameters.gateDependsOn }} - jobs: - - job: noop - displayName: 'sdk_v1 build enabled' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - steps: - - checkout: none - - script: echo "sdk_v1 build enabled" - displayName: 'sdk_v1 build enabled' - -- stage: build_core - displayName: 'Build Core' - dependsOn: - - v1_gate - - compute_version - jobs: - - job: flc_win_x64 - displayName: 'win-x64' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'flc-win-x64' - targetPath: '$(Build.ArtifactStagingDirectory)/native' - steps: - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: neutron-server - ref: refs/heads/${{ parameters.neutronServerBranch }} - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: build-core-steps.yml - parameters: - flavor: win-x64 - platform: x64 - - - job: flc_win_arm64 - displayName: 'win-arm64' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'flc-win-arm64' - targetPath: '$(Build.ArtifactStagingDirectory)/native' - steps: - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: neutron-server - ref: refs/heads/${{ parameters.neutronServerBranch }} - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: build-core-steps.yml - parameters: - flavor: win-arm64 - platform: arm64 - - - job: flc_linux_x64 - displayName: 'linux-x64' - pool: - name: onnxruntime-Ubuntu2404-AMD-CPU - os: linux - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'flc-linux-x64' - targetPath: '$(Build.ArtifactStagingDirectory)/native' - steps: - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: neutron-server - ref: refs/heads/${{ parameters.neutronServerBranch }} - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: build-core-steps.yml - parameters: - flavor: linux-x64 - platform: x64 - - - job: flc_linux_arm64 - displayName: 'linux-arm64' - pool: - name: onnxruntime-linux-ARM64-CPU-2019 - os: linux - hostArchitecture: arm64 - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'flc-linux-arm64' - targetPath: '$(Build.ArtifactStagingDirectory)/native' - steps: - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: neutron-server - ref: refs/heads/${{ parameters.neutronServerBranch }} - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: build-core-steps.yml - parameters: - flavor: linux-arm64 - platform: arm64 - - - job: flc_osx_arm64 - displayName: 'osx-arm64' - pool: - name: AcesShared - os: macOS - demands: - - ImageOverride -equals ACES_VM_SharedPool_Sequoia - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'flc-osx-arm64' - targetPath: '$(Build.ArtifactStagingDirectory)/native' - steps: - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: neutron-server - ref: refs/heads/${{ parameters.neutronServerBranch }} - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - # AcesShared macOS agents don't have git-lfs pre-installed - - script: brew install git-lfs && git lfs install - displayName: 'Install Git LFS' - - template: build-core-steps.yml - parameters: - flavor: osx-arm64 - platform: arm64 - - - job: package_flc - displayName: 'Package Core' - dependsOn: - - flc_win_x64 - - flc_win_arm64 - - flc_linux_x64 - - flc_linux_arm64 - - flc_osx_arm64 - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'version-info' - targetPath: '$(Pipeline.Workspace)/version-info' - outputs: - - output: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Build.ArtifactStagingDirectory)/flc-nuget' - - output: pipelineArtifact - artifactName: 'flc-wheels' - targetPath: '$(Build.ArtifactStagingDirectory)/flc-wheels' - - output: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Build.ArtifactStagingDirectory)/deps-versions' - steps: - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: neutron-server - ref: refs/heads/${{ parameters.neutronServerBranch }} - - task: DownloadPipelineArtifact@2 - inputs: - buildType: current - artifactName: 'flc-win-x64' - targetPath: '$(Pipeline.Workspace)/flc-win-x64' - - task: DownloadPipelineArtifact@2 - inputs: - buildType: current - artifactName: 'flc-win-arm64' - targetPath: '$(Pipeline.Workspace)/flc-win-arm64' - - task: DownloadPipelineArtifact@2 - inputs: - buildType: current - artifactName: 'flc-linux-x64' - targetPath: '$(Pipeline.Workspace)/flc-linux-x64' - - task: DownloadPipelineArtifact@2 - inputs: - buildType: current - artifactName: 'flc-linux-arm64' - targetPath: '$(Pipeline.Workspace)/flc-linux-arm64' - - task: DownloadPipelineArtifact@2 - inputs: - buildType: current - artifactName: 'flc-osx-arm64' - targetPath: '$(Pipeline.Workspace)/flc-osx-arm64' - - task: PowerShell@2 - displayName: 'List downloaded platform artifacts' - inputs: - targetType: inline - script: | - foreach ($name in @('flc-win-x64','flc-win-arm64','flc-linux-x64','flc-linux-arm64','flc-osx-arm64')) { - $dir = "$(Pipeline.Workspace)/$name" - Write-Host "Contents of ${dir}:" - if (Test-Path $dir) { Get-ChildItem $dir -Recurse | ForEach-Object { Write-Host $_.FullName } } - else { Write-Host " (directory not found)" } - } - - template: package-core-steps.yml - parameters: - version: ${{ parameters.version }} - isRelease: ${{ parameters.isRelease }} - prereleaseId: ${{ parameters.prereleaseId }} - isWinML: false - platforms: - - name: win-x64 - artifactName: flc-win-x64 - - name: win-arm64 - artifactName: flc-win-arm64 - - name: linux-x64 - artifactName: flc-linux-x64 - - name: linux-arm64 - artifactName: flc-linux-arm64 - - name: osx-arm64 - artifactName: flc-osx-arm64 - -# ── Build C# SDK ── -- stage: build_cs - displayName: 'Build C# SDK' - dependsOn: - - v1_gate - - build_core - jobs: - - job: cs_sdk - displayName: 'Build' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'version-info' - targetPath: '$(Pipeline.Workspace)/version-info' - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - outputs: - - output: pipelineArtifact - artifactName: 'cs-sdk' - targetPath: '$(Build.ArtifactStagingDirectory)/cs-sdk' - steps: - - checkout: self - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: build-cs-steps.yml - parameters: - version: ${{ parameters.version }} - isRelease: ${{ parameters.isRelease }} - prereleaseId: ${{ parameters.prereleaseId }} - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - -# ── Build JS Node-API Addon (all platforms) ── -- stage: build_js_addon - displayName: 'Build JS Addon' - dependsOn: [v1_gate] - jobs: - - job: js_addon_win_x64 - displayName: 'Addon win32-x64' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'js-addon-win32-x64' - targetPath: '$(Build.ArtifactStagingDirectory)/js-addon' - steps: - - checkout: self - - template: build-js-addon-steps.yml - parameters: - repoRoot: $(Build.SourcesDirectory) - - task: PowerShell@2 - displayName: 'Stage addon artifact' - inputs: - targetType: inline - script: | - $destDir = "$(Build.ArtifactStagingDirectory)/js-addon" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Copy-Item "$(Build.SourcesDirectory)/sdk/js/prebuilds/win32-x64/foundry_local_napi.node" "$destDir/foundry_local_napi.node" -Force - - - job: js_addon_win_arm64 - displayName: 'Addon win32-arm64' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'js-addon-win32-arm64' - targetPath: '$(Build.ArtifactStagingDirectory)/js-addon' - steps: - - checkout: self - - template: build-js-addon-steps.yml - parameters: - repoRoot: $(Build.SourcesDirectory) - targetArch: arm64 - - task: PowerShell@2 - displayName: 'Stage addon artifact' - inputs: - targetType: inline - script: | - $destDir = "$(Build.ArtifactStagingDirectory)/js-addon" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Copy-Item "$(Build.SourcesDirectory)/sdk/js/prebuilds/win32-arm64/foundry_local_napi.node" "$destDir/foundry_local_napi.node" -Force - - - job: js_addon_linux_x64 - displayName: 'Addon linux-x64' - pool: - name: onnxruntime-Ubuntu2404-AMD-CPU - os: linux - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'js-addon-linux-x64' - targetPath: '$(Build.ArtifactStagingDirectory)/js-addon' - steps: - - checkout: self - - template: build-js-addon-steps.yml - parameters: - repoRoot: $(Build.SourcesDirectory) - - task: PowerShell@2 - displayName: 'Stage addon artifact' - inputs: - targetType: inline - script: | - $destDir = "$(Build.ArtifactStagingDirectory)/js-addon" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Copy-Item "$(Build.SourcesDirectory)/sdk/js/prebuilds/linux-x64/foundry_local_napi.node" "$destDir/foundry_local_napi.node" -Force - - - job: js_addon_linux_arm64 - displayName: 'Addon linux-arm64' - pool: - name: onnxruntime-linux-ARM64-CPU-2019 - os: linux - hostArchitecture: arm64 - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'js-addon-linux-arm64' - targetPath: '$(Build.ArtifactStagingDirectory)/js-addon' - steps: - - checkout: self - - template: build-js-addon-steps.yml - parameters: - repoRoot: $(Build.SourcesDirectory) - - task: PowerShell@2 - displayName: 'Stage addon artifact' - inputs: - targetType: inline - script: | - $destDir = "$(Build.ArtifactStagingDirectory)/js-addon" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Copy-Item "$(Build.SourcesDirectory)/sdk/js/prebuilds/linux-arm64/foundry_local_napi.node" "$destDir/foundry_local_napi.node" -Force - - - job: js_addon_osx_arm64 - displayName: 'Addon osx-arm64' - pool: - name: Azure Pipelines - vmImage: 'macOS-15' - os: macOS - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'js-addon-darwin-arm64' - targetPath: '$(Build.ArtifactStagingDirectory)/js-addon' - steps: - - checkout: self - - template: build-js-addon-steps.yml - parameters: - repoRoot: $(Build.SourcesDirectory) - targetArch: arm64 - - task: PowerShell@2 - displayName: 'Stage addon artifact' - inputs: - targetType: inline - script: | - $destDir = "$(Build.ArtifactStagingDirectory)/js-addon" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Copy-Item "$(Build.SourcesDirectory)/sdk/js/prebuilds/darwin-arm64/foundry_local_napi.node" "$destDir/foundry_local_napi.node" -Force - -# ── Build JS SDK ── -- stage: build_js - displayName: 'Build JS SDK' - dependsOn: - - v1_gate - - build_core - - build_js_addon - jobs: - - job: js_sdk - displayName: 'Build' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'version-info' - targetPath: '$(Pipeline.Workspace)/version-info' - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - outputs: - - output: pipelineArtifact - artifactName: 'js-sdk' - targetPath: '$(Build.ArtifactStagingDirectory)/js-sdk' - steps: - - checkout: self - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - # Download prebuilt Node-API addons for all platforms - - task: DownloadPipelineArtifact@2 - displayName: 'Download addon (win32-x64)' - inputs: - buildType: current - artifactName: 'js-addon-win32-x64' - targetPath: '$(Pipeline.Workspace)/js-addon-win32-x64' - - task: DownloadPipelineArtifact@2 - displayName: 'Download addon (win32-arm64)' - inputs: - buildType: current - artifactName: 'js-addon-win32-arm64' - targetPath: '$(Pipeline.Workspace)/js-addon-win32-arm64' - - task: DownloadPipelineArtifact@2 - displayName: 'Download addon (linux-x64)' - inputs: - buildType: current - artifactName: 'js-addon-linux-x64' - targetPath: '$(Pipeline.Workspace)/js-addon-linux-x64' - - task: DownloadPipelineArtifact@2 - displayName: 'Download addon (linux-arm64)' - inputs: - buildType: current - artifactName: 'js-addon-linux-arm64' - targetPath: '$(Pipeline.Workspace)/js-addon-linux-arm64' - - task: DownloadPipelineArtifact@2 - displayName: 'Download addon (osx-arm64)' - inputs: - buildType: current - artifactName: 'js-addon-darwin-arm64' - targetPath: '$(Pipeline.Workspace)/js-addon-darwin-arm64' - - - task: PowerShell@2 - displayName: 'Place prebuilt addons' - inputs: - targetType: inline - script: | - $multiCheckout = "$(Build.SourcesDirectory)/Foundry-Local" - if (Test-Path $multiCheckout) { $repoRoot = $multiCheckout } else { $repoRoot = "$(Build.SourcesDirectory)" } - $prebuildsDir = "$repoRoot/sdk/js/prebuilds" - foreach ($platform in @('win32-x64','win32-arm64','linux-x64','linux-arm64','darwin-arm64')) { - $src = "$(Pipeline.Workspace)/js-addon-$platform/foundry_local_napi.node" - $destDir = "$prebuildsDir/$platform" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Copy-Item $src "$destDir/foundry_local_napi.node" -Force - Write-Host "Placed addon for $platform ($($(Get-Item $src).Length) bytes)" - } - - - template: build-js-steps.yml - parameters: - version: ${{ parameters.version }} - isRelease: ${{ parameters.isRelease }} - prereleaseId: ${{ parameters.prereleaseId }} - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - -# ── Build Python SDK ── -- stage: build_python - displayName: 'Build Python SDK' - dependsOn: - - v1_gate - - build_core - jobs: - - job: python_sdk - displayName: 'Build' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'version-info' - targetPath: '$(Pipeline.Workspace)/version-info' - - input: pipelineArtifact - artifactName: 'flc-wheels' - targetPath: '$(Pipeline.Workspace)/flc-wheels' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - outputs: - - output: pipelineArtifact - artifactName: 'python-sdk' - targetPath: '$(Build.ArtifactStagingDirectory)/python-sdk' - steps: - - checkout: self - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: build-python-steps.yml - parameters: - version: ${{ parameters.version }} - isRelease: ${{ parameters.isRelease }} - prereleaseId: ${{ parameters.prereleaseId }} - isWinML: false - flcWheelsDir: '$(Pipeline.Workspace)/flc-wheels' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - -# ── Build Rust SDK ── -- stage: build_rust - displayName: 'Build Rust SDK' - dependsOn: - - v1_gate - - build_core - - build_core_winml - jobs: - - job: rust_sdk - displayName: 'Build' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'version-info' - targetPath: '$(Pipeline.Workspace)/version-info' - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - - input: pipelineArtifact - artifactName: 'deps-versions-winml' - targetPath: '$(Pipeline.Workspace)/deps-versions-winml' - outputs: - - output: pipelineArtifact - artifactName: 'rust-sdk' - targetPath: '$(Build.ArtifactStagingDirectory)/rust-sdk' - steps: - - checkout: self - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: build-rust-steps.yml - parameters: - version: ${{ parameters.version }} - isRelease: ${{ parameters.isRelease }} - prereleaseId: ${{ parameters.prereleaseId }} - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - depsVersionsWinmlDir: '$(Pipeline.Workspace)/deps-versions-winml' - -# Rust SDK has one package with different install options for standard vs WinML, -# so we only publish once under the standard stage and skip the WinML stage. Leaving -# it as a commented block incase we decide to publish separate Rust WinML package in the future. -# # ── Build Rust SDK (WinML) ── -# - stage: build_rust_winml -# displayName: 'Build Rust SDK (WinML)' -# dependsOn: -# - build_core_winml -# jobs: -# - job: rust_sdk_winml -# displayName: 'Build' -# pool: -# name: onnxruntime-Win-CPU-2022 -# os: windows -# templateContext: -# inputs: -# - input: pipelineArtifact -# artifactName: 'version-info' -# targetPath: '$(Pipeline.Workspace)/version-info' -# - input: pipelineArtifact -# artifactName: 'flc-nuget-winml' -# targetPath: '$(Pipeline.Workspace)/flc-nuget-winml' -# - input: pipelineArtifact -# artifactName: 'deps-versions-winml' -# targetPath: '$(Pipeline.Workspace)/deps-versions-winml' -# outputs: -# - output: pipelineArtifact -# artifactName: 'rust-sdk-winml' -# targetPath: '$(Build.ArtifactStagingDirectory)/rust-sdk-winml' -# steps: -# - checkout: self -# clean: true -# - checkout: test-data-shared -# lfs: true -# - template: build-rust-steps.yml -# parameters: -# version: ${{ parameters.version }} -# isRelease: ${{ parameters.isRelease }} -# prereleaseId: ${{ parameters.prereleaseId }} -# isWinML: true -# flcNugetDir: '$(Pipeline.Workspace)/flc-nuget-winml' -# depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-winml' -# outputDir: '$(Build.ArtifactStagingDirectory)/rust-sdk-winml' - -# ── Test C# SDK ── -- stage: test_cs - displayName: 'Test C#' - dependsOn: - - v1_gate - - build_cs - jobs: - - job: test_cs_win_x64 - displayName: 'win-x64' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - basePath: '$(Agent.BuildDirectory)' - - template: test-cs-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_cs_linux_x64 - displayName: 'linux-x64' - pool: - name: onnxruntime-Ubuntu2404-AMD-CPU - os: linux - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - basePath: '$(Agent.BuildDirectory)' - - template: test-cs-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_cs_linux_arm64 - displayName: 'linux-arm64' - pool: - name: onnxruntime-linux-ARM64-CPU-2019 - os: linux - hostArchitecture: arm64 - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - basePath: '$(Agent.BuildDirectory)' - - template: test-cs-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_cs_osx_arm64 - displayName: 'osx-arm64' - pool: - name: AcesShared - os: macOS - demands: - - ImageOverride -equals ACES_VM_SharedPool_Sequoia - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - # AcesShared macOS agents don't have git-lfs pre-installed - - script: brew install git-lfs && git lfs install - displayName: 'Install Git LFS' - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - basePath: '$(Agent.BuildDirectory)' - - template: test-cs-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - -# ── Test JS SDK ── -- stage: test_js - displayName: 'Test JS' - dependsOn: - - v1_gate - - build_js - jobs: - - job: test_js_win_x64 - displayName: 'win-x64' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: test-js-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_js_linux_x64 - displayName: 'linux-x64' - pool: - name: onnxruntime-Ubuntu2404-AMD-CPU - os: linux - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - basePath: '$(Agent.BuildDirectory)' - - template: test-js-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_js_linux_arm64 - displayName: 'linux-arm64' - pool: - name: onnxruntime-linux-ARM64-CPU-2019 - os: linux - hostArchitecture: arm64 - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - basePath: '$(Agent.BuildDirectory)' - - template: test-js-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_js_osx_arm64 - displayName: 'osx-arm64' - pool: - name: AcesShared - os: macOS - demands: - - ImageOverride -equals ACES_VM_SharedPool_Sequoia - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - # AcesShared macOS agents don't have git-lfs pre-installed - - script: brew install git-lfs && git lfs install - displayName: 'Install Git LFS' - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: test-js-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - -# ── Test Python SDK ── -- stage: test_python - displayName: 'Test Python' - dependsOn: - - v1_gate - - build_python - jobs: - - job: test_python_win_x64 - displayName: 'win-x64' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-wheels' - targetPath: '$(Pipeline.Workspace)/flc-wheels' - - input: pipelineArtifact - artifactName: 'python-sdk' - targetPath: '$(Pipeline.Workspace)/python-sdk' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: test-python-steps.yml - parameters: - isWinML: false - flcWheelsDir: '$(Pipeline.Workspace)/flc-wheels' - sdkWheelsDir: '$(Pipeline.Workspace)/python-sdk' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_python_linux_x64 - displayName: 'linux-x64' - pool: - name: onnxruntime-Ubuntu2404-AMD-CPU - os: linux - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-wheels' - targetPath: '$(Pipeline.Workspace)/flc-wheels' - - input: pipelineArtifact - artifactName: 'python-sdk' - targetPath: '$(Pipeline.Workspace)/python-sdk' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - basePath: '$(Agent.BuildDirectory)' - - template: test-python-steps.yml - parameters: - isWinML: false - flcWheelsDir: '$(Pipeline.Workspace)/flc-wheels' - sdkWheelsDir: '$(Pipeline.Workspace)/python-sdk' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_python_linux_arm64 - displayName: 'linux-arm64' - pool: - name: onnxruntime-linux-ARM64-CPU-2019 - os: linux - hostArchitecture: arm64 - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-wheels' - targetPath: '$(Pipeline.Workspace)/flc-wheels' - - input: pipelineArtifact - artifactName: 'python-sdk' - targetPath: '$(Pipeline.Workspace)/python-sdk' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - basePath: '$(Agent.BuildDirectory)' - - template: test-python-steps.yml - parameters: - isWinML: false - flcWheelsDir: '$(Pipeline.Workspace)/flc-wheels' - sdkWheelsDir: '$(Pipeline.Workspace)/python-sdk' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_python_osx_arm64 - displayName: 'osx-arm64' - pool: - name: AcesShared - os: macOS - demands: - - ImageOverride -equals ACES_VM_SharedPool_Sequoia - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-wheels' - targetPath: '$(Pipeline.Workspace)/flc-wheels' - - input: pipelineArtifact - artifactName: 'python-sdk' - targetPath: '$(Pipeline.Workspace)/python-sdk' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - # AcesShared macOS agents don't have git-lfs pre-installed - - script: brew install git-lfs && git lfs install - displayName: 'Install Git LFS' - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: test-python-steps.yml - parameters: - isWinML: false - flcWheelsDir: '$(Pipeline.Workspace)/flc-wheels' - sdkWheelsDir: '$(Pipeline.Workspace)/python-sdk' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - -# ── Test Rust SDK ── -- stage: test_rust - displayName: 'Test Rust' - dependsOn: - - v1_gate - - build_rust - jobs: - - job: test_rust_win_x64 - displayName: 'win-x64' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: test-rust-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_rust_linux_x64 - displayName: 'linux-x64' - pool: - name: onnxruntime-Ubuntu2404-AMD-CPU - os: linux - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: test-rust-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_rust_linux_arm64 - displayName: 'linux-arm64' - pool: - name: onnxruntime-linux-ARM64-CPU-2019 - os: linux - hostArchitecture: arm64 - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: test-rust-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - - - job: test_rust_osx_arm64 - displayName: 'osx-arm64' - pool: - name: AcesShared - os: macOS - demands: - - ImageOverride -equals ACES_VM_SharedPool_Sequoia - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget' - targetPath: '$(Pipeline.Workspace)/flc-nuget' - - input: pipelineArtifact - artifactName: 'deps-versions-standard' - targetPath: '$(Pipeline.Workspace)/deps-versions-standard' - steps: - # AcesShared macOS agents don't have git-lfs pre-installed - - script: brew install git-lfs && git lfs install - displayName: 'Install Git LFS' - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: test-rust-steps.yml - parameters: - isWinML: false - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-standard' - -# ── Build FLC (WinML) ── -- stage: build_core_winml - displayName: 'Build Core (WinML)' - dependsOn: - - v1_gate - - compute_version - jobs: - - job: flc_winml_win_x64 - displayName: 'win-x64 (WinML)' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'flc-winml-win-x64' - targetPath: '$(Build.ArtifactStagingDirectory)/native' - steps: - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: neutron-server - ref: refs/heads/${{ parameters.neutronServerBranch }} - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: build-core-steps.yml - parameters: - flavor: win-x64 - platform: x64 - isWinML: true - - - job: flc_winml_win_arm64 - displayName: 'win-arm64 (WinML)' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - outputs: - - output: pipelineArtifact - artifactName: 'flc-winml-win-arm64' - targetPath: '$(Build.ArtifactStagingDirectory)/native' - steps: - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: neutron-server - ref: refs/heads/${{ parameters.neutronServerBranch }} - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: build-core-steps.yml - parameters: - flavor: win-arm64 - platform: arm64 - isWinML: true - - - job: package_flc_winml - displayName: 'Package Core (WinML)' - dependsOn: - - flc_winml_win_x64 - - flc_winml_win_arm64 - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'version-info' - targetPath: '$(Pipeline.Workspace)/version-info' - outputs: - - output: pipelineArtifact - artifactName: 'flc-nuget-winml' - targetPath: '$(Build.ArtifactStagingDirectory)/flc-nuget' - - output: pipelineArtifact - artifactName: 'flc-wheels-winml' - targetPath: '$(Build.ArtifactStagingDirectory)/flc-wheels' - - output: pipelineArtifact - artifactName: 'deps-versions-winml' - targetPath: '$(Build.ArtifactStagingDirectory)/deps-versions' - steps: - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: neutron-server - ref: refs/heads/${{ parameters.neutronServerBranch }} - - task: DownloadPipelineArtifact@2 - inputs: - buildType: current - artifactName: 'flc-winml-win-x64' - targetPath: '$(Pipeline.Workspace)/flc-winml-win-x64' - - task: DownloadPipelineArtifact@2 - inputs: - buildType: current - artifactName: 'flc-winml-win-arm64' - targetPath: '$(Pipeline.Workspace)/flc-winml-win-arm64' - - task: PowerShell@2 - displayName: 'List downloaded WinML platform artifacts' - inputs: - targetType: inline - script: | - foreach ($name in @('flc-winml-win-x64','flc-winml-win-arm64')) { - $dir = "$(Pipeline.Workspace)/$name" - Write-Host "Contents of ${dir}:" - if (Test-Path $dir) { Get-ChildItem $dir -Recurse | ForEach-Object { Write-Host $_.FullName } } - else { Write-Host " (directory not found)" } - } - - template: package-core-steps.yml - parameters: - version: ${{ parameters.version }} - isRelease: ${{ parameters.isRelease }} - prereleaseId: ${{ parameters.prereleaseId }} - isWinML: true - platforms: - - name: win-x64 - artifactName: flc-winml-win-x64 - - name: win-arm64 - artifactName: flc-winml-win-arm64 - -# ── Build C# SDK (WinML) ── -- stage: build_cs_winml - displayName: 'Build C# SDK (WinML)' - dependsOn: - - v1_gate - - build_core_winml - jobs: - - job: cs_sdk_winml - displayName: 'Build' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'version-info' - targetPath: '$(Pipeline.Workspace)/version-info' - - input: pipelineArtifact - artifactName: 'flc-nuget-winml' - targetPath: '$(Pipeline.Workspace)/flc-nuget-winml' - - input: pipelineArtifact - artifactName: 'deps-versions-winml' - targetPath: '$(Pipeline.Workspace)/deps-versions-winml' - outputs: - - output: pipelineArtifact - artifactName: 'cs-sdk-winml' - targetPath: '$(Build.ArtifactStagingDirectory)/cs-sdk-winml' - steps: - - checkout: self - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: build-cs-steps.yml - parameters: - version: ${{ parameters.version }} - isRelease: ${{ parameters.isRelease }} - prereleaseId: ${{ parameters.prereleaseId }} - isWinML: true - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget-winml' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-winml' - outputDir: '$(Build.ArtifactStagingDirectory)/cs-sdk-winml' - -# ── Build JS SDK (WinML) ── -- stage: build_js_winml - displayName: 'Build JS SDK (WinML)' - dependsOn: - - v1_gate - - build_core_winml - - build_js_addon - jobs: - - job: js_sdk_winml - displayName: 'Build' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'version-info' - targetPath: '$(Pipeline.Workspace)/version-info' - - input: pipelineArtifact - artifactName: 'flc-nuget-winml' - targetPath: '$(Pipeline.Workspace)/flc-nuget-winml' - - input: pipelineArtifact - artifactName: 'deps-versions-winml' - targetPath: '$(Pipeline.Workspace)/deps-versions-winml' - outputs: - - output: pipelineArtifact - artifactName: 'js-sdk-winml' - targetPath: '$(Build.ArtifactStagingDirectory)/js-sdk' - steps: - - checkout: self - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - # Download prebuilt Node-API addons for all platforms - - task: DownloadPipelineArtifact@2 - displayName: 'Download addon (win32-x64)' - inputs: - buildType: current - artifactName: 'js-addon-win32-x64' - targetPath: '$(Pipeline.Workspace)/js-addon-win32-x64' - - task: DownloadPipelineArtifact@2 - displayName: 'Download addon (win32-arm64)' - inputs: - buildType: current - artifactName: 'js-addon-win32-arm64' - targetPath: '$(Pipeline.Workspace)/js-addon-win32-arm64' - - task: DownloadPipelineArtifact@2 - displayName: 'Download addon (linux-x64)' - inputs: - buildType: current - artifactName: 'js-addon-linux-x64' - targetPath: '$(Pipeline.Workspace)/js-addon-linux-x64' - - task: DownloadPipelineArtifact@2 - displayName: 'Download addon (linux-arm64)' - inputs: - buildType: current - artifactName: 'js-addon-linux-arm64' - targetPath: '$(Pipeline.Workspace)/js-addon-linux-arm64' - - task: DownloadPipelineArtifact@2 - displayName: 'Download addon (osx-arm64)' - inputs: - buildType: current - artifactName: 'js-addon-darwin-arm64' - targetPath: '$(Pipeline.Workspace)/js-addon-darwin-arm64' - - - task: PowerShell@2 - displayName: 'Place prebuilt addons' - inputs: - targetType: inline - script: | - $multiCheckout = "$(Build.SourcesDirectory)/Foundry-Local" - if (Test-Path $multiCheckout) { $repoRoot = $multiCheckout } else { $repoRoot = "$(Build.SourcesDirectory)" } - $prebuildsDir = "$repoRoot/sdk/js/prebuilds" - foreach ($platform in @('win32-x64','win32-arm64','linux-x64','linux-arm64','darwin-arm64')) { - $src = "$(Pipeline.Workspace)/js-addon-$platform/foundry_local_napi.node" - $destDir = "$prebuildsDir/$platform" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - Copy-Item $src "$destDir/foundry_local_napi.node" -Force - Write-Host "Placed addon for $platform ($($(Get-Item $src).Length) bytes)" - } - - - template: build-js-steps.yml - parameters: - version: ${{ parameters.version }} - isRelease: ${{ parameters.isRelease }} - prereleaseId: ${{ parameters.prereleaseId }} - isWinML: true - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget-winml' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-winml' - -# ── Build Python SDK (WinML) ── -- stage: build_python_winml - displayName: 'Build Python SDK (WinML)' - dependsOn: - - v1_gate - - build_core_winml - jobs: - - job: python_sdk_winml - displayName: 'Build' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'version-info' - targetPath: '$(Pipeline.Workspace)/version-info' - - input: pipelineArtifact - artifactName: 'flc-wheels-winml' - targetPath: '$(Pipeline.Workspace)/flc-wheels-winml' - - input: pipelineArtifact - artifactName: 'deps-versions-winml' - targetPath: '$(Pipeline.Workspace)/deps-versions-winml' - outputs: - - output: pipelineArtifact - artifactName: 'python-sdk-winml' - targetPath: '$(Build.ArtifactStagingDirectory)/python-sdk-winml' - steps: - - checkout: self - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: build-python-steps.yml - parameters: - version: ${{ parameters.version }} - isRelease: ${{ parameters.isRelease }} - prereleaseId: ${{ parameters.prereleaseId }} - isWinML: true - flcWheelsDir: '$(Pipeline.Workspace)/flc-wheels-winml' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-winml' - outputDir: '$(Build.ArtifactStagingDirectory)/python-sdk-winml' - -# ── Test C# SDK (WinML) ── -- stage: test_cs_winml - displayName: 'Test C# (WinML)' - dependsOn: - - v1_gate - - build_cs_winml - jobs: - - job: test_cs_winml_win_x64 - displayName: 'win-x64' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget-winml' - targetPath: '$(Pipeline.Workspace)/flc-nuget-winml' - - input: pipelineArtifact - artifactName: 'deps-versions-winml' - targetPath: '$(Pipeline.Workspace)/deps-versions-winml' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - basePath: '$(Agent.BuildDirectory)' - - template: test-cs-steps.yml - parameters: - isWinML: true - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget-winml' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-winml' - -# ── Test JS SDK (WinML) ── -- stage: test_js_winml - displayName: 'Test JS (WinML)' - dependsOn: - - v1_gate - - build_js_winml - jobs: - - job: test_js_winml_win_x64 - displayName: 'win-x64' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-nuget-winml' - targetPath: '$(Pipeline.Workspace)/flc-nuget-winml' - - input: pipelineArtifact - artifactName: 'deps-versions-winml' - targetPath: '$(Pipeline.Workspace)/deps-versions-winml' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: test-js-steps.yml - parameters: - isWinML: true - flcNugetDir: '$(Pipeline.Workspace)/flc-nuget-winml' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-winml' - -# ── Test Python SDK (WinML) ── -- stage: test_python_winml - displayName: 'Test Python (WinML)' - dependsOn: - - v1_gate - - build_python_winml - jobs: - - job: test_python_winml_win_x64 - displayName: 'win-x64' - pool: - name: onnxruntime-Win-CPU-2022 - os: windows - templateContext: - inputs: - - input: pipelineArtifact - artifactName: 'flc-wheels-winml' - targetPath: '$(Pipeline.Workspace)/flc-wheels-winml' - - input: pipelineArtifact - artifactName: 'python-sdk-winml' - targetPath: '$(Pipeline.Workspace)/python-sdk-winml' - - input: pipelineArtifact - artifactName: 'deps-versions-winml' - targetPath: '$(Pipeline.Workspace)/deps-versions-winml' - steps: - - checkout: self - clean: true - - template: ../../templates/checkout-steps.yml@self - parameters: - repoName: test-data-shared - - template: test-python-steps.yml - parameters: - isWinML: true - flcWheelsDir: '$(Pipeline.Workspace)/flc-wheels-winml' - sdkWheelsDir: '$(Pipeline.Workspace)/python-sdk-winml' - depsVersionsDir: '$(Pipeline.Workspace)/deps-versions-winml' \ No newline at end of file diff --git a/.pipelines/v1/templates/test-cs-steps.yml b/.pipelines/v1/templates/test-cs-steps.yml deleted file mode 100644 index 605b36cf2..000000000 --- a/.pipelines/v1/templates/test-cs-steps.yml +++ /dev/null @@ -1,149 +0,0 @@ -# Steps to test the C# SDK. -# Expects the SDK to be already built and the NuGet package available. -parameters: -- name: isWinML - type: boolean - default: false -- name: flcNugetDir - type: string - displayName: 'Path to directory containing the FLC .nupkg' -- name: depsVersionsDir - type: string - default: '' - displayName: 'Path to deps-versions artifact directory' - -steps: -- task: PowerShell@2 - displayName: 'Set source paths' - inputs: - targetType: inline - script: | - $multiCheckout = "$(Build.SourcesDirectory)/Foundry-Local" - if (Test-Path $multiCheckout) { - $repoRoot = $multiCheckout - } else { - $repoRoot = "$(Build.SourcesDirectory)" - } - $testDataDir = "$(Build.SourcesDirectory)/test-data-shared" - Write-Host "##vso[task.setvariable variable=repoRoot]$repoRoot" - Write-Host "##vso[task.setvariable variable=testDataDir]$testDataDir" - -- task: UseDotNet@2 - displayName: 'Use .NET 9 SDK' - inputs: - packageType: sdk - version: '9.0.x' - -- task: UseDotNet@2 - displayName: 'Install .NET 8 runtime' - inputs: - packageType: runtime - version: '8.0.x' - -# Load dependency versions from deps_versions.json -- template: update-deps-versions-steps.yml - parameters: - repoRoot: $(repoRoot) - artifactDir: ${{ parameters.depsVersionsDir }} - isWinML: ${{ parameters.isWinML }} - -# Create a temporary NuGet.config that includes the local FLC feed -- task: PowerShell@2 - displayName: 'Create NuGet.config with local FLC feed' - inputs: - targetType: inline - script: | - $nugetConfig = @" - - - - - - - - - - "@ - $nupkg = Get-ChildItem "${{ parameters.flcNugetDir }}" -Recurse -Filter "Microsoft.AI.Foundry.Local.Core*.nupkg" -Exclude "*.snupkg" | Select-Object -First 1 - if (-not $nupkg) { throw "No FLC .nupkg found in ${{ parameters.flcNugetDir }}" } - $flcFeedDir = $nupkg.DirectoryName - $nugetConfig = $nugetConfig -replace [regex]::Escape("${{ parameters.flcNugetDir }}"), $flcFeedDir - $configPath = "$(Build.ArtifactStagingDirectory)/NuGet.config" - Set-Content -Path $configPath -Value $nugetConfig - Write-Host "##vso[task.setvariable variable=customNugetConfig]$configPath" - Write-Host "Local FLC feed directory: $flcFeedDir" - -- task: NuGetAuthenticate@1 - displayName: 'Authenticate NuGet feeds' - -- ${{ if eq(parameters.isWinML, true) }}: - - task: PowerShell@2 - displayName: 'Install Windows App SDK Runtime' - inputs: - targetType: 'inline' - script: | - $installerUrl = "https://aka.ms/windowsappsdk/1.8/latest/windowsappruntimeinstall-x64.exe" - $installerPath = "$env:TEMP\windowsappruntimeinstall.exe" - Invoke-WebRequest -Uri $installerUrl -OutFile $installerPath - & $installerPath --quiet --force - if ($LASTEXITCODE -ne 0) { throw "Windows App SDK Runtime install failed" } - errorActionPreference: 'stop' - -# Per-job NuGet isolation to prevent "Central Directory corrupt" / file-locking -# errors when multiple C# test jobs (regular + WinML) run concurrently on the -# same reused agent. Keyed by $(System.JobId); cleaned on each run. -# https://learn.microsoft.com/en-us/nuget/reference/cli-reference/cli-ref-environment-variables -- task: PowerShell@2 - displayName: 'Set isolated NuGet packages path' - inputs: - targetType: inline - script: | - $buildDir = "$(Build.BinariesDirectory)/nuget-isolated-$(System.JobId)" - # Clean any leftover state from previous runs on this agent - if (Test-Path $buildDir) { - Remove-Item -Recurse -Force $buildDir - } - $pkgDir = "$buildDir/packages" - $httpCacheDir = "$buildDir/http-cache" - $pluginsCacheDir = "$buildDir/plugins-cache" - foreach ($d in @($pkgDir, $httpCacheDir, $pluginsCacheDir)) { - New-Item -ItemType Directory -Force -Path $d | Out-Null - } - - Write-Host "##vso[task.setvariable variable=NUGET_PACKAGES]$pkgDir" - Write-Host "##vso[task.setvariable variable=NUGET_HTTP_CACHE_PATH]$httpCacheDir" - Write-Host "##vso[task.setvariable variable=NUGET_PLUGINS_CACHE_PATH]$pluginsCacheDir" - Write-Host "NuGet isolation directory: $buildDir" - -- task: PowerShell@2 - displayName: 'Restore & build tests' - inputs: - targetType: inline - script: | - $rid = dotnet msbuild "$(repoRoot)/sdk/cs/test/FoundryLocal.Tests/Microsoft.AI.Foundry.Local.Tests.csproj" -getProperty:NETCoreSdkRuntimeIdentifier - if ($LASTEXITCODE -ne 0 -or -not $rid) { throw "Failed to determine RuntimeIdentifier" } - Write-Host "Restoring for RuntimeIdentifier: $rid" - - dotnet restore "$(repoRoot)/sdk/cs/test/FoundryLocal.Tests/Microsoft.AI.Foundry.Local.Tests.csproj" ` - --configfile "$(customNugetConfig)" ` - /p:UseWinML=${{ parameters.isWinML }} ` - /p:RuntimeIdentifiers=$rid - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - - dotnet build "$(repoRoot)/sdk/cs/test/FoundryLocal.Tests/Microsoft.AI.Foundry.Local.Tests.csproj" ` - --no-restore --configuration Release ` - /p:UseWinML=${{ parameters.isWinML }} ` - /p:RuntimeIdentifiers=$rid - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -- task: PowerShell@2 - displayName: 'Run SDK tests' - inputs: - targetType: inline - script: | - dotnet test "$(repoRoot)/sdk/cs/test/FoundryLocal.Tests/Microsoft.AI.Foundry.Local.Tests.csproj" ` - --no-build --configuration Release ` - /p:UseWinML=${{ parameters.isWinML }} - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - env: - TF_BUILD: 'true' diff --git a/.pipelines/v1/templates/test-js-steps.yml b/.pipelines/v1/templates/test-js-steps.yml deleted file mode 100644 index 2d007f70b..000000000 --- a/.pipelines/v1/templates/test-js-steps.yml +++ /dev/null @@ -1,156 +0,0 @@ -# Steps to test the JS SDK. -# Expects the SDK to be already built via build-js-steps.yml. -parameters: -- name: isWinML - type: boolean - default: false -- name: flcNugetDir - type: string - default: '' - displayName: 'Path to directory containing the FLC .nupkg' -- name: depsVersionsDir - type: string - default: '' - displayName: 'Path to deps-versions artifact directory' - -steps: -- task: PowerShell@2 - displayName: 'Set source paths' - inputs: - targetType: inline - script: | - $multiCheckout = "$(Build.SourcesDirectory)/Foundry-Local" - if (Test-Path $multiCheckout) { - $repoRoot = $multiCheckout - } else { - $repoRoot = "$(Build.SourcesDirectory)" - } - $testDataDir = "$(Build.SourcesDirectory)/test-data-shared" - Write-Host "##vso[task.setvariable variable=repoRoot]$repoRoot" - Write-Host "##vso[task.setvariable variable=testDataDir]$testDataDir" - -- task: NodeTool@0 - displayName: 'Use Node.js 20' - inputs: - versionSpec: '20.x' - -# Load dependency versions from deps_versions.json -- template: update-deps-versions-steps.yml - parameters: - repoRoot: $(repoRoot) - artifactDir: ${{ parameters.depsVersionsDir }} - isWinML: ${{ parameters.isWinML }} - -# Install JS dependencies with native binaries -- task: Npm@1 - displayName: 'npm install (skip native downloads)' - inputs: - command: custom - workingDir: $(repoRoot)/sdk/js - customCommand: 'install --ignore-scripts' - -# esbuild (transitive dep via tsx) requires a platform-specific binary package. -# --ignore-scripts prevents it from installing automatically, so do it explicitly for Linux. -- task: Npm@1 - displayName: 'Install esbuild platform binary (Linux x64)' - condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'), ne(variables['Agent.OSArchitecture'], 'ARM64')) - inputs: - command: custom - workingDir: $(repoRoot)/sdk/js - customCommand: 'install @esbuild/linux-x64 --no-save' - -- task: Npm@1 - displayName: 'Install esbuild platform binary (Linux arm64)' - condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'), eq(variables['Agent.OSArchitecture'], 'ARM64')) - inputs: - command: custom - workingDir: $(repoRoot)/sdk/js - customCommand: 'install @esbuild/linux-arm64 --no-save' - -- task: PowerShell@2 - displayName: 'Extract FLC from pipeline-built artifact' - condition: and(succeeded(), ne('${{ parameters.flcNugetDir }}', '')) - inputs: - targetType: inline - script: | - $os = 'win32' - $arch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') { 'arm64' } else { 'x64' } - $platformKey = "$os-$arch" - $rid = if ($arch -eq 'arm64') { 'win-arm64' } else { 'win-x64' } - - if ($IsLinux) { - $os = 'linux' - $platformKey = "$os-$arch" - $rid = "linux-$arch" - } elseif ($IsMacOS) { - $os = 'darwin' - $platformKey = "$os-$arch" - $rid = "osx-$arch" - } - - $nupkg = Get-ChildItem "${{ parameters.flcNugetDir }}" -Recurse -Filter "Microsoft.AI.Foundry.Local.Core*.nupkg" -Exclude "*.snupkg" | Select-Object -First 1 - if (-not $nupkg) { throw "No FLC .nupkg found in ${{ parameters.flcNugetDir }}" } - - $extractDir = "$(Build.ArtifactStagingDirectory)/flc-extract" - $zip = [System.IO.Path]::ChangeExtension($nupkg.FullName, ".zip") - Copy-Item $nupkg.FullName $zip -Force - Expand-Archive -Path $zip -DestinationPath $extractDir -Force - - $destDir = "$(repoRoot)/sdk/js/foundry-local-core/$platformKey" - New-Item -ItemType Directory -Path $destDir -Force | Out-Null - $nativeDir = "$extractDir/runtimes/$rid/native" - if (Test-Path $nativeDir) { - Get-ChildItem $nativeDir -File | ForEach-Object { - Copy-Item $_.FullName -Destination "$destDir/$($_.Name)" -Force - Write-Host "Placed $($_.Name) from pipeline artifact" - } - } else { - Write-Warning "No native binaries found at $nativeDir for RID $rid" - } - -- task: PowerShell@2 - displayName: 'Run native binary install (ORT + GenAI)' - inputs: - targetType: inline - script: | - Set-Location "$(repoRoot)/sdk/js" - node script/preinstall.cjs - node script/install-standard.cjs - -# Build the Node-API native addon -- task: Npm@1 - displayName: 'npm build node-api addon' - inputs: - command: custom - workingDir: $(repoRoot)/sdk/js - customCommand: 'run build:native' - -- task: Npm@1 - displayName: 'npm build' - inputs: - command: custom - workingDir: $(repoRoot)/sdk/js - customCommand: 'run build' - -- ${{ if eq(parameters.isWinML, true) }}: - - task: PowerShell@2 - displayName: 'Install Windows App SDK Runtime' - inputs: - targetType: 'inline' - script: | - $installerUrl = "https://aka.ms/windowsappsdk/1.8/latest/windowsappruntimeinstall-x64.exe" - $installerPath = "$env:TEMP\windowsappruntimeinstall.exe" - Invoke-WebRequest -Uri $installerUrl -OutFile $installerPath - & $installerPath --quiet --force - if ($LASTEXITCODE -ne 0) { throw "Windows App SDK Runtime install failed" } - errorActionPreference: 'stop' - -- task: Npm@1 - displayName: 'npm test' - inputs: - command: custom - workingDir: $(repoRoot)/sdk/js - customCommand: 'test' - env: - TF_BUILD: 'true' - FOUNDRY_TEST_DATA_DIR: $(testDataDir) diff --git a/.pipelines/v1/templates/test-python-steps.yml b/.pipelines/v1/templates/test-python-steps.yml deleted file mode 100644 index cf2643982..000000000 --- a/.pipelines/v1/templates/test-python-steps.yml +++ /dev/null @@ -1,180 +0,0 @@ -# Steps to test the Python SDK. -# Expects the SDK wheel to be already built via build-python-steps.yml. -parameters: -- name: isWinML - type: boolean - default: false -- name: flcWheelsDir - type: string - displayName: 'Path to directory containing the FLC wheels' -- name: sdkWheelsDir - type: string - displayName: 'Path to directory containing the built SDK wheel' -- name: depsVersionsDir - type: string - default: '' - displayName: 'Path to deps-versions artifact directory' - -steps: -- task: PowerShell@2 - displayName: 'Set source paths' - inputs: - targetType: inline - script: | - $multiCheckout = "$(Build.SourcesDirectory)/Foundry-Local" - if (Test-Path $multiCheckout) { - $repoRoot = $multiCheckout - } else { - $repoRoot = "$(Build.SourcesDirectory)" - } - $testDataDir = "$(Build.SourcesDirectory)/test-data-shared" - Write-Host "##vso[task.setvariable variable=repoRoot]$repoRoot" - Write-Host "##vso[task.setvariable variable=testDataDir]$testDataDir" - -- task: PowerShell@2 - displayName: 'Detect Python architecture' - inputs: - targetType: inline - script: | - # UsePythonVersion defaults to x64 but arm64 agents only have arm64 Python. - # The hosted *Linux* arm64 image has no python-versions toolcache entry, - # so we skip UsePythonVersion there and rely on the system python. - # macOS arm64 hosted agents DO have a python-versions toolcache and work - # normally with UsePythonVersion (the toolcache provides `pip` on PATH). - $isArm64 = [System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64' - $isLinuxArm64 = $IsLinux -and $isArm64 - $arch = if ($isArm64) { 'ARM64' } else { 'x64' } - Write-Host "##vso[task.setvariable variable=PythonArch]$arch" - Write-Host "##vso[task.setvariable variable=IsLinuxArm64Agent]$($isLinuxArm64.ToString().ToLower())" - Write-Host "Detected Python architecture: $arch; IsLinuxArm64Agent=$isLinuxArm64" - -- task: UsePythonVersion@0 - displayName: 'Use Python 3.12' - condition: and(succeeded(), ne(variables['IsLinuxArm64Agent'], 'true')) - inputs: - versionSpec: '3.12' - architecture: '$(PythonArch)' - -- task: PowerShell@2 - displayName: 'Configure system Python for user-mode installs (Linux arm64)' - condition: and(succeeded(), eq(variables['IsLinuxArm64Agent'], 'true')) - inputs: - targetType: inline - script: | - # System python on hosted Linux arm64 cannot write to /usr/lib/python3.12/site-packages. - # Force all subsequent `pip install` calls to use the per-user site and put - # the user-script directory on PATH so console-scripts (e.g. pytest) are reachable. - # The image also only exposes `python3` (no `python` alias), so drop a wrapper - # symlink into the user-bin directory for scripts that invoke `python` directly. - python3 --version - $userBase = & python3 -c "import site; print(site.USER_BASE)" - $userBin = Join-Path $userBase 'bin' - New-Item -ItemType Directory -Force -Path $userBin | Out-Null - $python3Path = (& which python3).Trim() - & ln -sf $python3Path (Join-Path $userBin 'python') - Write-Host "User base: $userBase" - Write-Host "Created symlink: $userBin/python -> $python3Path" - Write-Host "##vso[task.setvariable variable=PIP_USER]1" - Write-Host "##vso[task.prependpath]$userBin" - -# Load dependency versions from deps_versions.json -- template: update-deps-versions-steps.yml - parameters: - repoRoot: $(repoRoot) - artifactDir: ${{ parameters.depsVersionsDir }} - isWinML: ${{ parameters.isWinML }} - -# Configure pip to use ORT-Nightly feed (plus PyPI as fallback) -- task: PowerShell@2 - displayName: 'Configure pip for Azure Artifacts' - inputs: - targetType: inline - script: | - pip config set global.index-url https://pkgs.dev.azure.com/aiinfra/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ - pip config set global.extra-index-url https://pypi.org/simple/ - pip config set global.pre true - -# Install the FLC wheel from the pipeline -- task: PowerShell@2 - displayName: 'Install pipeline-built FLC wheel' - inputs: - targetType: inline - script: | - $arch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') { 'arm64' } else { 'amd64' } - if ($IsLinux) { - $linuxArch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') { 'aarch64' } else { 'x86_64' } - $platTag = "manylinux*$linuxArch" - } - elseif ($IsMacOS) { $platTag = "macosx*$arch" } - else { $platTag = "win_$arch" } - - $filter = if ("${{ parameters.isWinML }}" -eq "True") { "foundry_local_core_winml*$platTag.whl" } else { "foundry_local_core-*$platTag.whl" } - $wheel = Get-ChildItem "${{ parameters.flcWheelsDir }}" -Recurse -Filter $filter | Select-Object -First 1 - if ($wheel) { - Write-Host "Installing pipeline-built FLC wheel: $($wheel.FullName)" - pip install $($wheel.FullName) - } else { - Write-Warning "No FLC wheel found matching $filter in ${{ parameters.flcWheelsDir }}" - } - -- task: PowerShell@2 - displayName: 'Install ORT native packages' - inputs: - targetType: inline - script: | - $isWinML = "${{ parameters.isWinML }}" -eq "True" - $fileName = if ($isWinML) { "deps_versions_winml.json" } else { "deps_versions.json" } - $deps = Get-Content "$(repoRoot)/sdk/$fileName" -Raw | ConvertFrom-Json - $ortVer = $deps.onnxruntime.version - $genaiVer = $deps.'onnxruntime-genai'.version - # Platform-specific ORT package selection: - # - Linux x86_64: onnxruntime-gpu / onnxruntime-genai-cuda (CUDA wheels). - # - Linux aarch64: onnxruntime / onnxruntime-genai (CUDA variants don't ship aarch64). - # - Other: onnxruntime-core / onnxruntime-genai-core (cross-platform). - $isArm64 = [System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64' - if ($IsLinux -and -not $isArm64) { - Write-Host "Installing onnxruntime-gpu==$ortVer onnxruntime-genai-cuda==$genaiVer (Linux x64)" - pip install "onnxruntime-gpu==$ortVer" "onnxruntime-genai-cuda==$genaiVer" - } elseif ($IsLinux -and $isArm64) { - Write-Host "Installing onnxruntime==$ortVer onnxruntime-genai==$genaiVer (Linux arm64)" - pip install "onnxruntime==$ortVer" "onnxruntime-genai==$genaiVer" - } else { - Write-Host "Installing onnxruntime-core==$ortVer onnxruntime-genai-core==$genaiVer" - pip install "onnxruntime-core==$ortVer" "onnxruntime-genai-core==$genaiVer" - } - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -- script: pip install "pydantic>=2.0.0" "requests>=2.32.4" "openai>=2.24.0" - displayName: 'Install pure python dependencies' - -# Install the built SDK wheel -- task: PowerShell@2 - displayName: 'Install SDK wheel' - inputs: - targetType: inline - script: | - $wheel = (Get-ChildItem "${{ parameters.sdkWheelsDir }}/*.whl" | Select-Object -First 1).FullName - pip install --no-deps $wheel - -- ${{ if eq(parameters.isWinML, true) }}: - - task: PowerShell@2 - displayName: 'Install Windows App SDK Runtime' - inputs: - targetType: 'inline' - script: | - $installerUrl = "https://aka.ms/windowsappsdk/1.8/latest/windowsappruntimeinstall-x64.exe" - $installerPath = "$env:TEMP\windowsappruntimeinstall.exe" - Invoke-WebRequest -Uri $installerUrl -OutFile $installerPath - & $installerPath --quiet --force - if ($LASTEXITCODE -ne 0) { throw "Windows App SDK Runtime install failed" } - errorActionPreference: 'stop' - -- script: pip install coverage pytest>=7.0.0 pytest-timeout>=2.1.0 - displayName: 'Install test dependencies' - -- script: python -m pytest test/ -v - displayName: 'Run tests' - workingDirectory: $(repoRoot)/sdk/python - env: - TF_BUILD: 'true' - FOUNDRY_TEST_DATA_DIR: $(testDataDir) diff --git a/.pipelines/v1/templates/test-rust-steps.yml b/.pipelines/v1/templates/test-rust-steps.yml deleted file mode 100644 index 4d382b1de..000000000 --- a/.pipelines/v1/templates/test-rust-steps.yml +++ /dev/null @@ -1,136 +0,0 @@ -# Steps to test the Rust SDK. -# Expects the SDK to be already built via build-rust-steps.yml. -parameters: -- name: isWinML - type: boolean - default: false -- name: flcNugetDir - type: string - displayName: 'Path to directory containing the FLC .nupkg' -- name: depsVersionsDir - type: string - default: '' - displayName: 'Path to deps-versions artifact directory' - -steps: -- task: PowerShell@2 - displayName: 'Set source paths' - inputs: - targetType: inline - script: | - $multiCheckout = "$(Build.SourcesDirectory)/Foundry-Local" - if (Test-Path $multiCheckout) { - $repoRoot = $multiCheckout - } else { - $repoRoot = "$(Build.SourcesDirectory)" - } - $testDataDir = "$(Build.SourcesDirectory)/test-data-shared" - Write-Host "##vso[task.setvariable variable=repoRoot]$repoRoot" - Write-Host "##vso[task.setvariable variable=testDataDir]$testDataDir" - -# Load dependency versions from deps_versions.json -- template: update-deps-versions-steps.yml - parameters: - repoRoot: $(repoRoot) - artifactDir: ${{ parameters.depsVersionsDir }} - isWinML: ${{ parameters.isWinML }} - -# Extract FLC native binaries from the pipeline-built .nupkg -- task: PowerShell@2 - displayName: 'Extract FLC native binaries' - inputs: - targetType: inline - script: | - $nupkg = Get-ChildItem "${{ parameters.flcNugetDir }}" -Recurse -Filter "Microsoft.AI.Foundry.Local.Core*.nupkg" -Exclude "*.snupkg" | Select-Object -First 1 - if (-not $nupkg) { throw "No FLC .nupkg found in ${{ parameters.flcNugetDir }}" } - - $extractDir = "$(Build.ArtifactStagingDirectory)/flc-extract-rust" - $zip = [System.IO.Path]::ChangeExtension($nupkg.FullName, ".zip") - Copy-Item $nupkg.FullName $zip -Force - Expand-Archive -Path $zip -DestinationPath $extractDir -Force - - $arch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') { 'arm64' } else { 'x64' } - if ($IsLinux) { $rid = "linux-$arch" } - elseif ($IsMacOS) { $rid = "osx-$arch" } - else { $rid = "win-$arch" } - - $nativeDir = "$extractDir/runtimes/$rid/native" - if (-not (Test-Path $nativeDir)) { throw "No native binaries found at $nativeDir for RID $rid" } - - $flcNativeDir = "$(Build.ArtifactStagingDirectory)/flc-native-rust" - New-Item -ItemType Directory -Path $flcNativeDir -Force | Out-Null - Get-ChildItem $nativeDir -File | Copy-Item -Destination $flcNativeDir -Force - Write-Host "##vso[task.setvariable variable=FOUNDRY_NATIVE_OVERRIDE_DIR]$flcNativeDir" - Write-Host "Extracted FLC native binaries to $flcNativeDir" - -# Copy deps_versions files -- task: PowerShell@2 - displayName: 'Copy deps_versions for Rust' - inputs: - targetType: inline - script: | - Copy-Item "$(repoRoot)/sdk/deps_versions.json" "$(repoRoot)/sdk/rust/deps_versions.json" -Force - Copy-Item "$(repoRoot)/sdk/deps_versions_winml.json" "$(repoRoot)/sdk/rust/deps_versions_winml.json" -Force - -# Install Rust toolchain -- task: PowerShell@2 - displayName: 'Install Rust toolchain' - inputs: - targetType: inline - script: | - if ($IsWindows -or (-not $IsLinux -and -not $IsMacOS)) { - Invoke-WebRequest -Uri https://win.rustup.rs/x86_64 -OutFile rustup-init.exe - .\rustup-init.exe -y --default-toolchain stable --profile minimal -c clippy,rustfmt - Remove-Item rustup-init.exe - $cargoPath = "$env:USERPROFILE\.cargo\bin" - } else { - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable --profile minimal -c clippy,rustfmt - $cargoPath = "$env:HOME/.cargo/bin" - } - Write-Host "##vso[task.prependpath]$cargoPath" - -# Remove .cargo/config.toml crates-io redirect -- task: PowerShell@2 - displayName: 'Use crates.io directly' - inputs: - targetType: inline - script: | - $configPath = "$(repoRoot)/sdk/rust/.cargo/config.toml" - if (Test-Path $configPath) { - Remove-Item $configPath - Write-Host "Removed .cargo/config.toml crates-io redirect" - } - -# Build before testing -- task: PowerShell@2 - displayName: 'Build' - inputs: - targetType: inline - script: | - Set-Location "$(repoRoot)/sdk/rust" - $features = if ("${{ parameters.isWinML }}" -eq "True") { "--features winml" } else { "" } - Invoke-Expression "cargo build $features" - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -- task: PowerShell@2 - displayName: 'Run unit tests' - inputs: - targetType: inline - script: | - Set-Location "$(repoRoot)/sdk/rust" - $features = if ("${{ parameters.isWinML }}" -eq "True") { "--features winml" } else { "" } - Invoke-Expression "cargo test --lib $features" - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - -- task: PowerShell@2 - displayName: 'Run integration tests' - inputs: - targetType: inline - script: | - Set-Location "$(repoRoot)/sdk/rust" - $features = if ("${{ parameters.isWinML }}" -eq "True") { "--features winml" } else { "" } - Invoke-Expression "cargo test --tests $features -- --include-ignored --test-threads=1 --nocapture" - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - env: - TF_BUILD: 'true' - FOUNDRY_TEST_DATA_DIR: $(testDataDir) diff --git a/.pipelines/v1/templates/update-deps-versions-steps.yml b/.pipelines/v1/templates/update-deps-versions-steps.yml deleted file mode 100644 index 6f0ebcc34..000000000 --- a/.pipelines/v1/templates/update-deps-versions-steps.yml +++ /dev/null @@ -1,44 +0,0 @@ -# Shared template to update deps_versions.json / deps_versions_winml.json -# from pipeline artifacts. The isWinML parameter determines which file gets -# overwritten. -parameters: -- name: repoRoot - type: string - default: '$(repoRoot)' -- name: artifactDir - type: string - default: '' - displayName: 'Path to artifact directory containing pipeline-generated deps_versions JSON' -- name: isWinML - type: boolean - default: false - -steps: -- task: PowerShell@2 - displayName: 'Update deps_versions from pipeline artifact' - inputs: - targetType: inline - script: | - $isWinML = "${{ parameters.isWinML }}" -eq "True" - $fileName = if ($isWinML) { "deps_versions_winml.json" } else { "deps_versions.json" } - $repoJson = "${{ parameters.repoRoot }}/sdk/$fileName" - $artifactDir = "${{ parameters.artifactDir }}" - - if ($artifactDir -eq '' -or -not (Test-Path "$artifactDir/$fileName")) { - throw "Pipeline-built $fileName not found in artifact directory: $artifactDir" - } - - Copy-Item "$artifactDir/$fileName" $repoJson -Force - Write-Host "Updated repo $fileName from pipeline artifact at $artifactDir" - - $deps = Get-Content $repoJson -Raw | ConvertFrom-Json - - # Log resolved versions for debugging - Write-Host "Dependency versions from ${fileName}:" - Write-Host " FLC Core (NuGet): $($deps.'foundry-local-core'.nuget)" - Write-Host " FLC Core (Python): $($deps.'foundry-local-core'.python)" - Write-Host " OnnxRuntime: $($deps.onnxruntime.version)" - Write-Host " GenAI: $($deps.'onnxruntime-genai'.version)" - if ($isWinML -and $deps.'windows-ai-machinelearning') { - Write-Host " Windows AI ML: $($deps.'windows-ai-machinelearning'.version)" - }