Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ composer.phar
*.env.json
/.phpunit.result.cache
/phpunit.xml
/docs/superpowers
51 changes: 51 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -488,6 +488,57 @@ Behavior:
- Prints a summary: projects checked/disabled/errored, configurations scanned/touched, tasks migrated/skipped
(unsupported vs. unresolvable).

### Migrate keboola.orchestrator configurations to keboola.flow
Batch driver for the automated `keboola.orchestrator` → `keboola.flow` migration
(see [AJDA-3117](https://linear.app/keboola/issue/AJDA-3117)). All migration logic lives in the
`keboola.flow-migration-tool` component; this command only creates one migration job per project
and supervises the batch. Safe to re-run with the same list: already-migrated orchestrations are
reported as skipped by the component, and projects with a live migration job are skipped here.

```
php cli.php manage:migrate-orchestrations-to-flow [-f|--force] <token> <url> [<projects>] \
[--projects-file=PATH] [--concurrency=10] [--poll-interval=5] [--report=PATH]
```

Arguments:
- `token` (required): Manage API token.
- `url` (required): Stack URL, including `https://` (e.g. `https://connection.north-europe.azure.keboola.com`).
- `projects` (optional): Comma-separated project IDs (e.g. `1,7,146`), or `@path/to/file` with one ID
per line (blank lines and `#` comments are ignored). Exactly one of `projects`/`--projects-file`
must be given.

Options:
- `--force` / `-f`: Run the real migration. Without it, jobs are created with `dryRun: true`.
**Note:** even without `--force` a real `keboola.flow-migration-tool` job and a real ephemeral
storage token are created in every eligible project — on PAYGO stacks mind the billing.
- `--projects-file=PATH`: File with one project ID per line (alternative to `@file` in the argument).
- `--concurrency=N` (default 10): Max migration jobs in flight at once.
- `--poll-interval=N` (default 5): Seconds between job status polls.
- `--report=PATH` (default `flow-migration-<stack>-<timestamp>.csv`): CSV report path.

Behavior:
- For each project: skips disabled/deleted projects; creates an ephemeral 1h storage token with full
project rights (`canManageBuckets`, `canManageTokens`, `canReadAllFileUploads`, `canPurgeTrash`) so
the component cannot be short of a permission mid-migration; skips projects with no
`keboola.orchestrator` configurations (no empty jobs in customers' job history).
- Creates the migration job via `configData` (no stored configuration is left behind) with
`parameters: {mode: "project", orchestrationIds: [], skipBroken: true, dryRun: <!force>}`.
- Keeps at most `--concurrency` jobs in flight, polls each job and refills the window as jobs finish.
A transient poll failure is tolerated up to 3 consecutive times per job; after that the project is
reported as failed and the job is left to finish server-side (its job ID stays in the report).
- Appends a CSV row (`projectId;jobId;status;durationSeconds;error`) the moment each project
resolves, so an interrupted run is still auditable. Every input project gets a row; skipped
projects carry the skip reason in `status`/`error` and an empty `jobId`. Re-running with the same
`--report` path appends to the existing file without repeating the header.
- A failing project never aborts the batch. Exit code is `1` if at least one project failed
(job `error`/`terminated`/`cancelled` or a driver-side error), `0` otherwise.
- Final summary: projects attempted / migrated / migrated with warning / skipped (no
orchestrations, disabled) / failed.
- Re-running the same project list is the intended recovery path: the component reports
already-migrated orchestrations as skipped. The command does **not** check for a migration job
already running in the project, so before re-running an interrupted batch let the jobs it already
created finish - two concurrent migrations of one project can both create the same flows.

### Mass enablement of dynamic backends for multiple projects
Prerequisities: https://keboola.atlassian.net/wiki/spaces/KB/pages/2135982081/Enable+Dynamic+Backends#Enable-for-project

Expand Down
2 changes: 2 additions & 0 deletions cli.php
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
use Keboola\Console\Command\MassProjectEnableDynamicBackends;
use Keboola\Console\Command\MassProjectExtendExpiration;
use Keboola\Console\Command\MigrateDataAppsOrchestratorTasks;
use Keboola\Console\Command\MigrateOrchestrationsToFlow;
use Keboola\Console\Command\OrganizationIntoMaintenanceMode;
use Keboola\Console\Command\OrganizationResetWorkspacePasswords;
use Keboola\Console\Command\OrganizationsAddFeature;
Expand Down Expand Up @@ -48,6 +49,7 @@
$application->add(new MassProjectExtendExpiration());
$application->add(new MassProjectEnableDynamicBackends());
$application->add(new MigrateDataAppsOrchestratorTasks());
$application->add(new MigrateOrchestrationsToFlow());
$application->add(new AddFeature());
$application->add(new CleanupLeakedTestFeatures());
$application->add(new AllStacksIterator());
Expand Down
Loading