Skip to content

refactor: Replace minio/mc with garage/rclone - #459

Merged
NickLarsenNZ merged 42 commits into
mainfrom
minio-to-garage
Oct 8, 2026
Merged

NickLarsenNZ merged 42 commits into
mainfrom
minio-to-garage

Conversation

@NickLarsenNZ

@NickLarsenNZ NickLarsenNZ commented Oct 5, 2026 •

Copy link
Copy Markdown
Member
  • Replace minio with garage
    • I use region-1 as the S3 region to keep things vendor agnostic.
  • replace mc commands with rclone
    • I chose rclone over aws-cli to make it vendor agnostic. There were other options but they were less maintained.
  • Update docs
  • Update storage requirements for the big demo/stack.
  • todo: Only request public parts for S3 clients

Testing

I ran parts of the demos to verify S3 read/write paths (except the big demo).

  • airflow-scheduled-job
  • data-lakehouse-iceberg-trino-spark
  • jupyterhub-keycloak
  • nifi-kafka-druid-earthquake-data (hard to test since Druid writes to S3 after 1 hour)
  • nifi-kafka-druid-water-level-data (hard to test since Druid writes to S3 after 1 hour)
  • spark-k8s-anomaly-detection-taxi-data
  • trino-iceberg
  • trino-taxi-data
  • monitoring (stack)

Also use drawio.png suffix to make the images easier to edit.
spark-k8s-anomaly-detection-taxi-data/overview.drawio.png was made from scratch as it didn't contain the source before.
Note: Using a single node deployment. If the big demo really needs replicas, we will need to adjust how we bootstrap the Garage cluster
@NickLarsenNZ NickLarsenNZ changed the title refactor: Minio to garage refactor: Replace minio with garage Oct 5, 2026
@NickLarsenNZ
NickLarsenNZ requested a review from sbernauer October 7, 2026 13:06
@NickLarsenNZ
NickLarsenNZ marked this pull request as ready for review October 7, 2026 13:07
@NickLarsenNZ
NickLarsenNZ requested a review from adwk67 October 7, 2026 13:27
@NickLarsenNZ NickLarsenNZ changed the title refactor: Replace minio with garage refactor: Replace minio/mc with garage/rclone Oct 7, 2026
@NickLarsenNZ

Copy link
Copy Markdown
Member Author

We could look at putting the Garage helm chart somewhere so we can helm install it.
https://garagehq.deuxfleurs.fr/documentation/cookbook/kubernetes/

@sbernauer sbernauer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May thanks for this PR, it surely was a big effort.
I also like all of the small cleanups. The only "real" problems are related to missing icons in images

Comment thread docs/modules/demos/images/airflow-scheduled-job/overview.drawio.png
Comment thread docs/modules/demos/images/signal-processing/overview.drawio.png
Comment thread docs/modules/demos/images/trino-taxi-data/overview.drawio.png
Comment thread docs/modules/demos/partials/inspect-s3.adoc
Comment thread stacks/_templates/garage.yaml
Comment thread stacks/jupyterhub-keycloak/jupyterhub.yaml
Comment thread stacks/jupyterhub-keycloak/process-s3.ipynb
@NickLarsenNZ
NickLarsenNZ requested a review from sbernauer October 8, 2026 14:24

@sbernauer sbernauer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Images now look good, thanks!

@NickLarsenNZ
NickLarsenNZ merged commit 45ad661 into main Oct 8, 2026
2 checks passed
@NickLarsenNZ
NickLarsenNZ deleted the minio-to-garage branch October 8, 2026 17:26
@NickLarsenNZ

Copy link
Copy Markdown
Member Author

I merged too soon. @adwk67 has some suggestions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Development: Done

Development

Successfully merging this pull request may close these issues.

2 participants