Skip to content

feat(gcp): Dataproc Metastore services - #1251

Merged
NitinKumar004 merged 1 commit into
developmentfrom
feat/gcp-metastore
Sep 7, 2026
Merged

feat(gcp): Dataproc Metastore services#1251
NitinKumar004 merged 1 commit into
developmentfrom
feat/gcp-metastore

Conversation

@NitinKumar004

Copy link
Copy Markdown
Collaborator

Summary

New greenfield GCP control-plane service: Dataproc Metastore (metastore.googleapis.com/v1, resource type services). Track-1 control-plane emulation only (no Hive data plane). Real google.golang.org/api/metastore/v1 clients, gcloud, and the Terraform google_dataproc_metastore_service resource hit it unchanged.

Provider tier: GA. google_dataproc_metastore_service is in the STABLE hashicorp/google provider at the default /v1/ path (verified live with provider v8.1.0); the metastore/v1 SDK confirms the GA surface. No beta path needed.

What's covered

  • POST .../services?serviceId= · GET (item + list) · PATCH ?updateMask= · DELETE, plus location-scoped operations polls.
  • Every mutation returns a google.longrunning.Operation done:true synchronously, resource embedded in response as an Any typed type.googleapis.com/google.cloud.metastore.v1.Service — no LRO hang. Operations flow through the shared lro.Registry; Matches yields operations paths to the shared poller so no sibling GCP service regresses.
  • Client-named create via ?serviceId= (body name fallback); 409 dup / 404 missing / 400 bad input.
  • Raw passthrough for every caller-supplied body key (encryptionConfig, networkConfig, scalingConfig, maintenanceWindow, hiveMetastoreConfig, labels, …); updateMask top-level replace, unmentioned fields survive.
  • COW memstore with deep clone on read; Snapshottable compile-asserted and auto-discovered by persist completeness.

Computed-field stability (drift prevention)

name, state (seeded ACTIVE), stateMessage, uid, endpointUri (deterministic Thrift endpoint from name+port), artifactGcsUri (deterministic gs://…/hive-metastore), createTime/updateTime are minted once at create, stored, and stable across GETs. The Computed port (9083) and provider-defaulted databaseType (MYSQL) / releaseChannel (STABLE) / tier are filled when omitted.

Per the provider schema (terraform providers schema -json), telemetryConfig and hiveMetastoreConfig are Optional, non-Computed nested blocks — so they are carried verbatim and deliberately not server-injected; injecting them was the sole plan-drift source and is avoided.

Live e2e (Terraform, hashicorp/google STABLE, default path)

Ran a real cloudemu serve --gcp-port 18050 + terraform (provider v8.1.0) against a google_dataproc_metastore_service with service_id/location/tier=DEVELOPER, leaving port/database_type/release_channel/telemetry/hive UNSET:

  • apply → created, LRO did not hang, state = ACTIVE.
  • plan -detailed-exitcodeexit 0, no drift, specifically on the defaulted/computed port, database_type, release_channel, state, endpoint_uri, artifact_gcs_uri, uid, create_time.
  • update (add labels + maintenance_window) → in-place PATCH applied; re-plan exit 0 clean.
  • destroy → removed; endpoint returns 404.

Gates

go build, full go vet, gofmt, golangci-lint (0 issues), go test -race on new packages, full go test ./server/gcp/... (no sibling operations regression), ./providers/..., ./persist/... (completeness), root ., and go generate idempotency — all green. Docs regenerated (metastore-only diff).

Deferred

metadataImports, backups, federations subcollections, IAM policy verbs, and the real Hive metastore data plane.

Add the metastore.googleapis.com/v1 services control plane (GA /v1/): Create/
Get/List/Patch/Delete, all LRO-wrapped and resolved inline via the shared lro
registry with the typed Service Any response. Server fills the Computed defaults
(port, tier, network, state, uid, endpointUri, artifactGcsUri) and the
provider-defaulted database_type/release_channel, minted once at create and
stable across reads so terraform stays drift-free. Nested telemetryConfig/
hiveMetastoreConfig are Optional non-Computed blocks, carried through verbatim
and never injected.
@NitinKumar004
NitinKumar004 merged commit 3e4fcb5 into development Sep 7, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant