Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 0 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,15 +142,6 @@ curl -X POST \
-i http://localhost:8000/
```

Worker capacity is versioned: the dataplane (the atelet DaemonSet and the
worker pods) schedules only on nodes that carry the
`ate.dev/substrate-version` label, and the install stamps it on every node
that exists when it runs. A node added later hosts no workers until you label
it with the installed version
(`kubectl label node <node> ate.dev/substrate-version=<build version>`).
`kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version`
prints the installed version, off the atelet DaemonSet the install created.

### GKE Quickstart (Development)

1. Create and configure your environment file:
Expand Down Expand Up @@ -194,11 +185,6 @@ prints the installed version, off the atelet DaemonSet the install created.
grant each atespace the namespaces it may read; see
[Enable it](docs/egress-credential-injection.md#enable-it).

Nodes that GKE adds later (autoscaling, auto-repair, node upgrades) are
born with the node pool's labels, so the pool needs
`ate.dev/substrate-version` too; see
[Node version labels](tools/setup-gcp/README.md).

5. You can then deploy the sample applications. See [demos/counter/README.md](demos/counter/README.md) or [demos/sandbox/README.md](demos/sandbox/README.md) for detailed walkthroughs.
```bash
./hack/install-ate.sh --deploy-demo-counter
Expand Down
8 changes: 1 addition & 7 deletions cmd/ate-setup/differences.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,13 +127,7 @@ unpinned reference is rejected by admission and every demo fails to deploy.
Pinning also gives a pre-built install the property a `ko` install had for free,
that a tag moving afterwards cannot change what is running.

That tag also becomes the substrate version, which names the atelet DaemonSet
and sets the node label partitioning nodes across coexisting versions. It has
to: that label must describe the atelet actually running, which came from the
image, not the checkout `git describe` happens to be sitting on. `VERSION`
overrides it, as it always has.

The rewrite is textual, one token at a time, matching `SubstituteVersion` above.
The rewrite is textual, one token at a time.
A reference is replaced wherever it appears, so the code needs no list of which
fields hold images — `workerImage:` in a CRD spec is rewritten like any other —
and the applied manifest differs from the checked-in YAML only in the image
Expand Down
4 changes: 1 addition & 3 deletions cmd/ate-setup/internal/cmd/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -164,9 +164,7 @@ func recordRun(cmd *cobra.Command, runErr error) {
// failed before it had an environment to ask.
var version string
if env != nil {
if v, _, err := env.SubstrateVersion(); err == nil {
version = v
}
version = env.SubstrateVersion()
}

var path string
Expand Down
3 changes: 0 additions & 3 deletions cmd/ate-setup/internal/demos/demotest/demotest.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,6 @@ import (
// configuration, so they can be exercised without a kubeconfig.
func Env(t *testing.T) *steps.Env {
t.Helper()
// Pin VERSION so rendering does not depend on whatever tags (e.g.
// archive/*) happen to be in the local git checkout.
t.Setenv("VERSION", "v0.0.0-test")
root, err := config.RepoRoot()
if err != nil {
t.Fatalf("RepoRoot: %v", err)
Expand Down
8 changes: 1 addition & 7 deletions cmd/ate-setup/internal/demos/render.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,16 +33,10 @@ var ExternalVolumePlaceholders = []string{
"EXTERNAL_VOLUMES",
}

// Render expands a demo template with the configured bucket name and the
// build version (pool templates pin worker pods to version-labeled nodes).
// Render expands a demo template with the configured bucket name.
func Render(e *steps.Env, relPath string, extraValues map[string]string, drop []string) ([]byte, error) {
version, _, err := e.SubstrateVersion()
if err != nil {
return nil, err
}
values := map[string]string{
bucketNamePlaceholder: e.Cfg.BucketName,
"SUBSTRATE_VERSION": version,
}
for k, v := range extraValues {
values[k] = v
Expand Down
9 changes: 4 additions & 5 deletions cmd/ate-setup/internal/images/prebuilt.go
Original file line number Diff line number Diff line change
Expand Up @@ -123,11 +123,10 @@ func (p *Prebuilt) Pin(ctx context.Context, image string) (string, error) {

// rewrite replaces every ko:// reference with the image it maps to.
//
// Substitution is textual, the same way SubstituteVersion fills
// ${SUBSTRATE_VERSION}. References appear in CRD fields as well as pod specs (a
// WorkerPool's spec.workerImage, an ActorTemplate's image), so a schema walk
// would need to know every such field, and rewriting the bytes leaves
// everything else -- the multi-kilobyte Envoy configuration blocks in
// Substitution is textual. References appear in CRD fields as well as pod
// specs (a WorkerPool's spec.workerImage, an ActorTemplate's image), so a
// schema walk would need to know every such field, and rewriting the bytes
// leaves everything else -- the multi-kilobyte Envoy configuration blocks in
// particular -- exactly as committed.
func (p *Prebuilt) rewrite(ctx context.Context, manifest []byte) ([]byte, error) {
// Collect the failures rather than stopping at the first. Diagnosing them
Expand Down
15 changes: 0 additions & 15 deletions cmd/ate-setup/internal/kube/resources.go
Original file line number Diff line number Diff line change
Expand Up @@ -155,21 +155,6 @@ func (c *Client) PatchDeployment(ctx context.Context, namespace, name string, pa
return nil
}

// DaemonSetNames lists the DaemonSets in a namespace matching a label
// selector. The atelet DaemonSet name carries a substrate version suffix, so
// the installed versions can only be found by label.
func (c *Client) DaemonSetNames(ctx context.Context, namespace, selector string) ([]string, error) {
list, err := c.Typed.AppsV1().DaemonSets(namespace).List(ctx, metav1.ListOptions{LabelSelector: selector})
if err != nil {
return nil, fmt.Errorf("while listing daemonsets in %s matching %q: %w", namespace, selector, err)
}
names := make([]string, 0, len(list.Items))
for _, ds := range list.Items {
names = append(names, ds.Name)
}
return names, nil
}

// SetServiceAccountAnnotation adds or removes one annotation on a
// ServiceAccount; an empty value removes it. A missing ServiceAccount is not
// an error, matching the `|| true` the shell installer removed one under.
Expand Down
9 changes: 5 additions & 4 deletions cmd/ate-setup/internal/steps/csi.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import (
"os"
"os/exec"
"strings"
"time"

apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Expand Down Expand Up @@ -290,8 +291,8 @@ spec:

// The image cache directories under ateomHostDir were just wiped, so
// atelet has to recreate them.
log.Infof("Restarting the atelet DaemonSets (if present)...")
return e.RestartAteletDaemonSets(ctx)
log.Infof("Restarting the atelet DaemonSet (if present)...")
return e.Kube.RolloutRestart(ctx, e.Namespace(), "atelet", time.Now())
}

func (e *Env) setupCSINFS(ctx context.Context) error {
Expand Down Expand Up @@ -422,8 +423,8 @@ spec:
return err
}

log.Infof("Restarting the atelet DaemonSets (if present)...")
return e.RestartAteletDaemonSets(ctx)
log.Infof("Restarting the atelet DaemonSet (if present)...")
return e.Kube.RolloutRestart(ctx, e.Namespace(), "atelet", time.Now())
}

// checkNFSDSupport verifies the host kernel can serve NFS. The in-cluster NFS
Expand Down
11 changes: 1 addition & 10 deletions cmd/ate-setup/internal/steps/delete.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,6 @@ func (e *Env) DeleteAteSystem(ctx context.Context) error {
return err
}

// atelet DaemonSet names carry a version suffix.
if err := e.Kube.Typed.AppsV1().DaemonSets(e.Namespace()).DeleteCollection(ctx,
metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: "app=atelet"}); err != nil {
return fmt.Errorf("while deleting atelet daemonsets: %w", err)
}

for _, path := range [][]string{
{"components", "agentgateway", "configmap.yaml"},
{"postgres", "postgres.yaml"},
Expand All @@ -94,10 +88,7 @@ func (e *Env) DeleteAteSystem(ctx context.Context) error {
if err := e.Kube.WaitDeleted(ctx, namespace, "", NamespacePodCert, e.Cfg.WaitTimeout(BootstrapTimeout)); err != nil {
return err
}
if err := e.DeleteTrustBundles(ctx); err != nil {
return err
}
return e.UnlabelNodesSubstrateVersion(ctx)
return e.DeleteTrustBundles(ctx)
}

// releaseSandboxConfigs removes the WorkerPool protection finalizer from
Expand Down
38 changes: 4 additions & 34 deletions cmd/ate-setup/internal/steps/deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,8 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error {
if err := e.RequireCanonicalNamespace("deploy ate-system"); err != nil {
return err
}
// Fail fast on an unusable build version or credential provider selection
// before touching the cluster.
if _, _, err := e.SubstrateVersion(); err != nil {
return err
}
// Fail fast on an unusable credential provider selection before touching
// the cluster.
provider, err := e.Cfg.CredentialProvider()
if err != nil {
return err
Expand All @@ -85,12 +82,6 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error {
return err
}

// Before the bundle: the atelet DaemonSet applied below and the demo
// WorkerPools' version-pinned pods schedule only to version-labeled nodes.
if err := e.LabelNodesSubstrateVersion(ctx); err != nil {
return err
}

// DeployCRDs, not EnsureCRDs: an existence check would skip upgrades,
// stranding stale CRD schemas and RBAC (role.yaml has no other apply
// path).
Expand Down Expand Up @@ -139,12 +130,6 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error {
if err != nil {
return err
}
// The atelet DaemonSet in the bundle is version-keyed; fill its
// placeholders after the render (kustomize and ko pass them through).
manifests, err = e.SubstituteVersion(manifests)
if err != nil {
return err
}
if err := e.applyWithAPIServerEnvHash(ctx, manifests); err != nil {
return err
}
Expand Down Expand Up @@ -172,10 +157,6 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error {
return err
}

ateletName, err := e.AteletDaemonSetName()
if err != nil {
return err
}
log.Step("Waiting for ATE system components to be ready...")
type rollout struct{ kind, name string }
var waits []rollout
Expand All @@ -184,7 +165,7 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error {
rollout{kube.KindDeployment, "ate-controller"},
rollout{kube.KindDeployment, "atenet-router"},
rollout{kube.KindDeployment, "atenet-egress"},
rollout{kube.KindDaemonSet, ateletName},
rollout{kube.KindDaemonSet, "atelet"},
)
if provider.Kubernetes() {
waits = append(waits, rollout{kube.KindDeployment, k8sCredentialProviderDeployment})
Expand Down Expand Up @@ -354,9 +335,6 @@ func (e *Env) DeployAtelet(ctx context.Context) error {
if err := e.EnsureAteSystemNamespace(ctx); err != nil {
return err
}
if err := e.LabelNodesSubstrateVersion(ctx); err != nil {
return err
}
if err := e.applyOtelConfig(ctx); err != nil {
return err
}
Expand All @@ -375,18 +353,10 @@ func (e *Env) DeployAtelet(ctx context.Context) error {
if err != nil {
return err
}
manifest, err = e.SubstituteVersion(manifest)
if err != nil {
return err
}
if err := e.Kube.ApplyBytes(ctx, manifest); err != nil {
return err
}
ateletName, err := e.AteletDaemonSetName()
if err != nil {
return err
}
return e.Kube.RolloutStatus(ctx, kube.KindDaemonSet, e.Namespace(), ateletName, e.Cfg.RolloutTimeout)
return e.Kube.RolloutStatus(ctx, kube.KindDaemonSet, e.Namespace(), "atelet", e.Cfg.RolloutTimeout)
}

// DeployAtenet redeploys the atenet dataplane: router and egress.
Expand Down
5 changes: 0 additions & 5 deletions cmd/ate-setup/internal/steps/env.go
Original file line number Diff line number Diff line change
Expand Up @@ -81,11 +81,6 @@ type Env struct {
// work on a machine with no container registry credentials configured, and
// the build-from-source resolver fails at construction without them.
resolver imageResolver

// substrateVersion caches the derived build version and its object-name
// suffix; see SubstrateVersion.
substrateVersion string
substrateVersionSuffix string
}

// Namespace is the namespace the control plane is installed into. It is
Expand Down
13 changes: 1 addition & 12 deletions cmd/ate-setup/internal/steps/overlay.go
Original file line number Diff line number Diff line change
Expand Up @@ -529,16 +529,5 @@ func (e *Env) applyOtelEndpointOverride(ctx context.Context) error {
return err
}
}
// atelet DaemonSet names carry a version suffix; restart whichever
// versions are installed.
daemonSets, err := e.Kube.DaemonSetNames(ctx, e.Namespace(), "app=atelet")
if err != nil {
return err
}
for _, name := range daemonSets {
if err := e.Kube.RolloutRestart(ctx, e.Namespace(), name, now); err != nil {
return err
}
}
return nil
return e.Kube.RolloutRestart(ctx, e.Namespace(), "atelet", now)
}
7 changes: 2 additions & 5 deletions cmd/ate-setup/internal/steps/overlay_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -772,13 +772,10 @@ func TestApplyOtelEndpointOverride(t *testing.T) {
})

t.Run("patches and restarts the consumers", func(t *testing.T) {
// The atelet DaemonSet name carries a substrate version suffix, so it
// can only be found by label.
atelet := &appsv1.DaemonSet{
ObjectMeta: metav1.ObjectMeta{
Namespace: NamespaceAteSystem,
Name: "atelet-v1-2-3",
Labels: map[string]string{"app": "atelet"},
Name: "atelet",
},
}
e := &Env{
Expand All @@ -799,7 +796,7 @@ func TestApplyOtelEndpointOverride(t *testing.T) {
if !restartedAt(t, e, "deployment", "ate-api-server") {
t.Error("ate-api-server was not restarted")
}
if !restartedAt(t, e, "daemonset", "atelet-v1-2-3") {
if !restartedAt(t, e, "daemonset", "atelet") {
t.Error("the atelet DaemonSet was not restarted")
}
})
Expand Down
10 changes: 2 additions & 8 deletions cmd/ate-setup/internal/steps/publish.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,7 @@ var workerImages = []string{"ateom-gvisor", "ateom-microvm"}
// output. A WorkerPool moves to this build by pointing its workerImage at
// the ref.
func (e *Env) PublishWorkerImages(ctx context.Context, w io.Writer) error {
version, _, err := e.SubstrateVersion()
if err != nil {
return err
}
version := e.SubstrateVersion()
log.Stepf("publish_worker_images (%s)", version)
runner, err := e.koRunner()
if err != nil {
Expand Down Expand Up @@ -71,10 +68,7 @@ func (e *Env) PublishReleaseImages(ctx context.Context, w io.Writer) error {
if e.Cfg.KODockerRepo == "" {
return fmt.Errorf("publishing release images needs a registry to push to; set KO_DOCKER_REPO or --ko-docker-repo")
}
tag, _, err := e.SubstrateVersion()
if err != nil {
return err
}
tag := e.SubstrateVersion()
log.Stepf("publish_release_images (%s)", tag)
runner, err := e.koRunner()
if err != nil {
Expand Down
Loading
Loading