Rd 1598 by pegahnikbakht · Pull Request #7 · canarybit/tower

pegahnikbakht · 2026-10-13T08:31:37Z

There are a couple of fixes in this PR for RHEL:

First fixing the package name.

Then fixing the permission deny issue because of SELinux policy on running cbclient, now the tokens will move to /etc/canarybit/tokens

sudo namei -l /home/tower/tokens
f: /home/tower/tokens
dr-xr-xr-x root root /
drwxr-xr-x root root home
drwx------ tower tower tower
-rw-r--r-- tower tower tokens
[tower@tower-cvm ~]$ sudo ls -lZ /home/tower /home/tower/tokens
-rw-r--r--. 1 tower tower unconfined_u:object_r:user_home_t:s0 4444 Oct 7 11:05 /home/tower/tokens

/home/tower:
total 12
-rw-r--r--. 1 tower tower unconfined_u:object_r:user_home_t:s0 3268 Oct 7 11:05 signing-key.pem
-rw-r--r--. 1 tower tower unconfined_u:object_r:user_home_t:s0 4444 Oct 7 11:05 tokens
[tower@tower-cvm ~]$ sudo ausearch -m AVC -ts recent
time->Wed Oct 7 11:05:45 2026
type=AVC msg=audit(1791371145.176:176): avc: denied { read } for pid=1 comm="systemd" name="tokens" dev="dm-0" ino=132396 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0

The RHEL error is caused by SELinux, not ordinary file permissions. The AVC record showed systemd (init_t) was denied access to tokens labeled user_home_t. The file being readable by root (0644) doesn’t override SELinux policy. Ubuntu usually doesn’t enforce SELinux with this policy, which explains why it behaves differently there.

Then fix the cloud-init on GCP. Now there is a launcher script on GCP that first install cloud-init and do a reboot and then the cloud-init will run after that.

antoniacobaeus

Looks good overall but it really gets a bit complicated to just enable cloud-init for RHEL on GCP.

Overall it is really annoying that RHEL decided to go this route for GCP. It seems like there are three possible solutions/approaches:

Use only the startup-script from GCP, which would mean we have to maintain that script for GCP and cannot rely on the same cloud-init
Bake a custom image with cloud init. I think we wouldn't do this and leave that to the customer.
Bootstrap cloud-init in the startup-script, which is the approach of this PR

I like the approach of the PR, but it gets really quite messy fast with approach 3. I personally believe that in these cases it might be better to just require cloud-init to be enabled in the image and simply not support RHEL on GCP for that purpose. Customers can then do approach 2, with maybe some guidance, and then we kind of leave Tower out of it.

@xstecuc what is your opinion on this?

If we decide to go this route I have some comments below. Changes to the cloud-init will likely conflict or get lost with the introduction of tower-commons so we should go over what is necessary.

antoniacobaeus · 2026-10-13T08:35:10Z

-  - libtss2-dev
  - jq

+runcmd:


With tower-commons we can instead declare which packages to install from terraform. It might be worth syncing this with the tower commons changes (when it gets merged), to avoid this workaround.

antoniacobaeus · 2026-10-13T08:37:34Z

+      set -euo pipefail

-      cat /home/${USERNAME}/tokens >> /etc/environment
+      if [[ -f /home/${USERNAME}/tokens ]]; then


What does this do?

Does it translate between /home/${USERNAME}/tokens and /etc/canarybit/tokens?

I got permission deny on rhel because of SELinux policy:

The RHEL error is caused by SELinux, not ordinary file permissions. The AVC record showed systemd (init_t) was denied access to tokens labeled user_home_t. The file being readable by root (0644) doesn’t override SELinux policy. Ubuntu usually doesn’t enforce SELinux with this policy, which explains why it behaves differently there.

antoniacobaeus · 2026-10-13T08:39:01Z

+      mv ${CB_INSPECTOR_CLIENT_V}/cb-inspector-client /usr/local/bin/cb-inspector-client

-      # Start cb-inspector-client as System service once and then a defined frequency
+      systemctl daemon-reload


Is this needed?

antoniacobaeus · 2026-10-13T08:40:40Z

-
-      # Download the CanaryBit Inspector agent (cb-inspector-client) into /usr/local/bin
-      cb download cb-inspector-client ${CB_INSPECTOR_CLIENT_V}/cb-inspector-client; chmod +x ${CB_INSPECTOR_CLIENT_V}/cb-inspector-client; mv ${CB_INSPECTOR_CLIENT_V}/cb-inspector-client /usr/local/bin/cb-inspector-client
+      curl -fsSL https://canarybit-public-binaries.s3.eu-west-1.amazonaws.com/cb-cli/${CBCLI_V}/cb-x86_64-unknown-linux-gnu -o /usr/local/bin/cb


I also prefer this, but I am curious why it got changed. It can still be one line if we want (just add the ;)

This will likely be replaced by tower-commons so if this format is needed for some cloud-init implementation make sure to document it

Ah just saw the description to the PR, that makes sense, then we will likely prefer this over chmod +x in the future.

antoniacobaeus · 2026-10-13T08:44:56Z

+  }
+}
+
+resource "null_resource" "cloud_init_reboot" {


Can this be moved to a separate file, since it is quite a specific workaround?

Agree that it get a bit messy in GCP.

antoniacobaeus · 2026-10-13T08:48:25Z

+variable "os_name" {
+  description = "Operating system family used to configure OS-specific bootstrap steps"
+  type        = string
+  default     = "ubuntu"


Does the value "ubuntu" currently do anything?

In case it doesn't we could consider to have the variable be "rhel_workarounds_enabled" or something similar. Then it is fairly clear that it only relates to rhel. On the other hand this shows the end user directly what we have tested on.

ubuntu does not do anything now, but there might be some other OS in future that we test and see they require some specific change, do you think we still can use "workarounds_enabled" for them?

antoniacobaeus · 2026-10-13T09:06:09Z

+      "echo 'Waiting for cloud-init installation'",
+      "for i in $(seq 1 120); do command -v cloud-init >/dev/null 2>&1 && break; sleep 5; done",
+      "command -v cloud-init >/dev/null 2>&1 || { echo 'Timed out waiting for cloud-init installation' >&2; exit 1; }",
+      "sudo systemd-run --no-block --unit=terraform-cloud-init-reboot --on-active=10s /usr/sbin/reboot",


Not sure if it works, but potentially we could skip the reboot if we don't use anything from cloud-init that runs early on.

Something like (got it from LLM):

# Run all stages in order (init-local, init, modules:config, modules:final) cloud-init clean --logs cloud-init init --local cloud-init init cloud-init modules --mode=config cloud-init modules --mode=final

It might be quite messy though if someone wants to use something extra from cloud-init

xstecuc · 2026-10-14T11:44:24Z

@antoniacobaeus yes, I agree with you! I think we should just mention in our docs that "Tower does not support OS images that do not have cloud-init pre-installed!".
This way we will keep a standard approach and we will avoid corner cases that are specific to certain deployment configs.

antoniacobaeus

LGTM

We can add the same OS_NAME variable to tower-commons. Then it might make sense to only do the validation there, to kind of reduce the space, but we can leave that for later.

These same changes are probably needed later in tower premium as well

Fix rhel package name, cloud-init and permission issue

8cd1a56

pegahnikbakht requested a review from antoniacobaeus October 13, 2026 08:31

github-actions Bot added 3 commits October 13, 2026 08:31

terraform-docs: automated action

4cdda22

terraform-docs: automated action

28a3409

terraform-docs: automated action

8e99305

antoniacobaeus reviewed Oct 13, 2026

View reviewed changes

pegahnikbakht requested a review from xstecuc October 13, 2026 09:44

pegahnikbakht and others added 5 commits October 16, 2026 13:14

Revert gcp changes

4f9f6b8

Merge remote-tracking branch 'origin/main' into RD-1598

8bcd47b

terraform-docs: automated action

2b8750d

terraform-docs: automated action

0347304

terraform-docs: automated action

d0a48dd

antoniacobaeus self-requested a review October 19, 2026 07:24

antoniacobaeus approved these changes Oct 19, 2026

View reviewed changes

Conversation

pegahnikbakht commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antoniacobaeus left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antoniacobaeus Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antoniacobaeus Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xstecuc commented Oct 14, 2026

Uh oh!

antoniacobaeus left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pegahnikbakht commented Oct 13, 2026 •

edited

Loading

antoniacobaeus left a comment •

edited

Loading

antoniacobaeus Oct 13, 2026 •

edited

Loading

antoniacobaeus Oct 13, 2026 •

edited

Loading