ntp: use per-server chrony settings by xiaotianlyu · Pull Request #4923 · bottlerocket-os/bottlerocket

xiaotianlyu · 2026-09-02T03:29:21Z

Description of changes:

Updates the shared NTP defaults to use named per-server chrony settings:

configures 169.254.169.123 as a preferred server polled every 16 seconds;
replaces 2.amazon.pool.ntp.org with the time.aws.com pool; and
adds configurable chrony file logging through settings.ntp.logging; logging
is disabled by default.

Adds a datastore migration for existing nodes. On upgrade, it converts the old settings.ntp.time-servers list and shared options into named servers. The new recommended settings are applied only when the old configuration matches the previous defaults; customer-configured servers and options are preserved.

On rollback, the migration converts named servers back into a list and shared options that the old API can read. Since the old format supports only one shared options list, rollback preserves options that are common to all named servers.

The migration is required because Storewolf otherwise retains the old list while adding the new named keys. This was reproduced during upgrade as:

duplicate field time_servers

The migration is registered for the 1.66.0 -> 1.67.0 transition.

This change updates Bottlerocket to:

bottlerocket-settings-models 0.30.0;
bottlerocket-sdk 0.80.0;
bottlerocket-core-kit 17.1.0;
bottlerocket-kernel-kit 9.3.0.

Testing done:

Local tests:

cargo test -p ntp-time-servers: 12 passed
cargo test -p migration-helpers: 32 passed
cargo check -p settings-plugin-aws-k8s: passed
cargo clippy -p ntp-time-servers --all-targets -- -D warnings: passed
cargo clippy -p migration-helpers --all-targets -- -D warnings: passed
cargo fmt --manifest-path sources/Cargo.toml --all -- --check: passed
git diff --check: passed

Migration tests cover:

converting the old AWS and public defaults to named servers;
preserving custom servers, options, and metadata;
handling an empty old server list;
converting named servers back to the old list form;
preserving an explicit empty shared-options list during rollback;
restoring metadata during rollback;
removing named data that the old API cannot represent.

Full image upgrade and rollback test:

built an aws-k8s-1.32 x86_64 1.67.0 image using bottlerocket-settings-models 0.30.0, SDK 0.80.0, core-kit 17.1.0, and kernel-kit 9.3.0;
generated a signed TUF repository and verified it contained migrate_v1.67.0_ntp-time-servers.lz4;
launched a Bottlerocket 1.66.0 node with the old server list;
upgraded the node from 1.66.0 to the test 1.67.0 image;
rolled the node back to 1.66.0.

After upgrade:

the migrator journal reported that the NTP migration completed successfully;
apiclient get settings.ntp returned only the named link-local and amazon-pool entries, with no duplicate-field error;

/etc/chrony.conf rendered:

pool time.aws.com iburst
server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4
driftfile /var/lib/chrony/drift
makestep 1.0 3
dumponexit
dumpdir /var/lib/chrony
logdir /var/log/chrony
user chrony
rtcsync

no log directive was rendered by default;
changing logging to ["tracking"] rendered log tracking;
changing logging to [] removed the log directive;
chronyd was active with NRestarts=0.

After rollback:

the node returned to Bottlerocket 1.66.0;
the backward migration completed successfully;
the old API read 169.254.169.123 and time.aws.com in the restored list;
the old chrony template rendered both entries with the shared iburstoption;
chronyd was active with NRestarts=0.

Fresh install test:

rebuilt an aws-k8s-1.32 x86_64 1.67.0 image from the final PR commit;
launched a new node directly from the image, without running a migration;
confirmed apiclient get settings.ntp returned the default named link-local and amazon-pool servers;
confirmed /etc/chrony.conf rendered the expected server and pool directives with no log directive by default;
chronyd selected 169.254.169.123 about five seconds after boot;
chronyd corrected a natural startup offset of 1.316646 seconds;
chronyd remained active with NRestarts=0.

Related PRs:

settings-sdk: feat: ntp: support per-server time source configuration bottlerocket-settings-sdk#146
core-kit: chrony: render per-server time source configuration bottlerocket-core-kit#1030
bottlerocket: ntp: use per-server chrony settings #4923

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed
under the terms of both the Apache License, version 2.0, and the MIT license.

ginglis13 · 2026-09-17T17:31:15Z

+# chrony log categories, rendered as the `log` line.
+logging = ["measurements", "statistics", "tracking"]


logging should be opt-in, not on by default

Suggested change

# chrony log categories, rendered as the `log` line.

logging = ["measurements", "statistics", "tracking"]

Updated, thanks!

ginglis13 · 2026-09-17T17:31:29Z

+# chrony log categories, rendered as the `log` line.
+logging = ["measurements", "statistics", "tracking"]


logging should be opt-in, not on by default

Suggested change

# chrony log categories, rendered as the `log` line.

logging = ["measurements", "statistics", "tracking"]

Updated, thanks! Chrony file logging is now opt-in and is no longer enabled in the shared defaults.

vyaghras · 2026-09-29T22:48:40Z

+        Ok(input)
+    }
+
+    fn backward(&mut self, mut input: MigrationData) -> Result<MigrationData> {


Lets write a migration helper for list-to-named-map conversion and use that in the backward and forward migrations.

Thank you! I added shared helpers for list-to-named-map and named-map-to-list conversion, and updated both the forward and backward NTP migration to use them.

ginglis13 · 2026-09-30T21:03:48Z

this commit just removes lines that were added in the previous commit. Can you please squash the 2 commits together instead ?

Thank you! Updated

ginglis13 · 2026-09-30T21:19:34Z

 /// Converts a list setting into scalar fields below `prefix.<name>.<field>`.
 ///
 /// The converter owns the value-specific mapping. Returning `None` leaves the input unchanged.
+/// This helper only changes data; callers must migrate related metadata separately.


This should go in the commit which added the migration helper.

Yes, updated

ginglis13 · 2026-09-30T21:20:16Z

it looks like we're duplicating logic here and in the shared helper.

you could make key-walkers generic over the value type and expose them like:

pub fn read_flattened_named_map<V: Clone>(data: &HashMap<String, V>, prefix: &[&str]) -> Result<Option<BTreeMap<String, BTreeMap<String, V>>>> pub fn replace_flattened_named_map<V: Clone>(data: &mut HashMap<String, V>, prefix: &[&str], replacement: &BTreeMap<String, BTreeMap<String, V>>) -> Result<()>

Then main.rs calls the same functions for both data (V = Value) and metadata (V = Metadata), deleting all three hand-rolled routines and getting Key-based parsing on metadata for free.

Makes sense! Thank you, I updated the helper

Signed-off-by: Melody Lyu <tianlyu@amazon.com>

Configure the link-local time source as a preferred server with fixed polling, and replace the legacy Amazon pool with time.aws.com. Add support for configurable chrony logging while leaving it disabled by default. Signed-off-by: Melody Lyu <tianlyu@amazon.com>

Signed-off-by: Melody Lyu <tianlyu@amazon.com>

Convert legacy NTP server lists and shared options into named per-server settings during upgrade. Use the shared named-map migration helper. Restore the legacy server list and common options during rollback. Register the migration for the 1.66.0 to 1.67.0 transition. Signed-off-by: Melody Lyu <tianlyu@amazon.com>

maherthomsi · 2026-10-01T06:19:28Z

Are you able to test custom server configurations or configurations with NTP is turned off being preserved after an upgrade? I want to test to see if custom configuration is preserved or if this would affect users who remove AWS servers on purpose if they upgrade.

This was referenced Sep 2, 2026

feat: ntp: support per-server time source configuration bottlerocket-os/bottlerocket-settings-sdk#146

Merged

chrony: render per-server time source configuration bottlerocket-os/bottlerocket-core-kit#1030

Merged

xiaotianlyu force-pushed the ntp-per-server-config branch from d2acce8 to 97ae203 Compare September 2, 2026 03:37

xiaotianlyu changed the title ~~ntp: improve Amazon Time Sync recovery~~ ntp: add per-server Amazon Time Sync configuration Sep 2, 2026

xiaotianlyu changed the title ~~ntp: add per-server Amazon Time Sync configuration~~ ntp: use per-server chrony settings for Amazon Time Sync Sep 2, 2026

xiaotianlyu changed the title ~~ntp: use per-server chrony settings for Amazon Time Sync~~ ntp: use per-server chrony settings Sep 2, 2026

ginglis13 reviewed Sep 17, 2026

View reviewed changes

xiaotianlyu force-pushed the ntp-per-server-config branch from cc4d647 to f0e9cba Compare September 24, 2026 18:30

xiaotianlyu marked this pull request as ready for review September 29, 2026 22:11

vyaghras reviewed Sep 29, 2026

View reviewed changes

xiaotianlyu force-pushed the ntp-per-server-config branch 8 times, most recently from 1887701 to 9143cf0 Compare September 30, 2026 16:25

ginglis13 reviewed Sep 30, 2026

View reviewed changes

ginglis13 requested changes Sep 30, 2026

View reviewed changes

xiaotianlyu force-pushed the ntp-per-server-config branch from 9143cf0 to e6e731d Compare September 30, 2026 21:59

xiaotianlyu added 3 commits September 30, 2026 22:02

sources: bump settings-models to 0.30.0

faac33f

Signed-off-by: Melody Lyu <tianlyu@amazon.com>

twoliter: bump SDK 0.80.0, core-kit 17.1.0, kernel-kit 9.3.0

6c673e4

Signed-off-by: Melody Lyu <tianlyu@amazon.com>

xiaotianlyu force-pushed the ntp-per-server-config branch 2 times, most recently from 78314a4 to 9a45bf1 Compare September 30, 2026 22:10

xiaotianlyu added 2 commits September 30, 2026 22:16

migration-helpers: support list and named map conversion

7ad9d5c

Signed-off-by: Melody Lyu <tianlyu@amazon.com>

xiaotianlyu force-pushed the ntp-per-server-config branch from 9a45bf1 to b88ee61 Compare September 30, 2026 22:18

ginglis13 approved these changes Sep 30, 2026

View reviewed changes

This was referenced Oct 1, 2026

storewolf: preserve parent settings when applying defaults bottlerocket-os/bottlerocket-core-kit#1073

Closed

chrony: revert per-server time source configuration bottlerocket-os/bottlerocket-core-kit#1074

Merged

Navigation Menu

Repository navigation

ntp: use per-server chrony settings - #4923

ntp: use per-server chrony settings#4923
xiaotianlyu wants to merge 5 commits into
bottlerocket-os:developfrom
xiaotianlyu:ntp-per-server-config

xiaotianlyu commented Sep 2, 2026 •

edited

Loading

Uh oh!

ginglis13 Sep 17, 2026

Uh oh!

xiaotianlyu Sep 29, 2026

Uh oh!

ginglis13 Sep 17, 2026

Uh oh!

xiaotianlyu Sep 24, 2026

Uh oh!

vyaghras Sep 29, 2026

Uh oh!

xiaotianlyu Sep 30, 2026

Uh oh!

ginglis13 Sep 30, 2026

Uh oh!

xiaotianlyu Sep 30, 2026

Uh oh!

ginglis13 Sep 30, 2026

Uh oh!

xiaotianlyu Sep 30, 2026

Uh oh!

ginglis13 Sep 30, 2026

Uh oh!

xiaotianlyu Sep 30, 2026

Uh oh!

maherthomsi commented Oct 1, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		# chrony log categories, rendered as the `log` line.
		logging = ["measurements", "statistics", "tracking"]

Conversation

xiaotianlyu commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maherthomsi commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xiaotianlyu commented Sep 2, 2026 •

edited

Loading

maherthomsi commented Oct 1, 2026 •

edited

Loading