Repository navigation
timezone: reject out-of-range offsets and unsupported zones at configure time - #5576
Conversation
NUMERIC_PATTERN was documented as [+-]HH:MM / [+-]HHMM / [+-]HH but written as [+-]\d\d(:?\d\d)?\z, so any two digits passed for the hours and any two for the minutes. validate only ever compares against that pattern, so a mistyped zone was accepted everywhere it is configurable -- timezone, timekey_zone and in_tail's path_timezone. Time.zone_offset then adds the fields instead of reading them, so minutes of 60 or more carry into the hours: "timezone +09:60" is a valid-looking spelling of +10:00, and formatted records are an hour off with a %z that contradicts the time it printed. Anything past +/-23:60 leaves the range Time#localtime can apply, so validation passes, the plugin starts, and every record then fails at format time with "ArgumentError: utc_offset out of range". The pattern now requires HH in 00..23 and MM in 00..59, which is exactly the set of numeric offsets Time can apply: enumerating all 40,000 [+-]HH(:?)MM strings, the 5,760 it accepts (24 hours x 60 minutes x 2 spellings x 2 signs) all resolve to the offset they spell, while the 34,240 it rejects were 3,680 silent hour shifts and 30,560 offsets that raise as soon as they are formatted. test/test_timezone.rb covers validate, validate!, utc_offset and formatter for both lists. Signed-off-by: Qingyuan Wang <162230229@nuaa.edu.cn>
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the fix. The numeric bounds look correct, but some callers do not validate the timezone before using it. On those paths, this change causes invalid offsets to fall back silently or produce nil values instead of raising a configuration error.
I reproduced the following on 7f94b6a and on master (086fa17).
1. <inject> and <extract>
plugin_helper/inject.rb and plugin_helper/extract.rb copy TIME_PARAMETERS, but they never call validate!. For an out-of-range offset, Timezone.formatter and Timezone.utc_offset now return nil. TimeFormatter and TimeParser then fall back to the host timezone.
Conditions: TZ=America/New_York, time_key t and time_type string. <inject> uses time_format %Y-%m-%dT%H:%M:%S%z and event time 0. <extract> uses time_format %Y-%m-%d %H:%M:%S and the record {"t":"1970-01-02 00:00:00"}.
| config | master | this PR |
|---|---|---|
<inject> timezone +09:60 |
1970-01-01T10:00:00+1000 |
1969-12-31T19:00:00-0500 |
<inject> timezone +24:00 |
ArgumentError on every record |
1969-12-31T19:00:00-0500 |
<extract> timezone +0960 |
50400 |
104400 |
For <inject>, +24:00 previously raised an error during formatting. After this change, it silently falls back to the host timezone. For <extract>, the event time itself changes.
2. types with time: in parsers
Parser#build_type_converters uses validate, not validate!. When it returns false, the whole option is used as the time format. So types {"date":"time:+0960:%Y-%m-%d %H:%M:%S"} still configures. For example, "2026-10-26 12:00:00" becomes nil without any warning. On master it was parsed as +10:00.
What I think is needed
- MUST: Reject out-of-range numeric timezones during configuration in both helpers. Calling
Fluent::Timezone.validate!from theirconfiguremethods would address this, asTimeMixin::TimeParameters#configuredoes. - MUST: Reject out-of-range numeric timezones in
typesduring configuration. RaisingConfigErrorinbuild_type_converterswhen the zone part looks like a numeric offset would address this. - MUST: Add tests that go through these helpers and the parser, not only through
Fluent::Timezone.
MUST items block the merge. The SHOULD and NIT inline comments on the boundary wording do not.
The bounds make an out-of-range offset resolve to nil instead of to a
shifted hour, and the three callers that build a formatter or a parser
from a timezone read nil as "no timezone was given": TimeFormatter and
TimeParser fall back to the host timezone, and Parser#build_type_converters
re-reads the whole option as the time format. A typo then changes the
records instead of stopping the config from loading.
Both helpers copy TimeMixin::TIME_PARAMETERS but never called validate!,
so do it there, as TimeParameters#configure does. In types, a value that
starts with a sign followed by a digit can only be an attempted offset --
a strftime format never starts that way -- so raise instead of guessing.
With TZ=America/New_York and event time 0, <inject> timezone +09:60 now
raises where master formatted it as 1970-01-01T10:00:00+1000 and this PR
previously emitted 1969-12-31T19:00:00-0500; +24:00 raises at configure
instead of ArgumentError per record; <extract> timezone +0960 and
types {"date":"time:+0960:%Y-%m-%d %H:%M:%S"} raise instead of yielding
104400 and nil.
The tests go through the helpers and the parser, and keep the valid
spellings (+10:00, +0100, Asia/Tokyo, and a bare time format) working so
the rejection cannot over-reach.
Signed-off-by: Qingyuan Wang <162230229@nuaa.edu.cn>
|
Thanks for the careful review — you're right that tightening the pattern moved the failure from "wrong offset" to "silently ignored" on the paths that never validate. Addressed each point in 34c84a9. MUST 1 — MUST 2 — raise Fluent::ConfigError, "Unsupported timezone '#{timezone}' for key:'#{field_name}'"A MUST 3 — tests through the helpers and the parser. 8 new tests, and each of the four rejection cases fails on the previous head with "exception was expected but none was thrown", so they pin the behaviour you asked for rather than the pattern:
Both inline suggestions are applied verbatim, and the stale "exactly the range Measured on One path I did not touch, to keep this PR to the paths you reported: |
1 similar comment
|
Thanks for the careful review — you're right that tightening the pattern moved the failure from "wrong offset" to "silently ignored" on the paths that never validate. Addressed each point in 34c84a9. MUST 1 — MUST 2 — raise Fluent::ConfigError, "Unsupported timezone '#{timezone}' for key:'#{field_name}'"A MUST 3 — tests through the helpers and the parser. 8 new tests, and each of the four rejection cases fails on the previous head with "exception was expected but none was thrown", so they pin the behaviour you asked for rather than the pattern:
Both inline suggestions are applied verbatim, and the stale "exactly the range Measured on One path I did not touch, to keep this PR to the paths you reported: |
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the update. The <inject>, <extract> and types cases from my previous review now raise ConfigError, and the new tests fail on the previous head. I confirmed both.
One problem remains in the new check in build_type_converters. Details are inline.
MUST items block the merge. The SHOULD and NIT items do not.
We will update the release note on our side. It will mention that <inject> and <extract> now reject any unsupported timezone, such as UTC.
|
We will not backport this to v1.19, to keep compatibility with existing configurations. We will reconsider if a concrete need for a backport comes up. |
…arsers
The guard in build_type_converters raised for any option whose timezone
part starts with a sign and a digit, but a time format may legitimately
start with a signed numeric token, e.g.
`types {"date":"time:+0000 %Y-%m-%d %H:%M:%S"}` parses on master. It also
raised NoMethodError for `types {"date":"time:"}`, where the timezone part
is nil. Match the whole value instead so only a lone numeric token is
rejected, and add regression tests for both cases.
Signed-off-by: Qingyuan Wang <162230229@nuaa.edu.cn>
|
Thanks for the detailed review. All inline items are addressed in f45d7c2:
The comment updates you suggested in the first review ( |
|
The failed tests require #5583 fixes to pass. |
|
Thanks for merging, and for the clear review process! For the record: the windows-11-arm job failures on the final head were the pre-existing test breakage noted in this thread (the fixes land via #5583) — not related to this change. Happy to help if anything comes up around the timezone validation. |
Which issue(s) this PR fixes:
None — self-found while auditing
Fluent::Timezone.What this PR does / why we need it:
NUMERIC_PATTERNinlib/fluent/timezone.rbis documented as[+-]HH:MM / [+-]HHMM / [+-]HH, but it was written as[+-]\d\d(:?\d\d)?\z, which puts no bound on either field.validateonly compares against that pattern, so a mistyped zone is accepted by every setting that takes one:timezone, the parsers'time:…zone spec,timekey_zone, andin_tail'spath_timezone.Time.zone_offsetthen adds the fields instead of reading them, so minutes ≥ 60 carry into the hours:timezone+09:60/+0960+10:00;TimeFormatter.new(..., "+09:60").format(0)→1970-01-01T10:00:00+1000— a%zthat contradicts the time printed next to it+00:75+01:15+99:99/+24:00ArgumentError: utc_offset out of range(Time#localtimeaccepts `So minutes ≥ 60 mean the plugin runs and silently emits hour-shifted data, while hours ≥ 24 turn a two-line regex check into a per-record runtime crash — both shapes come from an offset that is not a time of day.
Red before the fix (the new test file against unpatched
lib):4 tests, 60 assertions, 3 failures—+09:60 should be invalid,<36000> was expected to be nil, and the formatter case.The fix: require
HHin00..23andMMin00..59:The bound is not arbitrary: the largest offset the pattern allows,
±23:59= 86340 s, is inside the|utc_offset| ≤ 86399thatTime#localtimeaccepts, so the accepted set is exactly the set of numeric offsets Ruby can apply.validate,validate!,utc_offsetandformatterall branch on this constant, so an out-of-range offset now takes the existing "invalid timezone" path (validate!→ConfigError: Unsupported timezone '+09:60'); none of them needed touching.Validation (all on one machine,
TZ=UTC, Ruby 3.3.6):test/test_timezone.rb(new file —lib/fluent/timezone.rbhad no test coverage at all)[+-]HH(:?)MMstrings, both spellings, both signs, checked againstTime.zone_offsetandTime#localtimeTwo files in the wider timezone area are red at the base commit in this environment (
plugin/test_out_secondary_file3 failures,plugin_helper/test_inject2 hostname auto-detection failures) and stay identical with the fix reverted — environment-shaped, not regressions.Deliberate behaviour change to flag: a config that today starts and then emits hour-shifted data (or crashes per record) now fails at load with
ConfigError: Unsupported timezone '+09:60'. That is the point — but an operator who was relying on+09:60meaning+10:00sees a startup error instead of wrong timestamps. If maintainers prefer to keep such values working, the alternative is to normalise them, which is what silently corrupting them already does. The new test file is additive; if it fits the repo better elsewhere, the four tests move as-is (they only needrequire 'fluent/timezone').Docs Changes:
None needed — the doc-comment in the module is extended to state the range.
Release Note:
timezone: reject out-of-range offsets and unsupported zones at configure time
+09:60and+24:00, now fail withConfigError. They used to shift timestamps or raiseArgumentErroron each record.<inject>and<extract>now reject unsupported timezones such asUTC,JSTand misspelled region names. UseEtc/UTC,Asia/Tokyoor a numeric offset such as+09:00instead.