Skip to content

feat(ext): introduce Intl support - #2221

Open
Guikingone wants to merge 69 commits into
illegalstudio:mainfrom
Guikingone:feat/intl-managed-icu
Open

Guikingone wants to merge 69 commits into
illegalstudio:mainfrom
Guikingone:feat/intl-managed-icu

Conversation

@Guikingone

@Guikingone Guikingone commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Ready for review — ext/intl parity work. The multi-model review findings are fixed, the branch is updated with current main, and the full CI matrix is green on the updated head.

What this PR adds

  • A managed ICU 78.3 native package (elephc native add icu) built from the pinned release archive, with a C++ shim owning every ICU object and a fixed C ABI into the elephc-intl bridge crate.
  • An AST-only Intl surface (no PHP parsed outside tests): every Intl class is built through synthetic_class, procedural functions through generated declarations, and bridge calls through typed EIR helpers.
  • The php-src ext/intl corpus migrated under tests/php_src/intl/, with a generated Rust harness (tests/codegen/intl_php_src_cases.rs) and per-fixture skip statuses.
  • Intl object classes for Locale, Normalizer, DateFormatter, TimeZone, Calendar/GregorianCalendar, Collator, NumberFormatter, MessageFormatter, ResourceBundle, Transliterator, Spoofchecker, UConverter, ListFormatter, DatePatternGenerator, IntlChar, IntlIterator/IntlPartsIterator, and the BreakIterator family.

Review fixes (multi-model audit)

A five-model audit of the Intl packet produced confirmed findings; this PR now fixes them all, with regressions in tests/codegen/intl_review_fixes.rs (21 tests):

  • ICU handles: Intl wrappers close their bridge handle from __destruct; clone owns a distinct handle, and elephc_intl_live_count() returns to baseline after unset.
  • TOCTOU and overflow: the shim registry stores shared_ptrs, so a concurrent close cannot free an object mid-call; oversized bridge payloads are fetched through elephc_icu_intl_fetch_result instead of being truncated to 1 MiB.
  • UConverter: the substitution-marker scrub only runs for single-byte destinations, so UTF-16 output such as FFFE 1A01 is preserved.
  • IntlDateFormatter::localtime(): reports tm_isdst from inDaylightTime() and no longer dereferences a null SimpleDateFormat; style-only formatters are supported.
  • locale_set_default(''): resets to the process system default like PHP, and ini_set('intl.*') alone links the bridge.
  • IntlIterator: current()/key() are bounds-guarded and the array-backed iterator keeps its deliberate self-cycle (no destructor), so gc_collect_cycles() still reports it; IntlPartsIterator::getRuleStatus() carries the part's ICU rule status instead of a hard-coded zero.
  • NumberFormatter: omitted $type is TYPE_DOUBLE, and ROUNDING_INCREMENT is a double attribute.
  • Normalizer: unpublished forms throw ValueError (getRawDecomposition returns "" like PHP) instead of silently normalizing as NFC.
  • html_entity_decode(): default/ENT_QUOTES flags and UTF-8 are supported; numeric references still fold under ENT_NOQUOTES when the literal contains no quote entity. ENT_COMPAT, HTML5 doctypes, runtime flags, and non-UTF-8 encodings (or quote entities the runtime cannot decode) are refused at compile time.
  • defined(): exact Intl constant membership (including LC_*) instead of prefix matching.
  • General compiler fixes: literal str_replace() search arrays apply in source order (the builtin contract stays string-only, so an unlowerable array search is refused with PHP's message), objects at builtin string parameters are coerced only for native SimpleXML views, strtotime() carries $baseTimestamp, large literal range() stops folding past 256 elements, dynamic callable guards evaluate arguments before throwing and use PHP's messages, and unset($object) only defers the safe point for IntlIterator.

Final parity ledger

cargo test --test codegen_tests test_php_src_intl:

434 passed / 0 failed / 48 ignored (previous: 426/8, 425/9, 424/10, 423/11, 412/22, 393/41, 388/46, 378/56, 377/57, 375/59, 373/61, 367/67, 359/75, 351/83, 323/111).

The 48 ignored cases are exactly what php-src itself does on this platform, verified against reference PHP 8.5.10 + ICU 78.3 on macOS ARM64: 47 upstream SKIP (ICU upper-bound gates, 32-bit-only, Windows-only, x86-only, missing optional extensions) and 1 upstream XFAIL (dateformat_format_variant3.phpt). No applicable fixture is skipped by elephc.

Parity work completed in this branch

  • MessageFormatter: named-argument formatting through ICU's named entry point, pattern argument types with php's "Inconsistent types" diagnostics, numeric-string/DateTime/IntlCalendar date arguments, default-timezone adoption for date subpatterns, php's parse/format error texts, and the OO/procedural prefixes.
  • IntlDateFormatter: TRADITIONAL/GREGORIAN/IntlCalendar calendars, timezone precedence, locale canonicalization and validation, style/calendar/pattern/UTF-8 validation, formatObject error paths, localtime tm arrays, DateTime offset zones, and the procedural aliases.
  • NumberFormatter: all ICU styles through unum_open, pattern styles, compact styles, padding/currency text attributes, parse offsets, TYPE_INT64, php's TYPE_* / type-currency ValueErrors, symbol validation, and serialization refusal.
  • Collator: SORT_REGULAR/STRING/NUMERIC semantics including whitespace-padded numeric strings, stable ordering, attribute validation, locale-length creation errors, and OO/procedural error prefixes. sort()/asort()/sortWithSortKeys() order by a string key while keeping the original Mixed cells, so object values survive (bug75193) and object __toString side effects keep php-src's collator_sort_modify_during_compare behaviour.
  • Locale: compose plural variant/private/extlang arrays, numbered-key handling, lookup tag validation and null-byte diagnostics, locale-length limits, and procedural/global prefixes.
  • ResourceBundle: ures_openDirect fallback semantics, sub-table ownership, key iteration, numeric-string keys, localized error contexts, and the fallback warning state.
  • Calendar/GregorianCalendar: creation/date/time validation and php's deprecation texts, ICU millisecond clearing, uninitialized comparison arguments, exact Gregorian procedural construction, and IntlCalendar toDateTime error wrapping.
  • Grapheme: ICU StringSearch with locale/collation semantics, offset/type ValueErrors, empty-needle rules, negative reverse offsets, grapheme_substr length handling, UTF-8 MAXBYTES/MAXCHARS extraction with next, and collation-aware Levenshtein.
  • BreakIterator: debug info (valid/text/typeid), rule-based construction diagnostics and runtime arity/type checks, __clone handle ownership, and code-point method reachability.
  • UConverter: user subclasses that override toUCallback()/fromUCallback() run a PHP-side UTF-8 <-> ASCII callback path with php-src's exact callback sequence (reset/illegal/unassigned/close), because ICU's C++ hooks cannot re-enter generated PHP. Converters without overrides and every other encoding keep the managed bridge. The callback marker table is keyed by the bridge handle and cleared on destruction.
  • IntlIterator: keeps a self-cycle while alive so a dropped IntlIterator is reclaimed and reported by gc_collect_cycles() exactly like php-src's cycle-management test.
  • Normalizer / ListFormatter: raw-decomposition validation, object-conversion errors, UTF-8/range errors, numeric encoding names, and substitution characters.
  • ICU bridge: locale canonicalization for formatters, per-call error context/prefix handling, intl_get_error_* status formatting, DateTimeZone unknown-id rejection, and a recipe revision bump (lock revision 40) so the shim is rebuilt.

Compiler/runtime support added for the corpus

  • Literal foreach unrolling with per-iteration continue skip flags and distinct source columns, so ReflectionFunction/ReflectionMethod migration loops carry statically-known names; get_extension_funcs("intl") and two-literal range() fold to literal arrays, and literal array facts survive unrelated runtime-state setters (set_error_handler, …).
  • Dynamic static-property receivers: $class::$prop / ($expr)::$prop parse through StaticReceiver::Dynamic, the optimizer folds a literal class-string receiver to Named before type checking, and C::$prop =& $local aliases the static slot in both directions.
  • ReflectionClass::newInstance()/newInstanceArgs() constructs a statically-known class directly (including $subclass::$ARGS tables); abstract targets and any non-public constructor (declared or inherited) take the reflective path and raise PHP's catchable ReflectionException: Access to non-public constructor of class …. Dynamic callable sites evaluate arguments before raising PHP-shaped ArgumentCountError/TypeError errors, with variadic collector parameters excluded from the per-element type guard.
  • gc_collect_cycles() counts unreachable object/callable graph nodes (falling back to total nodes for array-only graphs); unsetting a destructor-less object local only drops a root, so the next explicit collection observes the cycle as PHP does, while classes with __destruct keep the eager safe point that destructor-resurrection paths rely on.
  • The WeakMap prelude types WeakReference::get()'s receiver as mixed, so an unrelated class's get signature (Intl's ResourceBundle::get) cannot pin the local and reject the following private WeakMap call. This keeps method_exists() / get_class_methods() on Intl objects working in programs that also inject the WeakMap prelude.
  • Dynamic callable sites throw PHP's TypeError/ArgumentCountError classes (not a generic Error), resolve $this and unresolved object receivers without false type mismatches, and leave the language-construct clone() to its descriptor path so argument side effects precede the error. Effectful-key/property and root-drop unsets keep unconditional collection; only destructor-less object locals defer to the next explicit collector.
  • Syntactic class constants are mixed until the checker resolves their value, so static defaults like ['x', SomeClass::VALUE] no longer materialize as strings.

Validation

  • Review regressions: cargo test --test codegen_tests codegen::intl_review_fixes — 21 passed / 0 failed on the updated head; the full CI run for 945c0d0b46 is green, and the badargs/all-target unit tests now have explicit 300s nextest budgets.
  • Contract ledgers after merging origin/main: cargo test -p elephc-builtin-contract — 67 passed / 0 failed.
  • Full php-src Intl corpus: 434 passed / 0 failed / 48 upstream skips-XFAIL in CI on 6792ab60be. The harness adaptations restore the fixture INI locale for ini_restore('intl.default_locale') (locale_get_default.phpt) and use a typed-bool helper for normalizer_is_normalized, matching the OO isNormalized results.
  • The focused non-ICU regressions pass on the merged head; the ICU-backed module passed 21/21 on the final artifact locally before the merge.
  • cargo build is warning-free; git diff --check is clean; generated builtin docs/registry and parity/audit gates are green (Errors: 0; 2881 generated pages validated); the PHP coverage page regenerated to 1265 functions / 191 classes / 1420 constants after the merge.
  • libsqlite3-sys is bumped to 0.39.1 for RUSTSEC-2026-0459; cargo test -p elephc-pdo is 140 passed / 0 failed / 2 ignored. The Linux codegen shard cap is raised to 60 minutes (the Intl corpus widened the shards; an aarch64 shard hit the old 40-minute cap).
  • origin/main was merged at 67a956b17d (merge commit 3af94a77be), resolving the builtin-count and generated-doc ledgers for the new eval/reflection contracts.
  • CI provisions the pinned managed ICU package for the macOS, Linux x86_64 and Linux aarch64 codegen shards. The build-archive steps now live in main’s reusable ci-build-host.yml; elephc-intl is in the shared bridge list there and in the release-artifacts action, and the ICU cache/install steps warm the shards’ key once per platform. examples/intl/ carries ICU recipe revision 40.

Notes

  • array_intersect() sparse-key preservation is a separate general gap (elephc-d63l); the Intl case only needs values and is unaffected.
  • General by-reference assignment shapes beyond C::$prop =& $local remain tracked by elephc-4ml.12; the static-property form required by bug62915_2 is implemented here.
  • Dynamic-callable and dynamic-Intl-surface gaps discovered during the audit are tracked on issue General PHP compiler gaps surfaced by the ext/intl corpus #2276.
  • Branch head: 6792ab60be; the PR is mergeable with current main.

@Guikingone Guikingone added the ci:macos Run the macOS test matrix and iOS links on this PR, not only in the merge queue. label Nov 2, 2026
@github-actions github-actions Bot added area:builtins Touches PHP builtin declarations or emitters. area:runtime Touches runtime helpers, GC, ownership, or bridge runtimes. area:types Touches type checking, inference, or compatibility. scope:multi-area Touches more compiler areas than the automatic area-label cap. size:xl Very large pull request that needs deliberate review planning. target:windows-x86_64 Contains behavior specific to the Windows x86_64 target. type:feature Introduces new user-visible behavior or capabilities. labels Nov 2, 2026
@Guikingone Guikingone changed the title feat(ext/intl): managed ICU bridge with AST-only declarations feat(ext): introduce Intl support Nov 2, 2026
@Guikingone Guikingone self-assigned this Nov 2, 2026
@github-actions github-actions Bot added area:codegen Touches target-aware assembly or backend lowering. and removed area:builtins Touches PHP builtin declarations or emitters. labels Nov 3, 2026
@Guikingone
Guikingone force-pushed the feat/intl-managed-icu branch from 6a2f7e0 to 8c6028b Compare November 3, 2026 17:58
@github-actions github-actions Bot added area:builtins Touches PHP builtin declarations or emitters. area:magician Touches eval, include execution, or elephc-magician. target:linux-aarch64 Contains behavior specific to the Linux AArch64 target. and removed area:codegen Touches target-aware assembly or backend lowering. area:runtime Touches runtime helpers, GC, ownership, or bridge runtimes. labels Nov 3, 2026
- Add the pinned `icu` native package (ICU 78.3), the C++ shim that owns the
  ICU objects and error state, and the `elephc-intl` Rust bridge behind the
  fixed C ABI.
- Declare the Intl class family and the object-bearing procedural surface as
  synthetic AST; scalar entry points are registry builtins sharing one typed
  EIR lowering. No PHP source is parsed outside the test corpus.
- Migrate the php-src ext/intl PHPT corpus to inline Rust codegen cases, with
  platform skips preserved.
- Generate the Intl builtin reference, area index, module section, and
  comparison data.
- Build and ship `libelephc_intl.a` in CI, the Docker scripts, the release
  tarball, the Homebrew formula, and the nextest archive.
@Guikingone
Guikingone force-pushed the feat/intl-managed-icu branch from 80871b0 to b511f21 Compare November 3, 2026 19:57
- DateTime splits a trailing zone token, applies the fixed offset, and stores PHP's zone name plus fixed-offset state; getTimezone/getOffset/format preserve it.
- A shared compile-time tzdb abbreviation table feeds both the synthetic splitter and a literal strtotime() fold.
- IntlTimeZone::fromDateTimeZone/IntlCalendar::setTimeZone/fromDateTime reject ICU-unknown abbreviations and too-large offsets with php-src's messages, recording on the calendar object where php does.
- The ICU shim gains calendar __elephc_set_error; the managed recipe moves to revision 32.
- strtotime literal zones use a UTC helper, preserving the fixed offset for non-UTC processes.
- Collator by-reference arrays stay mixed until typed ref storage lands; parity documents the subset.
- IDN rejects non-UTS46 variants and empty domains, and returns false on ICU IDNA errors while filling the details array.
- UConverter::getStandards is a string array so array_unique takes the string path.
- IntlBreakIterator/IntlTimeZone implement php's compare handlers; IntlTimeZone debug info hides unset fields and inherited __debugInfo is dispatched.
- BreakIterator clones copy the code-point state.
IntlTimeZone keeps initialized backing properties so cloning an unconstructed subclass reaches __clone and throws php's Error instead of dereferencing unset slots. Document the trailing tzdb abbreviation/fixed-offset parsing.
…ne errors

- IntlPartsIterator follows the break iterator lazily and syncs its position per part.
- IntlTimeZone::createEnumeration reports php's TypeError for invalid argument shapes.
- Deprecated U_MULTIPLE_DECIMAL_SEPERATORS reads and constant() calls emit php's deprecation.
- str_replace with a literal search array lowers to nested scalar replacements.
- IDN length failures leave the info array empty.
…nstants

new IntlBreakIterator() reaches php's private-constructor Error at runtime, and reads of NumberFormatter::TYPE_CURRENCY (direct or through constant()) emit php's runtime deprecation.
The constructor throws php's generic creation failure, create() returns null with a create-prefixed global error, and msgfmt_create() returns null with the procedural prefix. Invalid locales report the generic creation failure text.
The shim gained calendar object errors, IDN error returns, break-iterator clone state, root-locale handling, and message-formatter creation text; revision 33 invalidates stale native caches.
formatMessage()/parseMessage() report 'Creating message formatter failed' with the caller's zero code, while the constructor/create path reports the generic ICU-code failure.
numfmt_format()/parse() and the NumberFormatter methods reject TYPE_CURRENCY with php's ValueError but no longer duplicate the read-site deprecation.
…rmatter

- Constructor/create/msgfmt_create reset the shared Intl error, then raise php's ArgumentCountError at runtime while accepting the call statically.\n- Passing null emits php's parameter deprecation and coerces to an empty string; arrays raise the runtime TypeError.\n- The constructor throws the detailed creation context, empty patterns fail like php, and invalid UTF-8 patterns report the UTF-16 conversion error.\n- U_UNMATCHED_BRACES reports the generic creation failure while U_PATTERN_SYNTAX_ERROR keeps the parse detail.\n- Bump the managed ICU recipe to 34 so the shim changes invalidate native caches.
# Conflicts:
#	crates/elephc-builtin-contract/src/registry.rs
#	crates/elephc-builtin-contract/src/support.rs
#	docs/php/compatibility.json
#	docs/php/compatibility.md
#	scripts/docs/audit_builtins.py
#	scripts/docs/elephc_builtins/test_contract_pipeline.py
#	tests/builtin_parity_tests.rs
- The direct-callable type guard now accepts a null argument for an object
  parameter, matching the dynamic-call path the optimizer tests rely on; the
  callee's own access/entry guard produces the observable error.
- Apply the user-function type guard only to the Intl synthetic surface. Ordinary
  user functions already emit PHP's standard "Argument #N ($x) must be of type"
  TypeError and must observe argument side effects first.
- Restore main's expectation that a bare null local passed to ?int &$slot is
  rejected; the explicit ?int declaration remains the supported recovery.
@Guikingone

Guikingone commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator Author

@nahime0 Ready for review when you have time

# Conflicts:
#	crates/elephc-builtin-contract/src/registry.rs
#	crates/elephc-builtin-contract/src/support.rs
# Conflicts:
#	docs/php/compatibility.md
#	src/codegen/lower_inst/core_builtins/handlers.rs
#	src/optimize/fold/expr.rs
#	tests/codegen/gmp_math.rs
# Conflicts:
#	.github/workflows/ci.yml
#	.github/workflows/nightly.yml
#	.github/workflows/release.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:builtins Touches PHP builtin declarations or emitters. area:magician Touches eval, include execution, or elephc-magician. area:types Touches type checking, inference, or compatibility. ci:macos Run the macOS test matrix and iOS links on this PR, not only in the merge queue. scope:multi-area Touches more compiler areas than the automatic area-label cap. size:xl Very large pull request that needs deliberate review planning. state: ready for review target:linux-aarch64 Contains behavior specific to the Linux AArch64 target. target:windows-x86_64 Contains behavior specific to the Windows x86_64 target. type:feature Introduces new user-visible behavior or capabilities.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant