Add new EF 11 features - #296
Open
ChrisJollyAU wants to merge 41 commits into
Open
Conversation
Probed ACE (OLE DB 16.0/12.0) at the counter boundary: there is no overflow error. The TDEF high-water (0x14) is a plain signed int32 and the next id is 0x14 + increment computed unchecked, so an ascending counter runs ... 2147483647, -2147483648, -2147483647 ... and a descending one mirrors it. ACE writes the wrapped id to 0x14 and carries on. A wrapped id that is already occupied is an ordinary duplicate-key rejection, and ACE burns the id anyway (0x14 advances despite the failed insert) so the next insert steps over it. LibRed generated the same wrapped id but then wedged: UpdateTdefCounters' monotone guard - the deliberate KB 884185 immunity - read the wrapped value as going backwards and left 0x14 pinned at int.MaxValue, so every later auto insert reissued int.MinValue. The damage was the on-disk 0x14, so ACE opening the file failed the same way. AssignAutoNumbers now returns a per-value flag array of the ids it generated, and UpdateTdefCounters takes a generated id as the new high-water unconditionally: it came from 0x14 + increment, so it is the next value in the sequence by construction, wrap included. The monotone guard now applies only to caller-supplied explicit ids, which is the only case KB 884185 was ever about. Confirmed that guard is still needed: making explicit ids leave 0x14 untouched instead fails 5 tests, including ACE reusing id 1 after LibRed bulk-writes rows 1-3 - explicit ids are how data gets into a counter column, and 0x14 is the only record of where the counter is. Also moves the AssignAutoNumbers doc comment onto the method (it had been orphaned above MaterializeLongValues). Spec: page-02a-tdef.md gains a verified wrap note under 3.1, and the 0x14/0x18 rows plus the appendix entry record that the value wraps. Tests: AceAutoNumberOverflowProbeTest covers both engines at both boundaries, the explicit-int.MaxValue route to the wrap, and the occupied-wrapped-id case. LibRed.Core 481/481, LibRed.Engine 898/898, LibRed.Ado 47/47. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Implemented JetParseTranslator to map .NET Parse methods (e.g., int.Parse, double.Parse) to Jet/Access SQL type conversion functions (CBYTE, CDBL, CINT, CLNG, CDEC). Registered the translator in JetMethodCallTranslatorProvider. Updated related tests to expect Jet/Access SQL syntax for Parse operations instead of SQL Server-style CAST/CONVERT.
Squashes a batch of engine fixes and the test work that found them. Engine and format fixes (LibRed.Core / LibRed.Engine / LibRed.Ado): - Empty Binary index key is the start flag alone (7F asc / 80 desc), not a zero-padded chunk. ACE oracle: ACE writes the index, LibRed reads the stored entry back and re-encodes it. A LibRed-written empty key previously could not compare equal to an ACE-written one in the same index. - Office-Standard EncryptionInfo is parsed from the declared frame (len@0x299, blob at 0x29B) instead of scanning page 0 for a signature. The length is authoritative for ACE itself - a file with key + descriptor but len = 0 is read by Access as plaintext - so the old scan opened databases ACE cannot. Verified by falsification: with the scan restored, LibRed opens such a file successfully. - UPDATE (Flag 4) and DELETE (Flag 5) stored action queries are recognised and reported as not-yet-executable, rather than falling into the generic unsupported bucket. Flags verified against ACE-authored procedures. - Row pointers are bounds-checked before decode, so a corrupt index entry raises InvalidDataException instead of decoding arbitrary bytes. - Commit validates every overlay page against the committed image it was derived from, under a per-file publish gate, so overlapping writers conflict deterministically instead of losing an update. A failed publication restores the already-published prefix and keeps the transaction rollbackable, reporting both the publish failure and any restore failure. - Schema-changing commits advance a per-file catalog generation; other open connections reload their parsed catalog on next access, while plain DML does not force a reload. - Function argument arity is validated against ACE's JES, including Jet quirks (two-argument IIf yields Null). Aggregates go through the same contract. - SQL COMMIT/ROLLBACK reconciles the ADO transaction handle. Page scope is now reader/writer rather than a mutex. A statement that cannot write (SELECT, set operation, system-variable select) takes it shared, so concurrent readers on one file still run together; everything else takes it exclusive for the whole statement, which is what makes a multi-page write atomic to readers. Anything not provably read-only takes the exclusive scope - the shared scope cannot be upgraded and says so rather than deadlocking. Parsing happens before the scope is taken. Test suite: - Wall-clock guards replaced with structural assertions (the planner is asked directly whether the rewrite engaged), and ThrowsAny<Exception> replaced with specific exception types plus message assertions. - Shared AceTestDatabase / TemporaryDatabase helpers, in test/LibRed.Shared so the EF functional projects (which glob test/Shared wholesale and do not all reference LibRed.Core) are unaffected - EFCore.Jet.FunctionalTests had stopped building. - Temp databases are released per test. A database opened and abandoned by a static Fresh() helper kept its file locked for the whole process, so the copy survived every cleanup path; ~22 GB had accumulated in %TEMP%. Handles are now owned by the helper, closed before deletion, and released when the test ends, with a process-exit backstop. Peak temp copies during an engine run: 609 -> 19. - The five ACE-driving classes share one xunit collection. Concurrent ACE use faults natively (SEHException, then 0xC0000005 kills the run): those classes alone crashed 3 of 3 back-to-back runs, the other ~950 tests were clean 3 of 3. Parallelism is not disabled - only those five are serialized, against each other. - New: SchemaVisibilityTests (cross-connection catalog freshness, previously untested), reader/writer scope tests, and a zero-length-descriptor guard. Each was verified to fail with its mechanism disabled. - The Access-output legacy password comparison is restored as a fixture-gated test that skips with a reason (LIBRED_ENCTEST_DIR) instead of silently passing. Docs updated: transactions.md (scope semantics), page-00-database.md (descriptor framing, password fixtures), page-03-04-index-btree.md (empty Binary key), system-catalog.md (action flags), functions.md (arity). LibRed.Core 599, LibRed.Engine 962, LibRed.Ado 47 - all passing, no crashes, zero temp files leaked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…m tables Access 2010 added a second "General" sort order, and the collation version byte selects a genuinely different weight table rather than being metadata: the same values indexed in a v0 and a v1 database differ in all 28 samples. v1's primaries are the Windows NLS (Script Member, Alphabetic Weight) pair verbatim - "apple" is 0E02 0E7E 0E7E 0E48 0E21 - where General Legacy compacts the same ordering into one byte per character. That is why v1 can be derived from a published table while v0 has to be measured. Also records what ACE needs MSysComplexColumns for, which is its own feature rather than part of the collation work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…gaps are for Bytes 0x0B-0x0E (page-0 0x6E-0x71) are ONE Windows LCID with the sort-order version in its otherwise-unused top byte: LANGID at 0x0B, sort id at 0x0D, version at 0x0E. The sort id is what separates a Windows alternate sort order from its base locale - German Phone Book 0x00010407 against German 0x00000407 - and they differ in nothing else. The spec had warned for months that 0x0D was "0 in every file seen, keep an eye on it". The gaps in the v0 letter table are insertion slots for language letters. It steps by +2 everywhere except B-C, Q-R and X-Y, and Spanish lands on exactly the free value in each relevant gap. The second byte is a SUB-POSITION ordering letters that share a slot, proved by locales that put several in one. Version 1 is not General-only: Croatian and Romanian ship in both generations. Five of DAO's collating orders are dead metadata, byte-identical to General, so appearing in the UI does not imply an implementation. DAO can author a locale but not a sort-order version, which is what fixtures need Access itself for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JetLocaleTailoring keys per-locale overrides by STRING rather than character, so a contraction works: a digraph weighing as one letter is the inverse of the existing expansions, and the one primitive neither encoder had. Matching is greedy longest-first with no backtracking, and looks up the ORIGINAL text before the uppercased text, which is what lets Turkish disagree with invariant casing. Only Hungarian doubles, and that test must run before the longest match. Tailoring is not only insertion. Six devices, all inside the existing framing: insertion, contraction, expansion, secondary retune, remapping the base table, and reordering. Empty tailorings mean "measured to need no change", which is different from having none. Also fills in the General diacritic table and fixes the long s, which is a letter of its own rather than a fold onto s - found by testing each locale against a set far wider than its own tailoring. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…venance v0 IS the NT4-era NLS order, renumbered into one byte, and the compaction is ORDER-PRESERVING. Sorting every character by the Windows NT 4.0 - Server 2003 table's (SM, AW) primary and checking v0's bytes come out non-decreasing keeps 507 of 510 strictly-ordered pairs and 947 of 955 ties, with 12 of 14 blocks perfect. So the +2 stride, the language-letter insertion gaps and the 0x79 page are one decision rather than three observations. Jet also NARROWED it: 88 of the 552 v0 ignorables are weighted by NLS and dropped anyway, an editorial call no published table would reveal. Locales SHARE the block tables, with per-locale deltas in their own tailoring, which is what makes 21 orders cost 27 entries between them. A ligature weighs as its DECOMPOSITION - there is no ligature mechanism in the format. Components are weighed individually and never re-enter the contraction matcher, and decomposition sits below the tailoring because some locales do not decompose at all. The last gap was the word-sort ignorables, 20 of them rather than 3: every dash, the Arabic harakat, and fullwidth apostrophe and hyphen, which share their ASCII counterparts codes exactly - the one place fullwidth really does collapse onto ASCII, unlike the letters. Coverage is 2147/2147 for all 23 orders. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Prefix compression covers the WHOLE entry including its trailer, not just the key bytes. Reading it as key-only worked until a page held many equal keys, which is exactly what a full-BMP sweep produces - thousands of ignorable characters all encoding to the same empty key - and then the reader rejected the page outright. Found by the probes added here rather than by a test written for it, which is the argument for sweeping a whole range instead of sampling it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every character ACE stores a key for, LibRed now encodes identically - 63,422 of them. No published table describes v0, since its primaries are a Jet compaction rather than the NLS weights, so ACE ITSELF is the source: SortKeyTableV0GeneratorTest inserts every code point into an indexed text column, reads the stored keys back and writes the embedded resource (74 KB; 63,105 weights, 40 word-sort ignorables, 276 kana). Far past anything hand-maintainable, and hand-transcribing hex is exactly the work that introduces a wrong byte nobody notices. Two things only a full sweep shows: ACE weighs every CJK ideograph and the entire private-use area, and across all 65,536 code points it refused exactly one. KANA take a two-byte primary 7F <sound>, with voicing as an ordinary secondary and the small/normal distinction bit-packed into a section of its own - three per byte, two bits each, most significant first, under a 10 marker. Two rules only multi-character strings reveal: the halfwidth voicing marks are COMBINING (alone they look ignorable, which is what hid it), and the inline section's introducer becomes FF 01 when a kana section precedes it. The prolonged sound mark lengthens the preceding kana's VOWEL, which is what the character means - がー is "ga" lengthened by "a", not by "ga" - so the vowel is a property of each kana and has to be measured per character rather than derived. Also: inline positions count primary WEIGHTS, not bytes, and the hand-verified expansions stay ahead of the measured table because a key cannot show whether two bytes are one weight or two. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v1 now encodes all 63,422 BMP characters exactly as ACE stores them, matching v0. It was 29,776 differences when the sweep started. Most of it was four structural rules rather than missing weights, which is why the sweep was worth running before generating anything: a Han character takes a FOUR-byte primary (the marker FD FF then its own weights) and no secondary, which alone was 28,200 wrong keys; Hangul jamo take (AW, DW) as the primary with no marker; a zero alphabetic weight means NO primary at all; and where something precedes it, such a weight FOLDS into the one before rather than taking a slot. The rest is that the published Server 2008 table is not quite what ACE carries. That identification came from 25 reconstructed keys, all Latin and symbols, and it holds for 57,793 characters and fails for 501 - Balinese and Canadian syllabics get Latin weights, and the Arabic harakat and several ligature blocks differ. Scripts added or reweighted since. Rather than hunt for the right NLS revision, the disagreements are measured and embedded (2.0 KB), along with 5,082 characters ACE treats as wholly ignorable that the published file has no entry for at all. An override stores raw primary and secondary bytes, not (SM, AW, DW) weights: that reading assumes a two-byte primary carrying one secondary, and ACE breaks it both ways - the harakat have a secondary and no primary, the Lao vowels take a one-byte primary. A primary byte can also BE 0x01, so the section delimiter is the last 0x01 in a key rather than the first. Splitting at the first made five characters look like an unknown mechanism; measuring them in combination showed ordinary two-weight expansions. Kana turn out to be shared: same sound weights, same section, byte for byte under both orders, so JetKanaSection is extracted rather than duplicated. Two narrow differences remain - v1 weighs a compatibility form by its base kana's sound where v0 gives it its own, and five kana absent from the v0 table come from v1's own script member 3, whose smallness cannot be inferred from reaching that path. A generator must run with its own resource suppressed. It records where the encoder DISAGREES with ACE, so measuring an encoder that already consults it would find no disagreements and write an empty file. Behaviour change: an unassigned character such as U+0378 no longer throws. ACE stores an empty key for it, so refusing would reject a value the engine accepts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ACE stores an index entry of at most 510 bytes as built. At exactly 510 it comes back byte-for-byte; a value needing 511 comes back as 510 with the weights cut short and the last two bytes replaced by a value that varies with the string - a truncated key plus a checksum, which is why two long values never collide. The checksum function is not known, so LibRed cannot reproduce a truncated key and refuses the value instead. It was writing the full-length key, which put bytes in the index ACE would never write, and a wrong index key is silent: ACE writes its own into the same index and a seek misses rows. The cap is on the whole ENTRY, not per column. Two 200-character text columns weigh about 404 bytes of key each - comfortably under the cap individually - and ACE stores their combined entry hashed at 510. A per-column check would have let that through. Because it limits weights and not characters, what it buys varies with collation and script, which is the practical cost of General over General Legacy and is invisible in the schema: 255 characters for v0 Latin (the column limit is reached first), 253 for v1 Latin, 169 for v1 accented, 127 for v1 Han. Above the BMP the two orders disagree completely, measured over all of planes 1 and 2 and sampled across all sixteen. v0 ignores astral characters entirely - every one gets the empty key - so under General Legacy an astral character is invisible to the index. v1 weighs BOTH surrogate halves, each looked up like any other character: U+10000 is 7F B002 B4F8 01 3F 3F 00. Only the high surrogates to U+D87F carry weights, so from plane 3 upward the high half is ignorable and the low one stands alone, and those planes collapse onto 1,024 keys. The fix v1 needed was therefore narrow: an unweighted surrogate is ignorable rather than an error. The tempting reading of the plane-3 samples - "the high surrogate contributes nothing" - is wrong, and skipping every high surrogate breaks all 131,068 characters of planes 1 and 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… plus a byte An apostrophe or hyphen carries no primary weight and instead appends a record holding the position it sat at. That position is SIXTEEN bits, big-endian, with bit 15 set - [MS-UCODEREF] gives SpecialWeightType as (Position: 16 bit integer, ScriptMember, PrimaryWeight), emitted as "Byte1 = Position >> 8, Byte2 = Position & 0xff". The 0x80 is not a marker byte at all. LibRed read it as a marker followed by one position byte and truncated the rest. The two readings agree below 0x100 and diverge above it, and the offset 0x07 + 4 x position passes 0xFF at position 62 - so a hyphen at character 63 is 81 03 where LibRed wrote 80 03, and at 250 it is 83 EF where LibRed wrote 80 EF. Every indexed value with an apostrophe or hyphen past character 62 therefore got a wrong key. A hyphenated name in a 255-character column is enough, and nothing caught it: single characters encode correctly, short strings encode correctly, and the field only overflows when a value is long enough. It surfaced while reverse-engineering something unrelated. The lesson is in the spec beside it - measure combinations, not only characters, because a per-character sweep can be exhaustive over all 63,422 and still miss a whole class of bug. Measured against ACE across positions 10 to 250 under both sort orders. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…values Past 510 bytes ACE keeps the first 508 and replaces the rest with two bytes derived from what it dropped, which is why two long values sharing a prefix still sort apart. LibRed refused those values because the function was unknown. It is known now, and they are truncated exactly as ACE truncates them. Recovered by measurement rather than found documented. Three tails differing in one byte show the function is affine over GF(2) - L(0xA3) ^ L(0x13) = L(0xB0) exactly - and 173 observations show it is shift-invariant, so a byte at distance d from the end contributes S^(d-1) of itself whatever the length. Sweeping all 65,536 polynomials in five framings found NOTHING, and that negative was the clue. The standard reflected update is crc = (crc >> 8) ^ T[(crc ^ b) & 0xFF], passing the byte through the table; ACE computes crc = (crc >> 8) ^ T[crc & 0xFF] ^ b and injects it raw. Wrong injection point, so no polynomial could ever have matched. The step operator then came out of Gaussian elimination over the measured contributions and predicts all 657 of them - no name for the algorithm required. There is no initial value and no final XOR. The limit is on the whole ENTRY, not per column, which the measurement also settled: two 200-character text columns weigh about 404 bytes of key each, comfortably under the cap individually, and ACE stores their combined entry truncated. A per-column check would have let that through. Still refused where the dropped bytes hold an inline word-sort record. That cannot be verified even in principle - the record sits in the part ACE discarded, so what it contained is unobservable, and if ACE recomputes its position when truncating then the checksum's input is not what is reconstructed here. Guessing would write a silently wrong key. Text columns now index to the full 255 characters again rather than being refused past 127 to 253, depending on collation and script. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
French tailors no letter at all. It is General with the diacritic section written BACKWARDS, so accents are weighed from the end of the word and cote < cote-with-acute-on-e ... in short, coté sorts before côte where General has it the other way round. [MS-UCODEREF] names the flag IsReverseDW and gives both halves of the rule: the run of default diacritics comes off the LEFT rather than the right, and what remains is written right to left. Verified against ACE byte for byte. côté is [02 12 02 0E], trimmed to [12 02 0E] and stored as 0E 02 12. Across all of Latin-1 and Latin Extended-A with accents doubled and tripled per string, 1,289 values, zero differences. It had been recorded as "unclassified, secondary-section tailoring", which described the symptom rather than the rule, and the reason is worth keeping: a word with ONE accent encodes identically under both orders, and the sample set that measured every locale against General contained no two-accent word. The rule was invisible to the measurement rather than absent from it - the same blind spot as the inline position field. LibRed can now also CREATE a French database, which follows for free: creating one requires encoding the order, because the system-table indexes are built on the way. That circularity is why measuring French needed DAO to author the file first. ACE indexes into a LibRed-created French database with every key identical to LibRed's own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The script members were derived by measuring ACE, one class at a time, and the GetWindowsSortKey pseudocode names every one of them: PUNCTUATION 6 is the word-sort class, JAMO_SPECIAL 4 the Hangul jamo, EXTENSION_A 5 the Han shape, NONSPACE_MARK 1 the "no primary, only a secondary" rule. Everything at or below MAX_SPECIAL_CASE goes to its SpecialCaseHandler - which is exactly the set of classes that needed bespoke handling here. Script member 3 is EASTASIA_SPECIAL, not "kana", so the constant is renamed. That also turns an unexplained list into one rule: the class reserves PW_REPEAT 0 and PW_CHO_ON 1, and the seven characters ACE gives the unweighted FF FF primary are exactly those - the iteration marks and the lone prolonged sound mark. The 01 01 01 before a word-sort record is not an introducer but three SECTION SEPARATORS. The frame is primaries 01 diacritics 01 case 01 extra 01 specials 00, and Access emits it with the case-weight section EMPTY - which is the mechanism behind case and width folding, since width is bit 0 of the Case Weight. MIN_DW = 2 is the default secondary whose trailing run gets trimmed. Three further notes recorded in the spec. The contraction limit corroborates v0's provenance independently - 2 and 3 characters on NT4 through Server 2003, 4 to 8 from Vista, and every v0 tailoring here tops out at three, which is the same generation the weight-table comparison identified by a different route. The FD FF Han primary is NOT the Windows 7 three-byte weight, which is three bytes and postdates the table Access froze. And Access PACKS the East Asia extra weights three flags to a byte where Windows uses one byte per character. Nothing in that source covers the 510-byte cap, truncation or the checksum: a useful negative, since it means those are Jet inventions that had to be measured rather than looked up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These three were refused on the grounds that "the v1 encoder has no tailoring
hook: its primaries are 2-byte NLS values, a different shape". That was a fact
about the encoder, not about the orders. Measured, each is General v1 plus a
small override set using the same six devices every version-0 locale uses, and
all three are byte-identical to each other - the same order under three LCIDs,
so one table serves them.
The letters land where the Croatian alphabet puts them: L 0E48, LJ 0E4A, M 0E51;
D 0E1A, DZ-digraph 0E1D, D-with-stroke 0E1E. So the three digraphs are
contractions, which is why the loop had to become indexed. TailoredWeight
already carried byte[] primaries, so a two-byte primary needed no change at all.
Two things the conformance range had to be WIDENED to find, and both would have
written silently wrong keys for ordinary Croatian text:
- The caron retune reaches further than a hand-picked list of letters showed.
Eight more, and one of them moves its PRIMARY rather than only its accent.
- Expansion components were going straight to the base table, so the
precomposed digraph U+01C4 encoded as D + Z-with-caron instead of D + the
tailored Z-with-caron. Components take the LOCALE's letters - the rule
version 0 already followed - but must not re-enter the contraction matcher,
or expanding a ligature could trip a digraph the original text never had.
Version-1 fixtures were asserted over 447 values where version-0 ones got 2,444,
because the extended blocks were once measured for v0 only. That is no longer
true, and the narrowing only hid ground: it is removed, and all 27 fixtures now
run the same 2,444 values with zero mismatches.
One genuine ACE asymmetry recorded: U+016C takes the retuned secondary while
lowercase U+016D keeps General's, identically in all three locales. Every other
letter is case-symmetric, all three digraphs included.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n 0) General v0 gave a secondary-only character a slot of its own in the secondary section. ACE folds it into the preceding weight instead, adding its value: Thai ไก่ is two weights with secondaries 03 06 - the tone mark's 03 added to the consonant's 03 - not three weights. Emitting a slot desynchronises the whole section from the primaries, so everything after the mark is wrong too. This affects ALL 28 orders, not one. The class is "secondary-only" characters, and it holds Thai tone marks, every Hebrew niqqud, the Cyrillic combining marks and three Greek ones - any indexed text with a combining mark following a base character. The version-1 encoder has had this rule since the full-BMP work; version 0 never got it. It surfaced only when the Thai block entered the conformance range, because the rule needs a mark AFTER a base character and a per-character sweep cannot place one there. Nor could comparing one locale against another: both were wrong identically, so only comparing against ACE shows it. The conformance range gains the Thai block and words, which is what caught it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Thai writes the five vowels e/ae/o/ai/ai-maimuan BEFORE the consonant they are pronounced after, and collation follows speech. That was recorded as needing REORDERING, a device nothing else here uses, and it is why Thai stayed unimplemented as the last non-CJK order. Measurement says it is an ordinary contraction - the same device Croatian's lj uses. ACE gives the pair a SINGLE weight at the consonant's own primary plus a vowel offset: the pair with ko-kai is 7C99 where the consonant alone is 7C98, the next vowel gives 7C9A, and so on to +5. Every consonant sits on a six-wide block, itself plus a slot per leading vowel. And it is a contraction rather than a swap, because the reverse order does NOT collide: consonant-then-vowel stays two weights, 7C98 7C93. Built as the rule rather than 220 transcribed entries, with each consonant's primary read from the measured v0 table, so there is no hand-copied hex. Also re-verifies the five DAO-only orders that are recorded as inert. They were established over 31 samples of single characters, and French proved that shape of evidence can hide an entire rule - it tailors no letter at all, so a word with ONE accent looks identical to General and only two reveal it. Now 82 samples including words carrying two marks per script, and the Greek triple that is the direct analogue of the French one. All five are still inert, and the probe's positive controls still show their departures, so the null result means the orders are inert rather than the harness being dead. Every non-CJK sort order is now implemented: 28 of them, each verified against ACE over 2,559 values. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI runs LibRed.Engine.Tests on five platforms - Linux, Windows, macOS and both ARM legs - because LibRed is fully managed, and that run is what actually proves the cross-platform claim. Twenty-four tests in there open an ACE OLE DB connection, so they fail on four of the five, and for a reason that says nothing about the code: the driver is simply absent. They move to a new LibRed.Engine.AccessTests, which runs under the LibRedAccess job beside LibRed.Core.Tests, where the other comparisons against the real engine already live. They genuinely need the engine - QueryEngine, SQL, the lot - so LibRed.Core.Tests could not host them without inverting the layering, and LibRed.Ado.Tests would have meant filing engine tests under the ADO layer and introducing an ACE dependency to a project that has none. The five classes were already marked [Collection(AceCollection.Name)], so they identified themselves and the split needed no judgement about which were which. LibRed.Engine.Tests now has no System.Data.OleDb reference and no AceTestDatabase, so this cannot drift back: an ACE test added there does not compile rather than failing in CI on four platforms. 938 + 24 = 962, the count before the split. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LibRed synthesises a new .accdb page by page rather than copying a packaged empty file, so the collating order is a parameter of creation rather than a property of a template. That had been demonstrated for the two General orders and French; the other 27 configurations were an inference from sharing a code path. Now measured: 30 configurations - both General orders and every locale in JetLocaleTailoring, including the two orders that exist at a second sort id (German Phone Book, Hungarian Technical). For each, LibRed creates the file, ACE creates a table and index INSIDE it, and every key ACE writes matches LibRed's own. Nothing disagreed. The bar is deliberately that rather than "the file opens". Two engines agreeing on a shared index is the only check that catches a wrong key, because a disagreement does not error - it makes seeks miss rows. The list comes from asking IsIndexKeyEncodable rather than from a hardcoded set, so it cannot drift out of step with JetLocaleTailoring and a new order is covered the moment it lands. It also guards the entanglement between the two: the system-table indexes are built during creation, in the database's own order, so creating a database REQUIRES encoding its collation. A locale with wrong weights would not merely sort wrongly - it would make creation itself produce a file ACE disagrees with. That circularity is why measuring a new locale for the first time needs DAO to author the file. Also corrects the CreateEmpty doc, which still said only the two General orders could be encoded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Access has two append forms. The single-record one takes VALUES and was already
supported; the multiple-record one takes a query and was not, which is the only
way to append more than one row in a statement - Jet has no multi-row VALUES
syntax at all, so "many rows" and "from a query" are the same feature there.
INSERT INTO target [(field, ...)] SELECT [source.]field, ... FROM tableexpression
The IN externaldatabase clause both forms allow is deliberately left out:
appending into another file belongs to the linked-database subsystem LibRed does
not have, and a half-implementation would be worse than none.
Two behaviours were measured against ACE rather than reasoned, and one of them
caught a bug that all nine of my own tests had agreed with:
WITHOUT a column list, ACE resolves the source's output NAMES against the target
- not positionally, which is what this first implemented from the plausible
premise that only the count matters. The case that separates them is reversed
aliases: SELECT B AS Name, A AS Id emits ('seven', 7) in that order, and ACE
stores Id=7. Positionally it would have stored 'seven' in Id. ACE also rejects a
name the target lacks, SELECT * included, and LibRed now gives the same error.
WITH a column list the other rule applies: the list names the targets and values
map positionally onto it, whatever the source calls them.
Appending a table to ITSELF terminates. The source is materialised before a row
is written, or the scan consumes its own output forever; ACE doubles the table
and stops, and so does this.
The failure mode of getting the first one wrong is the silent kind - into
type-compatible columns it succeeds and puts the values in the wrong fields -
which is the argument for cross-checking rather than trusting green tests that
encode the same assumption as the code.
No cross-check exists for column DEFAULTs on the ACE side: its DDL rejects
DEFAULT in CREATE TABLE ("Syntax error in field definition"), a column property
Access sets through DAO/ADOX instead. That is a limit on what can be compared,
not a place the engines differ, and it is recorded in the test file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SELECT field1[, field2[, ...]] INTO newtable [IN externaldatabase] FROM source
A make-table looks like a SELECT and is an action query: it writes a table and
returns no rows. It runs with the writing statements, inside the implicit
transaction, so a failure part-way cannot leave a half-populated table.
Measured from ACE before implementing, because most of it could reasonably have
gone the other way:
- A make-table copies DATA AND COLUMN DEFINITIONS ONLY. The source's PRIMARY
KEY and its indexes are NOT copied, so archiving a keyed table gives an
unkeyed copy.
- A result column that IS a source column keeps that column's definition,
width included: a source Text(30) arrives as Text(30). A COMPUTED column has
no declared width to copy and gets Text at the 255 maximum.
- An empty result still creates the table.
- An existing target is an error - the docs call it "a trappable error" and
ACE says "Table 'X' already exists".
The width rule cost a round trip worth recording. The probe output showed both
cases - a named column at Text(60) and a concatenation at Text(510), two lines
apart - and this generalised from the second while looking at the first. So the
cross-check against ACE caught a misreading of a measurement, not just an
unmeasured guess.
Two routing points the page cache and the tests found rather than review:
ExecuteQuery reaches ExecuteQueryCore without passing Route, so a make-table
invoked that way returned the source's rows; and Scoped classified any
SelectStatement as read-only, so the table write was refused by the guard that
rejects a write in a shared scope rather than silently upgrading it. Both now
account for INTO.
IN externaldatabase is not implemented, as on INSERT: creating a table in
another file belongs to the linked-database subsystem LibRed does not have.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MigrationsInfrastructureLibRedTest never completed: the class hung, and a killed test host kept a handle on the .accdb so the next run could not delete the file either. Diagnosed by sampling the stuck process — every thread was parked in Thread.Sleep inside AcquireDatabaseLock — and then by instrumenting the loop, which showed it spinning on a returned 0 and never on an exception. It was not a deadlock. The retry delay started at 1s and doubled while under a minute, so after seven misses every contender retried once per 64 seconds; the delay never reset, and there was no jitter, so the threads woke in lockstep and exactly one won per round. Fifteen threads therefore took ~16 minutes, which is indistinguishable from a hang. Having no timeout, the loop could not fail, only wait, so nothing was ever logged. - Retry policy: 50ms start, 1s cap, +/-25% jitter, and a one minute deadline that throws a TimeoutException naming the lock table and how to clear it. Build the lock object on success rather than allocating and discarding one on every attempt. - AcquireDatabaseLockAsync was missing both guards its synchronous twin has: no catch around the racy lock-table CREATE, none around the insert's duplicate-key race. Mirrored. Fixing the wait exposed a second defect it had been hiding. Concurrent migrators all pass the non-atomic exists-then-create check and all issue CREATE TABLE; EF catches the losers as DbException, but LibRed threw InvalidOperationException, which escaped that guard and failed the migration outright. ACE raises OleDbException there, so translating is what makes LibRed behave like the engine it stands in for. - New SchemaObjectExistsException, deriving from InvalidOperationException as ConstraintViolationException does, thrown from the four DDL name collisions: CREATE TABLE, CREATE VIEW/PROCEDURE, SELECT INTO and ALTER TABLE RENAME. LibRedCommand translates it into LibRedException with ObjectAlreadyExists (2714). - Assertions on those paths now name the exact type, since Assert.Throws does not accept a derived one. - New tests pin the lock contract that had none: the statement itself through the engine, the acquire/release cycle through ADO, and N connections contending, all bounded so a regression fails instead of hanging. MigrationsInfrastructureLibRedTest goes from hanging to 34/36 in 84s. The two remaining failures assert SQL Server baselines (sp_getapplock, CREATE DATABASE, brackets) and were never ported. Engine 961/961, Ado 55/55, Core 789/789, Engine-ACE 32/32. JetHistoryRepository is shared, so the ACE path gets the same retry policy and async guards; that has not been exercised against a real driver. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
migrationBuilder.Sql("--Before") sends a command whose entire text is a
comment. The lexer skips WS/LINE_COMMENT/BLOCK_COMMENT outright, so that
text produces no tokens, and the statement rule has no empty production —
so it failed with "mismatched input '<EOF>' expecting {SELECT, IF, ...}".
Any user migration carrying a comment-only command hit this.
A comment is not a token, so this is not a missing statement kind and gets
no AST node: text with no tokens simply has no statement to run. The check
is answered by the lexer rather than by scanning for '--', because in
SELECT '--' the dashes belong to a string literal and a textual strip
would reduce a real statement to nothing.
- ISqlParser.IsStatementless, implemented by pulling one token and asking
whether it is already EOF.
- QueryEngine.ExecuteQuery/Execute short-circuit to an empty result. They
take no page scope: there is no work to isolate. Execute reports zero
rows affected rather than a query's -1, a comment being an action that
did nothing rather than a result set.
- LibRedCommand.ExecuteBatch skips such a fragment rather than running it.
ExecuteBatch returns the LAST statement's result, so without this
"INSERT ...; -- done" would report the comment's zero rows in place of
the insert's, and take @@rowcount with it.
Found in MigrationsInfrastructureLibRedTest, which died on the comment
before reaching anything it was testing. That test still fails, now on the
next command: its migration body is unported T-SQL (IF OBJECT_ID, THROW
65536, brackets), as is its baseline, and it fails on ACE too — it is
absent from the Jet green list.
Engine 970/970, Ado 59/59, Core 789/789, Engine-ACE 32/32.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reading MSysAccessStorage from the Northwind ACCDB failed outright: InvalidDataException: A relocation source must contain exactly one 4-byte pointer; found 55 bytes RowRelocationReader required a relocation slot to be exactly 4 bytes wide. That table has live overflow slots of 45-63 bytes, and the pointer is sitting in the leading 4 bytes of each one. The remainder is the row as it was BEFORE it moved, with only those 4 bytes overwritten. Discount them and every field lands where the row format puts it (the pointer covers the 2-byte column count plus the first 2 bytes of the first column, leaving that column's remaining 6 bytes at offset 4); the remnant's Id/ParentId/Type/Name equal those of the row it forwards to; the remnant's null bitmap differs from its target's in exactly one bit, the OLE column Lv, whose arrival grew the row and forced the move; and every remnant is shorter than its target. The slot kept the old row's width instead of being trimmed. So the width was never the contract - the leading pointer is. Require at least 4 bytes and read sourceBytes[..4]. The checks that actually validate a relocation are untouched and unchanged: the target must be in the file, owned by the same TDEF, and a nonempty hidden inline row. What writes the wide form is NOT known, and is deliberately not claimed in either the comment or the spec. Every writer reachable from code trims to exactly 4 - measured across 317 relocations with no exception, covering ACE on x64, the ACE 2010 runtime on x86, and LibRed's own writer, under growing and shrinking text, repeated re-relocation, page fragmentation by interleaved deletes, and an OLE column going from NULL to a value. Access's own maintenance of its system tables is not reachable through SQL DML and remains unexercised; the spec records that as the open avenue rather than guessing. Because no write path produces the shape, the wide case is covered by handing the resolver a wider source span directly rather than manufacturing it on disk. A companion test pins our own writer still trimming, so if that ever changes the spec's claim fails loudly instead of going stale quietly. page-01-data-and-rows.md said "contains exactly one 4-byte pointer" and "validates the exact source width". Both corrected, with the evidence. Core 793/793, Engine 970/970, Ado 59/59, Engine-ACE 32/32. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JetByteArrayTypeMapping built its literal by appending "0x" and then each
byte as hex, so an empty array produced the bare string "0x". That is SQL
Server's empty-varbinary literal and it is not valid Access SQL: ACE answers
"Syntax error in query expression '0x'". The SQL was therefore wrong for both
providers - it failed on ACE exactly as it failed on LibRed, whose parser was
right to reject it (HEX_LITERAL already requires at least one digit, matching
ACE).
Verified against ACE, an empty STRING is the literal that works, in a value
position and in a DEFAULT: '' stores and reads back as a zero-length byte[],
and stays distinct from NULL (IS NULL does not match it). 0x00 is NOT the
same thing - that is a one-byte zero.
The parity half is the larger part. LibRed's codec cast straight to byte[],
so any string reaching a binary column threw InvalidCastException - not only
the empty one. Fixing just the empty case would have left 'A' broken.
ACE's rule, measured for VARBINARY and LONGBINARY alike, is that a string in
a binary column stores its UTF-16LE bytes:
'' -> byte[0]
'A' -> 4100
'AB' -> 41004200
'e' -> E900 (U+00E9)
'41' -> 34003100 the digits '4','1', NOT the byte 0x41
The empty case falls out of that rather than being special. JetTypeCodec now
routes Binary and Ole through AsBinary, which encodes a string that way and
passes a byte[] through untouched - the same UTF-16 treatment Memo text
already had.
Tests cover the rule, not just the fix: empty stores zero-length, empty is
not NULL, 0x00 is a one-byte zero, the four UTF-16 cases, hex still round
trips, and a digitless 0x is still rejected as ACE rejects it.
Spec check (LibRed.Core type-codec change): no docs/format update. The stored
representation is unchanged - raw bytes either way; what changed is which
input types the codec accepts. ACE treating binary as a Unicode string
throughout its SQL surface is worth documenting on its own terms once
explored (Len/LenB, comparison, concatenation), not as a footnote here.
Engine 979/979, Core 793/793, Ado 59/59, Engine-ACE 32/32.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A binary column has two faces in Access SQL, and LibRed implemented only one.
Measured against ACE:
=, <, >, ORDER BY BYTES - case-sensitive, byte order
LIKE, Len, &, TypeName a UTF-16 STRING, and LIKE carries the text
collation with it
so `B = 0x4100` matches only 'A', while `B LIKE 'A%'` matches 'A' (0x4100)
AND 'a' (0x6100) - the same column, compared case-sensitively by '=' and
case-insensitively by LIKE.
LibRed had the byte half exactly right already. The text half called
ToString() on the byte[], so it operated on the literal string
"System.Byte[]":
Len(B) 13 for every value ("System.Byte[]".Length) -> now 1
B & 'x' "System.Byte[]x" -> now "Ax"
B LIKE 'A%' 0 rows -> now 3
TypeName(B) "Byte[]" -> now "String"
Silent nonsense rather than an error, which is worse than the
InvalidCastException fixed in d2264a0 on the write side.
The helper was already there: ToText() reinterprets a binary value as UTF-16
and was written for the byte functions (LenB, MidB, InStrB - which is why
LenB was correct all along). This just routes the text functions, both concat
operators, LIKE and TypeName through it as well.
TypeName now reports String for a binary column rather than the CLR type
LibRed actually holds. That is deliberate: ACE's expression service sees a
VT_BSTR there, and VarType already returned 8.
Verified end to end against ACE - Len/LenB 1/2 and 0/0 for empty, = 0x4100
matching one row, ORDER BY putting 0x4200 before 0x6100, LIKE 'A%'/'a%'/'A_'
matching 3/3/1, 0x0102 & 'x' giving "ȁx", TypeName String and VarType 8.
Every one of those is now a test.
Engine 989/989, Core 793/793, Ado 59/59, Engine-ACE 32/32.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ByteArrayTranslations.Any failed on both providers with "The LINQ expression ... .Any()' could not be translated", while its three siblings - Length, First and Index - fail with the provider's deliberate refusal: Returning the exact length of a byte array is not supported by Jet ... There is support for a 'EF.Functions.ByteArrayLength' method ... Those three are correct. LENB reports the UTF-16 byte count, so it rounds an odd length UP to even and an exact length is genuinely unobtainable; ByteArrayLength offers the LENB / ASCB(RIGHTB(x,1)) workaround and documents precisely when it is wrong (data ending in 0x00, where a real even-length value is indistinguishable from a zero-padded odd-length one). Refusing beats returning a wrong number. Any is different: it asks only whether there are any bytes at all, and that question the rounding cannot spoil. An empty array is 0 and every non-empty array is at least 2, so LENB(x) > 0 is exact - the trailing-0x00 ambiguity cannot arise for a > 0 test, and unlike ByteArrayLength this needs no caveat. Both baselines were still SQL Server's - DATALENGTH([b].[ByteArray]) > 0, square brackets and all - i.e. copied and never ported, which is why nobody noticed the translation was missing. Now: WHERE LENB(`b`.`ByteArray`) > 0 Verified on both providers, the Jet one against a real ACE driver: 9 tests, 6 passing where 5 passed before, and the same three by-design refusals. A full functional run was not done; the new case is guarded on a parameterless Any over byte[], so nothing else can reach it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three Enumerable cases were three sequential `if` blocks, each re-deriving
the same guard in a slightly different way - one used
`method.GetParameters().Length == 1`, another indexed `arguments[0]`/
`arguments[1]` directly, and none checked the declaring type. Upstream's
SqlServer translator states it once and switches:
if (method.IsGenericMethod && method.DeclaringType == typeof(Enumerable))
{
switch (method.Name)
{
case nameof(Enumerable.Contains) when arguments is [var source, var item] && ...
case nameof(Enumerable.First) when arguments is [var source] && ...
case nameof(Enumerable.Any) when arguments is [var source] && ...
}
}
Adopted here. The list patterns bind the operands by name instead of by index
and carry the arity check with them, so "First without a predicate" is
expressed by the pattern rather than by a separate GetParameters() test.
Behaviour is unchanged apart from one tightening that came free with the
shape: the Enumerable declaring-type check now applies to all three, where
previously none of them had it.
The emitted SQL is untouched and remains Jet's, not SQL Server's - INSTR over
STRCONV for Contains, ASCB(MIDB(...)) for First, LENB for Any. ByteArrayLength
stays outside the switch: it is a JetDbFunctions method, not an Enumerable one.
ByteArrayTranslations on both providers: 9 tests, 6 passing, and the same
three by-design refusals as before the change (Length, First, Index - Jet
cannot return an exact byte length).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These three need the EXACT byte length of a byte array, which Jet cannot
give: LenB reports the UTF-16 byte count and rounds an odd length up to even.
The provider refuses rather than returning a wrong number, and names
EF.Functions.ByteArrayLength as the opt-in workaround - whose remarks state
the one case it still gets wrong, data ending in 0x00, indistinguishable from
the zero pad.
So the tests were red for doing exactly what they should. They now assert the
InvalidOperationException, which is how the identical scenarios were already
handled in the GearsOfWar tests:
public override async Task Byte_array_filter_by_length_literal_does_not_cast_on_varbinary_n(bool async)
=> await Assert.ThrowsAsync<InvalidOperationException>(() => base....);
That adaptation dates from "Better support for byte arrays" (41dab6c, March
2024), when GearsOfWar was the only home of byte-array length coverage. EF10
moved these scenarios out into the simpler BasicTypesEntities suite, and the
new ByteArrayTranslations classes arrived carrying upstream's SQL Server
versions - which is why their baselines still read
CAST(DATALENGTH([b].[ByteArray]) AS int) in square brackets. Nothing
regressed; the reorganisation duplicated the coverage around the existing fix.
ByteArrayTranslations is now 9/9 on both providers, having been 6/9: one case
gained a real translation (byte[].Any() -> LENB > 0, 37df3d8) and three now
state that the refusal is the intended behaviour rather than sitting
permanently red and looking like bugs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The five Conditional_uncoalesce translations failed with
System.NotSupportedException : Function NULLIF is not supported.
EF Core emits NULLIF(a, b) for that pattern. ACE has no such function -
"Undefined function 'NULLIF' in expression", verified - so LibRed's refusal
matched ACE exactly, and the tests fail on the Jet provider too. Access's own
spelling is IIF(a = b, NULL, a), which works.
Implemented in LibRed rather than rewritten in the generator, so LibRed simply
accepts the name EF emits. That is a deliberate divergence, and the kind
LibRed exists for: the result is identical to the IIF form, so nothing about
the query's meaning changes - only whether the engine has to be worked around.
Equality reuses the same Compare() as the '=' operator, which makes the NULL
cases fall out without special-casing. Comparing with NULL is unknown rather
than equal, so NULLIF(x, NULL) is x and NULLIF(NULL, y) is NULL - matching the
IIF form, where an unknown condition takes the false branch. A test asserts
NULLIF and IIF agree row for row, including both NULL rows. Unlike the IIF
spelling the first operand is evaluated once.
Arity is registered as (2, 2) so a wrong argument count is rejected the same
way every other function's is.
The five baselines were still SQL Server's - brackets and N'' literals - i.e.
never ported, which is why nobody had noticed the function was missing. They
now read WHERE NULLIF(`b`.`Int`, 9) > 1.
MiscellaneousOperatorTranslations on LibRed: 10/10, was 5/10.
Engine 998/998, Core 793/793, Ado 59/59, Engine-ACE 32/32.
The Jet provider still fails these five: it would need the generator to emit
IIF instead, which is not done here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ACE has no NULLIF - "Undefined function 'NULLIF' in expression", verified - so
the five Conditional_uncoalesce translations produced SQL neither provider
could run.
There is nothing to override upstream. EF builds it inside
SqlExpressionFactory.Case, which collapses `a == b ? null : a` into
Function("NULLIF", [a, b]) as part of its conditional simplification; by the
time a tree reaches a provider it is already a function call, and overriding
Case would mean reimplementing that simplification. So it is handled where
dialect spelling belongs, in JetQuerySqlGenerator.VisitSqlFunction, which
already dispatches on name for @@-variables and COALESCE.
The SQL standard defines NULLIF(a, b) as shorthand for
CASE WHEN a = b THEN NULL ELSE a END, so this rebuilds exactly that and lets
VisitCase render it - which for Jet means IIF(a = b, NULL, a). Emitting the
CASE rather than writing the text reuses the existing operator and
NULL-literal rendering, and mirrors how binary Coalesce is already handled in
the same file. Placed before the null-guard logic so that does not wrap it.
'a' is written twice. The standard's own definition implies the same, and Jet
has no way to say it once.
WHERE IIF(`b`.`Int` = 9, NULL, `b`.`Int`) > 1
Both providers: MiscellaneousOperatorTranslations 10/10, was 5/10 on LibRed
and 5/10 on Jet. The Jet run is against a real ACE driver, so the emitted IIF
is known to execute rather than merely to parse.
LibRed shares JetQuerySqlGeneratorFactory today (its own generator is still
only planned), so it takes the same change and its baselines move from NULLIF
to IIF - one commit after gaining them. LibRed's NULLIF evaluator function
(a5aed06) stays: it is still reachable from hand-written SQL and still has
its own tests, it is simply no longer what EF emits.
The rest of Query.Translations is unchanged - 18 failures before and after,
the same DateTimeOffset/sub-second/Convert_ToInt64 set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three Jet translators emitted the interval as the full word "millisecond" -
TimeOnly.Millisecond, TimeSpan.Milliseconds and
DateTimeOffset.ToUnixTimeMilliseconds - which belongs to no dialect. Access's
interval vocabulary is abbreviations: yyyy, q, m, y, d, w, ww, h, n, s. So
they now send "ms", which is what LibRed's DatePart table already used, and
the mismatch that made those tests fail was a spelling, not a capability. The
three commented-out AddMilliseconds/Millisecond entries were changed with them
so the vocabulary stays consistent if they are ever enabled.
DateDiff then needed the interval it did not have. It is a LibRed extension -
ACE's list stops at "s" - available because LibRed stores the full OA double
instead of truncating to whole seconds as ACE does. Measured: 12:34:56.123
round-trips with zero tick loss.
It returns Int64, unlike every other interval, and is handled before the
switch rather than inside it. A millisecond difference overflows Int32 after
25 days and ToUnixTimeMilliseconds spans decades; a long arm within the switch
expression would silently widen every other interval's result type along with
it. Tests assert both halves - that ms exceeds Int32 without truncation, and
that s/n/h/d/yyyy still return Access's Long Integer.
Nothing below a millisecond was added. It would not survive storage anyway:
DateTime.ToOADate/FromOADate quantise to whole milliseconds, so .1234560 comes
back as .123 - a BCL conversion floor, not a format one, sitting four orders
of magnitude above the double's own ~0.63us resolution at present-day values.
DateTimeOffset.ToUnixTimeMilliseconds now passes:
WHERE DATEDIFF('ms', CDATE('1970-01-01 00:00:00'), `b`.`DateTimeOffset`) = @unixEpochMilliseconds
The Millisecond member tests still fail, and no longer for a reason this
change can reach: they translate and execute correctly but find nothing,
because JetDateTimeTypeMapping writes the literal as hh:mm:ss with no
fractional part, so the seeded .123 never reached the database. That is
storage, not vocabulary.
LibRed Query.Translations 361/378, was 360. Engine 1011/1011, Core 793/793,
Ado 59/59, Engine-ACE 32/32.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ACE truncates a date/time to whole seconds on write, so EFCore.Jet has never
had reason to emit or preserve a fractional part. LibRed stores the full OLE
Automation double, where a millisecond survives exactly - .NET's ToOADate and
FromOADate quantise there, making a millisecond both the finest unit that
round-trips and the natural floor.
Engine and ADO:
- ExpressionEvaluator.DateAdd gains an "ms" interval, matching DateDiff.
DatePart already had ms/mcs/ns.
- LibRedCommand.Normalize truncates parameters to a millisecond rather than a
second, so a literal and a parameter compare equal.
Provider: four LibRed type mappings (DateTime, DateTimeOffset, TimeOnly,
TimeSpan) mirroring their EFCore.Jet counterparts, deriving from the EF Core
bases rather than from Jet's, and differing only in emitting a .fff fraction
when it is non-zero. LibRedTypeMappingSource substitutes them for the mapping
Jet resolved, so store type, size and facets are preserved.
EFCore.Jet, shared with the ACE path:
- DateTime.Millisecond and DateTimeOffset.Millisecond now translate to
DATEPART('ms', ...). Both were already failing on ACE, which has no
sub-second interval, so this changes only which error they raise.
- AddMilliseconds now translates to DATEADD('ms', ...). On ACE this turns
DateTimeOffsetTranslationsJetTest.AddMilliseconds from passing to failing,
but it only passed because the call sat in a top-level projection - the one
position EF may evaluate on the client - so the SQL never contained the
addition at all. Nothing that exercised the feature is lost.
- The CDATE/TIMEVALUE parameter wraps in JetQuerySqlGenerator now match on the
EF Core base mapping instead of the Jet one, so a derived provider that
substitutes its own mapping still gets the coercion.
Known gap: DateOnly.ToDateTime still drops milliseconds, because it decomposes
through TimeSerial(h, n, s). Carrying them would need a fourth TimeSerial
argument - an arity change rather than an extra value in an existing parameter
- so it is deliberately left for separate work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
EF Core's base QuerySqlGenerator already emits FULL JOIN, so this needed no generator work - only the parser, plan and executor, none of which is shared with the ACE path. Grammar: a FULL token and a FullJoin alternative in joinType, parser regenerated. OUTER is optional and carries no meaning, there being no such thing as a full inner join - it is accepted only because people write it, the same way LEFT OUTER JOIN is. Executor, nested loop: the right side's matched rows are tracked across the whole left pass and the unmatched ones emitted null-padded afterwards, since a right row's fate is not settled until every left row has been tried. Executor, hash: FULL builds the right and probes with the left as INNER/LEFT do, but preserves the build side too, so it also records which build rows were hit. The build phase drops null-key rows as unmatchable, which under FULL is the very reason to emit them - those are set aside and joined to the unmatched tail rather than discarded. IndexSelection lets FULL hash so it does not fall back to an O(n*m) loop; the index-nested-loop path is untouched, its allow-list already excluding anything outer. Two things this drags along: - AstBuilder.ViewJoinKindOf now throws for a FULL JOIN inside a CREATE VIEW. That switch falls through to Inner, and Access's stored query format has no full outer join to encode, so the alternative was silently storing an inner join. - ExecuteJoin only ever honoured the left-preserving flag, so a RIGHT JOIN whose ON had no same-kind equi-key - the case the hash path declines - ran silently as an INNER join. The flag added for FULL fixes that too. FULL becomes a keyword despite not being reserved in Access, so a column actually named "Full" now needs bracketing or backticking, as LEFT, RIGHT and ORDER already do. FullJoinTests covers both quoting forms along with the preserved-null-key row, the nested-loop path, and FULL = LEFT + RIGHT - INNER. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Upstream e54991e7 ("Fix AssertAllMethodsOverridden for XUnit V3 Fact/Theory",
#38570) taught TestHelpers.AssertAllMethodsOverridden to recognise plain
[Fact]/[Theory] alongside ConditionalFact/ConditionalTheory. Our TestHelpers
already carries that fix, so the stricter check applies here too - it just was
not being run in five of the nine files upstream had to touch.
Check_all_tests_overridden added to AdHocAdvancedMappings, AdHocManyToMany,
AdHocNavigations, AdHocQueryFilters and AdHocQuerySplitting, for both
providers, as the last member of the class exactly as upstream places it (plus
a using Xunit in the four files that lacked one). The other four files already
had it.
That surfaced 28 missing overrides per provider, against upstream's 11 - ours
also lagged on base tests SQL Server had overridden before that commit:
AdHocAdvancedMappings 5 (2 TPC_query_with_generic_derived_types_*,
3 Projecting_*_with_converter_*closure)
AdHocNavigations 4 (3 Consecutive_selects_with_conditional_projection_*,
Filtered_collection_through_optional_navigation_*)
AdHocQueryFilters 13 (the whole #region 8576 Named_query_filters set,
plus 6 at the end)
AdHocQuerySplitting 6 (plus upstream's private AssertContainsSql helper,
which two of them need and we did not have)
AdHocManyToMany 0 (the check alone was enough, as upstream)
Each is copied verbatim from its SQL Server counterpart and placed at its
upstream position rather than appended - the QueryFilters set is a region that
sits first in the class, and three of the Navigations ones go mid-file.
Baselines came across in SQL Server's bracket form as the not-ported marker and
have since been regenerated for both providers where the test actually reaches
AssertSql. All Check_all_tests_overridden now pass: 96/96 on each provider.
Also here:
- Seed2951 used T-SQL's optional INTO ("INSERT ZeroKey VALUES (NULL)"), copied
from SQL Server and never adapted; Jet/ACE requires INSERT INTO. That is why
Query_when_null_key_in_database_should_throw failed on LibRed and was absent
from ACE's green list. Fixed for both.
- The millisecond work needed the same component-constructor widening in the
GearsOfWar fixtures and LibRedTestHelpers that BasicTypes already had. The
scaffolding compiled-model baselines now name LibRedDateTimeTypeMapping,
confirming the mapping substitution reaches design-time codegen; GearsOfWar on
Jet was unchanged, confirming the work is LibRed-only in effect.
- DateTimeOffset AddMilliseconds now emits DATEADD('ms', ...) where the baseline
was a bare projection - it used to be client-evaluated, which only worked
because it sat in a top-level projection.
Known open: Correlated_SelectMany_DefaultIfEmpty_whole_object still fails on
both providers. The base asserts the exception message contains "requires the
SQL APPLY"; Jet and LibRed correctly refuse OUTER APPLY but do it from the SQL
generator with their own "Unsupported Jet expression: OUTER APPLY" instead of
EF's canonical provider-capability error.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both of these were failing against expectations that no real provider holds - one stale in the base, one faked in our own file. Neither is a Jet or LibRed limitation. Join_local_bytes_closure_is_cached_correctly: the base wraps the query in AssertTranslationFailed, but a byte[] joined to a local collection translates perfectly well - SQL Server does it through VALUES and overrides the base with a real-results reimplementation. We inline the same thing through the #Dual UNION subquery, which is element-type agnostic and so no more troubled by bytes than by ints. Reimplemented following SQL Server's shape. LibRed carries the SQL as probed from a run; Jet keeps SQL Server's bracket baseline, because the whole Join_local family is unported there (the int sibling still has OPENJSON in it) and none of it appears in ACE's green list, so there is nothing to write real Jet SQL from. Correlated_collection_after_distinct_with_complex_projection_not_containing_ original_identifier: replaced a baseline that was SQL Server SQL wearing backticks - OUTER APPLY and DATEPART(month, ...), neither of which Jet can produce - with the assert-throws upstream uses verbatim, comment and issue number included. The exception is EF's own InsufficientInformationToIdentifyElementOfCollectionJoin, raised in SelectExpression.ApplyProjection before any provider SQL is generated: a Distinct projection that drops the identifying information, efcore #24440. It is provider-independent, so asserting the refusal is the whole of the story. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both fixes are ports of what JetConnection already does, which is in turn what SqlConnection does - all of ConnectionSpecificationTest passes on Jet, and now does on LibRed too (5 failures to 0 of 27). A disposed connection has to be unusable. Dispose() only called Close(), and nothing recorded that the connection was finished with, so a later Open() just reopened the file. Clearing the connection string is what enforces it: Open() then fails its existing "missing a Data Source" guard. That also fixes the exception TYPE, which matters here - the tests use xunit's exact-match Assert.Throws<InvalidOperationException>, so an ObjectDisposedException would be rejected despite deriving from it. Assigned to the field directly, since the property setter refuses to change while the connection is open. That one line fixed four tests rather than the one it was chased for: the two Can_specify_owned_connection_in_OnConfiguring variants and both Can_specify_no_connection_in_OnConfiguring(contextOwnsConnection: true) ones were the same contract seen from different angles. Open() and Close() now raise OnStateChange. They were assigning _state directly and telling nobody, so a caller watching StateChange - EF's connection diagnostics among them - saw a connection that never opened or closed. Close() is not guarded against being called on an already closed connection, and Dispose() calls it, so the event is raised only on a real transition; firing unconditionally would report a second close that never happened. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three unit tests still asserted the whole-second truncation that "LibRed: carry milliseconds through the temporal path" deliberately replaced. That commit updated the functional fixtures and missed these, so they have been red since. They are updated rather than deleted: each still pins a truncation boundary, just the one that now exists. Sub-millisecond is where values stop surviving, because .NET's ToOADate/FromOADate quantise there, so that is what the tests prove is discarded. - DateTime_parameter_is_truncated_to_whole_seconds is renamed for milliseconds and now inserts a sub-millisecond value, expecting the millisecond back. The `WHERE d = @p` half is kept, reusing the sub-millisecond parameter so the write and comparison paths still have to agree on the same boundary. - TimeSpan_parameter_round_trips_through_a_datetime_column expects its 678 ms back instead of whole seconds. - TemporalParameterTests' "sub-second stripped both ways" case becomes a millisecond that survives, plus a new case storing 7.5 ms and querying 7 ms to pin the boundary from the other side. LibRed.Ado.Tests 60/60. Engine 1020/1020 and Core 793 passed / 46 skipped were already clean and are unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New
Both: Add new Parse translator
Both: Implemented a jet compatible NULLIF to IIF... sql translation
LibRed only
Full BMP plane coverage of General and General Legacy collating orders
Implemented tailorings for all 30 non CJK collating orders
Database can be created with a specific collating order
Implemented INSERT INTO ... SELECT
Implemented SELECT ... INTO
Implemented NULLIF function
Implemented FULL JOIN
Bug Fixes
Continue the Autonumber counter past the int32 wrap
Fix the migrations-lock hang in JetHistoryRepository
Take the relocation pointer from the slot's first 4 bytes, not its width