Skip to content

Reduce exact observable-mask preparation time by 89.76% (no κ) #6466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
petalas wants to merge 1 commit into
base: main
Choose a base branch
from
+1,224 −0
Open

Reduce exact observable-mask preparation time by 89.76% (no κ) #6466

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
169 changes: 169 additions & 0 deletions research/observable-hamming-transform/BENCHMARK.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
{
"files_sha256": {
"SOURCES.json": "52801d3b116cf64ea6b88224efffe645f3eb1fe3a6cd231480b976ff4b10b61c",
"benchmark.py": "38e26dc2762cef373aa8b1bdb63f4f989bbf5d5fa965692e78338f46f5850cfd",
"planner.py": "fd75139c6d7b89cb92b3c13430538d154c8cc28f4833fe9dbe37425010fab965",
"source.py": "e7d372452ced80308634dd938d69c5be59eda6f01a74ebf7dd0cf577183e372a",
"workload.py": "2a987f6a1294751eac945544868b748ffb9b6e4c4be0b50019e748bf0030ceee"
},
"fractional_wall_reduction": 0.8976254132376362,
"load_average_at_end": [
2.53125,
3.52099609375,
2.97802734375
],
"logical_cpus": 18,
"platform": "macOS-27.0.1-arm64-arm-64bit",
"python": "3.12.15",
"records": [
{
"output": {
"cost": 58,
"errors": 0,
"gates": 616,
"sha256": "7d39c2f8daf285394ba611b234a02a5d78e2ddc910d40413cbb3c67f69baaeeb",
"stages": 64,
"table_entries": 16640
},
"side": "baseline",
"trial": 0,
"wall_seconds": 0.3611299160402268
},
{
"output": {
"cost": 58,
"errors": 0,
"gates": 616,
"sha256": "7d39c2f8daf285394ba611b234a02a5d78e2ddc910d40413cbb3c67f69baaeeb",
"stages": 64,
"table_entries": 16640
},
"side": "transform",
"trial": 0,
"wall_seconds": 0.03699633292853832
},
{
"output": {
"cost": 58,
"errors": 0,
"gates": 616,
"sha256": "7d39c2f8daf285394ba611b234a02a5d78e2ddc910d40413cbb3c67f69baaeeb",
"stages": 64,
"table_entries": 16640
},
"side": "baseline",
"trial": 1,
"wall_seconds": 0.36088516586460173
},
{
"output": {
"cost": 58,
"errors": 0,
"gates": 616,
"sha256": "7d39c2f8daf285394ba611b234a02a5d78e2ddc910d40413cbb3c67f69baaeeb",
"stages": 64,
"table_entries": 16640
},
"side": "transform",
"trial": 1,
"wall_seconds": 0.035862165968865156
},
{
"output": {
"cost": 58,
"errors": 0,
"gates": 616,
"sha256": "7d39c2f8daf285394ba611b234a02a5d78e2ddc910d40413cbb3c67f69baaeeb",
"stages": 64,
"table_entries": 16640
},
"side": "baseline",
"trial": 2,
"wall_seconds": 0.36138200014829636
},
{
"output": {
"cost": 58,
"errors": 0,
"gates": 616,
"sha256": "7d39c2f8daf285394ba611b234a02a5d78e2ddc910d40413cbb3c67f69baaeeb",
"stages": 64,
"table_entries": 16640
},
"side": "transform",
"trial": 2,
"wall_seconds": 0.036010124953463674
},
{
"output": {
"cost": 58,
"errors": 0,
"gates": 616,
"sha256": "7d39c2f8daf285394ba611b234a02a5d78e2ddc910d40413cbb3c67f69baaeeb",
"stages": 64,
"table_entries": 16640
},
"side": "baseline",
"trial": 3,
"wall_seconds": 0.36345991701819
},
{
"output": {
"cost": 58,
"errors": 0,
"gates": 616,
"sha256": "7d39c2f8daf285394ba611b234a02a5d78e2ddc910d40413cbb3c67f69baaeeb",
"stages": 64,
"table_entries": 16640
},
"side": "transform",
"trial": 3,
"wall_seconds": 0.03733570803888142
},
{
"output": {
"cost": 58,
"errors": 0,
"gates": 616,
"sha256": "7d39c2f8daf285394ba611b234a02a5d78e2ddc910d40413cbb3c67f69baaeeb",
"stages": 64,
"table_entries": 16640
},
"side": "baseline",
"trial": 4,
"wall_seconds": 0.37605391605757177
},
{
"output": {
"cost": 58,
"errors": 0,
"gates": 616,
"sha256": "7d39c2f8daf285394ba611b234a02a5d78e2ddc910d40413cbb3c67f69baaeeb",
"stages": 64,
"table_entries": 16640
},
"side": "transform",
"trial": 4,
"wall_seconds": 0.038020709063857794
}
],
"runs_per_side": 5,
"schema_version": 1,
"summaries": {
"baseline": {
"median_seconds": 0.36138200014829636,
"range_seconds": [
0.36088516586460173,
0.37605391605757177
]
},
"transform": {
"median_seconds": 0.03699633292853832,
"range_seconds": [
0.035862165968865156,
0.038020709063857794
]
}
},
"timed_region": "fresh process, imports and source authentication, demands, solve, complete word emission, every observation, full inverse, invoice and output digest"
}
62 changes: 62 additions & 0 deletions research/observable-hamming-transform/PROFILE.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
Profiled complete preparation, separate from all reported benchmark samples.

{
"baseline": {
"total_profiled_seconds": 0.5015532080000001,
"solve_cumulative_seconds": 0.49892225,
"solve_fraction": 0.9947543790807533
},
"transform": {
"total_profiled_seconds": 0.012790708000000001,
"solve_cumulative_seconds": 0.01031925,
"solve_fraction": 0.8067770759835968
}
}

BASELINE
4229003 function calls in 0.502 seconds

Ordered by: cumulative time
List reduced from 48 to 15 due to restriction <15>

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.502 0.502 workload.py:19(prepare)
1 0.007 0.007 0.499 0.499 model.py:65(solve)
16385 0.239 0.000 0.489 0.000 {built-in method builtins.min}
4145408 0.249 0.000 0.249 0.000 model.py:85(<genexpr>)
19808 0.002 0.000 0.003 0.000 model.py:12(dot)
1 0.000 0.000 0.002 0.002 model.py:197(check_word)
65 0.001 0.000 0.001 0.000 model.py:178(expected_rows)
19808 0.001 0.000 0.001 0.000 {method 'bit_count' of 'int' objects}
16385 0.001 0.000 0.001 0.000 {method 'items' of 'dict' objects}
385 0.000 0.000 0.001 0.000 {built-in method builtins.sum}
1 0.000 0.000 0.000 0.000 model.py:96(emit)
2 0.000 0.000 0.000 0.000 model.py:49(demands)
74 0.000 0.000 0.000 0.000 {built-in method builtins.all}
128 0.000 0.000 0.000 0.000 model.py:16(matvec)
1 0.000 0.000 0.000 0.000 model.py:214(invoice)



TRANSFORM
51425 function calls in 0.013 seconds

Ordered by: cumulative time
List reduced from 54 to 15 due to restriction <15>

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.013 0.013 workload.py:19(prepare)
1 0.000 0.000 0.010 0.010 planner.py:80(solve)
64 0.006 0.000 0.006 0.000 planner.py:32(distance_transform)
64 0.002 0.000 0.004 0.000 planner.py:54(advance)
64 0.001 0.000 0.002 0.000 {built-in method builtins.sorted}
1 0.000 0.000 0.002 0.002 model.py:197(check_word)
65 0.001 0.000 0.001 0.000 model.py:178(expected_rows)
19808 0.001 0.000 0.001 0.000 {method 'bit_count' of 'int' objects}
16448 0.001 0.000 0.001 0.000 planner.py:64(<genexpr>)
1 0.000 0.000 0.000 0.000 model.py:96(emit)
385 0.000 0.000 0.000 0.000 {built-in method builtins.sum}
2 0.000 0.000 0.000 0.000 model.py:49(demands)
3360 0.000 0.000 0.000 0.000 model.py:12(dot)
75 0.000 0.000 0.000 0.000 {built-in method builtins.all}
128 0.000 0.000 0.000 0.000 model.py:16(matvec)
130 changes: 130 additions & 0 deletions research/observable-hamming-transform/PROOF.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# Exact transition and witness preservation

## Declared model

Let d>=1 and m>=0 be integers. Use precisely PR1523's observable
dirty-connector correction model. A state
k in F2^d records the correction carried by the connector at one prescribed
observation. Coordinate j has positive complete correction price w_j.
Its distance is D(h,k)=sum_j w_j [h_j != k_j]. At stage i a reader a_i and
demand b_i require a_i dot k=b_i, unless an independently priced output
correction pays v_i for the one-bit defect. The initial and terminal states
are zero. Sources execute their prescribed restored mutation tour. This proof
changes how the existing preparation recurrence is evaluated; it assumes the
predecessor's correspondence between that recurrence and legal complete words.

Write F_i(k) for the minimum prefix price and p_i(k) for the lexicographically
smallest minimum-price state sequence ending at k. All such sequences have
length i. Initially F_0(0)=0, p_0(0) is empty and other states are unreachable.
For a given destination k, the predecessor scan minimizes

(F_(i-1)(h) + D(h,k), p_(i-1)(h)).

The stage penalty depends only on k, so adding it does not change the selected
predecessor. Appending k also preserves the predecessor order. In connector-only
mode, delete the states with nonzero defect. In paid-output mode, add v_i times
the defect. These are exactly the original filtering and penalty rules.

## Hypercube invariant

Give each reachable prefix a distinct integer rank r(h), in its complete
lexicographic path order. Initialize mask h with the label
(F(h),r(h),h), or infinity if it is unreachable. Process coordinates in fixed
order. For each pair of masks differing only in coordinate j, simultaneously
replace the two labels with their minimum against the opposite label plus
w_j. The rank and origin fields travel unchanged.

After processing a coordinate set J, the label at k is the lexicographic minimum

(F(h) + sum_(j in J) w_j [h_j != k_j], r(h), h)

over precisely those h equal to k outside J. Induct on the processed set.
The states equal to k in the newly processed bit come from its old label.
The remaining states come from the opposite endpoint's old label plus that
bit's price. Those two sets partition the possible origins. Their minimum is
the stated new label. Taking both old endpoints before either assignment
implements this induction literally. After all d coordinates, every origin
is included and the cost is the full weighted Hamming distance.

Ranks are injective on reachable canonical paths. Therefore the rank tie-break
chooses exactly the same whole predecessor path as the original solver. The
origin tie-break cannot change that choice. An unreachable layer remains
unreachable; no finite label is introduced without a finite origin.

## Maintaining path ranks and returning the witness

At destination k retain the selected predecessor h as a backpointer. Its new
canonical path is p_(i-1)(h) followed by k. Order the retained destinations
by (r(h),k), and assign their new dense integer ranks in that order. Equal-length
prefixes compare first by their predecessor path and then by the last state,
so these are exactly the lexicographic ranks of the new complete paths.
Different destinations have different last states, hence distinct paths and
distinct ranks. This proves the rank invariant by induction, including every
cost tie.

Store each F_i table unchanged. At the terminal boundary select the smallest
(F_m(k)+D(k,0),r(k),k). Every candidate appends the same final zero, so the
rank again selects the original lexicographic witness. Trace the stored
predecessors back to the initial zero and append the terminal zero. The empty
stage sequence returns [0]. The output-defect bits are then computed from
the same readers, demands and selected states. Thus every returned field
equals the original planner on their common input domain, including all
Bellman tables, the whole path, correction bits, copied weights and prices.

The original emitter receives the same plan and unchanged source specification.
It therefore emits the same gates in the same order, including both B calls,
all reader prewrites, source mutations, correction operations and final
connector restoration. The improvement cannot remove a data observation,
dirty-state obligation or paid inverse call.

## Complexity and bounded implementation

Let N=2^d. One transform visits dN/2 undirected pairs, considering at most two
directed relaxations per pair. Filtering and cost-table storage are O(N).
Sorting at most N keys for path ranking uses O(N log N)=O(dN) comparisons;
each key has two bounded integers, rather than a growing path tuple.
The terminal distances cost O(dN), and reconstructing one witness costs O(m).
Consequently the complete abstract preparation recurrence costs
O((m+1)dN) arithmetic operations and label comparisons, and O((m+1)N)
stored labels/integers, including the returned tables and all backpointers.

These are arithmetic and comparison counts. If prices use at most B bits,
finite prefix prices use at most B+O(log((m+1)(d+1))) bits and ranks/indices
use O(d) bits. Multiplying the operation bound by
O(max(d,B+log((m+1)(d+1)))) bounds elementary bit work for the displayed
operations. Program construction, input/output and physical data operations
retain their separate costs; they are measured in the full finite pipeline.

The executable enforces d<=8, m<=256 and B<=256, with positive integer prices.
These are resource bounds, not hypotheses of the abstract induction. Inputs
outside these bounds are rejected before table construction. The original
solver has no corresponding stage/price-bit cap; equivalence is claimed on
the declared common bounded domain. The maximum table and backpointer lengths
are each (256+1)*256 entries, and simultaneous transform/rank arrays have
256 entries. No recursion or unbounded retry is introduced. All functions
remain shorter than sixty physical lines.

## Implementation discipline and deviations

The owned new-code loops are bounded by the enforced width, stage, price-bit
limits or fixed verification inventories. Source consumers retain their
unchanged internal algorithms; they are authenticated comparison inputs and
are not new production dependencies. Their implementations are outside the
new algorithm's complexity claim.

Power-of-Ten rule 3 requires a managed-runtime deviation. Python allocates
integers, tuples and list elements during execution. The limits above bound
simultaneously live planner storage and integer sizes; verifier inventories
and benchmark subprocesses are finite, with a sixty-second child timeout.
Exact conformance, limit rejection and both optimization modes exercise these
controls. A fixed-buffer native implementation would require a separate change
if such a backend is requested.

Rule 8 also requires a bounded source-loading deviation. The compatibility
loader compiles only repository source bytes after checking every declared
live input's SHA256 against SOURCES.json. It restores temporary import names
and search paths in a finally block. This permits the unchanged complete
consumer to exercise the new solver without modifying its files. A corrupted
digest is rejected. No data-demand input is compiled as program text.
The scientific commands run with warning diagnostics enabled. Assertions are
not used for acceptance and optimized Python retains every rejection check.
Loading