Skip to content

Migrate native bridge to precompiled Swift archives and purego - #240

Closed
Code-Hex wants to merge 5 commits into
mainfrom
codex/purego-foundation
Closed

Code-Hex wants to merge 5 commits into
mainfrom
codex/purego-foundation

Conversation

@Code-Hex

@Code-Hex Code-Hex commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Fixes #239

Consumer builds currently recompile the Objective-C cgo bridge from source. This change links precompiled Swift archives directly into consumer executables and invokes them through purego. Public Go API signatures remain unchanged, and window UI code stays in Objective-C running on the main thread.

Application builds still require CGO_ENABLED=1, Clang, and a macOS SDK, but they do not compile Swift sources. A small cgo adapter provides native symbol addresses, and standard binary signing covers the bridge code. macOS continues to supply system Apple frameworks and the Swift runtime.

Why Swift

Apple Virtualization now includes Swift-only types such as VZVirtualMachineViewAdaptor. A Swift bridge allows writing adapters for these APIs and compiling them with Swift 6 concurrency checks.

The generator does not yet support VZVirtualMachineViewAdaptor, async methods, actor-isolated declarations, or arbitrary value types. Those interfaces still require handwritten adapters.

Generation and linkage

The generator tool (cmd/vzbridgegen) reads SDK Swift declarations and Clang enum widths to generate all supported public bindings. It also inspects the host Objective-C runtime to discover and generate all supported private methods, removing the need for a manually maintained private selection list.

SwiftSyntax splits native entry points into individual archive members, and each Go binding registers itself on first call. Normal Go builds link only reachable entry points and their dependencies. The compiler checks bridge types and C ABI compatibility. A small generated linker stub records verified weak Virtualization imports so older consumer SDKs can link the archives without altering runtime availability checks.

flowchart TB
    subgraph generation["Maintainer workflow"]
        sdk["SDK Swift declarations and Clang enum widths"] --> gen["cmd/vzbridgegen"]
        private["Host Objective-C runtime methods"] --> gen
        gen --> wrappers["Generated Swift wrappers"]
        wrappers --> compiler["Swift and Clang"]
        manual["Handwritten Swift adapters and Objective-C UI"] --> compiler
        compiler --> split["One entry point per archive member"]
        split --> native["libBridge.a and Virtualization.tbd"]
        gen --> bindings["Generated Go bindings"]
    end
    subgraph build["Application build, CGO_ENABLED=1"]
        native --> linker["go build and cgo linker"]
        bindings --> reachable["Reachable Go bindings"]
        reachable --> linker
        linker --> executable["Executable with reachable bridge functions"]
    end
    executable -->|"Runtime calls"| framework["System Virtualization.framework and Swift runtime"]
Loading

Ownership and asynchronous calls

Managed bridge objects share one Go owner across aliases. runtime.AddCleanup registers a cleanup that runs after that owner becomes unreachable. Cleanup timing is not deterministic, and releasing the Go-owned reference destroys the native object only if no other native references remain.

flowchart TD
    native["Swift returns one retained native reference"] --> owner["Go aliases share one owner<br/>NewManagedPointer and runtime.AddCleanup"]
    owner --> gc["Owner becomes unreachable<br/>Go runtime schedules cleanup later"]
    gc --> close["managedReference.close via sync.Once<br/>ReleaseObject via purego"]
    close --> queue["bridgeQueue.async<br/>Release the native reference"]
    queue --> last{"Last native reference?"}
    last -->|"Yes"| destroy["Native object is destroyed"]
    last -->|"No"| retained["Other native owners keep it alive"]
Loading

The following sequence shows a one-shot request such as VirtualMachine.Start. runtime.KeepAlive protects Go arguments through the synchronous bridge call. The Swift completion closure keeps the native receiver alive until completion. The Go closure stays in the callback registry, while Swift stores the numeric request ID and a shared callback function address.

sequenceDiagram
    participant G as Go API
    participant R as Go callback registry
    participant S as Swift bridge
    participant Q as Bridge queue
    participant F as Virtualization.framework
    G->>R: Register completion closure
    R-->>G: uint64 request ID
    G->>S: purego call with native receiver and request ID
    S->>Q: bridgeSync
    Q->>F: Start asynchronous operation
    Note over Q,F: Swift closure retains the native receiver
    Q-->>S: Operation scheduled
    S-->>G: Bridge call returns
    G->>G: runtime.KeepAlive(arguments)
    Note over G: Wait on the buffered completion channel
    F-->>S: Completion callback
    S->>Q: emit through bridgeSync
    Q->>R: purego callback with request ID and result
    R->>R: Look up and remove the one-shot entry
    R-->>G: Invoke closure and send the result
    G->>G: Return the result to the caller
Loading

The completion channel is buffered, so completion can arrive even before the Go caller starts waiting. Repeated event callbacks keep their registry entries until removal or a terminal event.

Maintenance and generator details:

  • Regenerating the bridge requires Xcode with Swift 6.4.
  • Private API ownership rules and queue dispatch requirements still require manual review.
  • Discovery reflects the host macOS version and architecture; it does not detect methods present only on the other architecture.
  • Unsupported blocks, function pointers, and value types are logged during generation.

Build time

The median package build fell from 47.99 s to 6.37 s, about 7.5x faster and 86.7% less time.

Measured against c03d09a on an Apple M2, macOS 27.0.1, Xcode 27.0.0, and Go 1.25.0. Both versions used CGO_ENABLED=1. Each ran go build -modcacherw . three times in alternating order with fresh Go build, dependency, Clang, and Swift module caches. Dependency downloads are included; source checkouts were already present, and OS file caches were retained. This measures package builds, not final application linking.

Executable size

A test program that creates and prints a MAC address shrank from 6,300,226 to 5,559,426 bytes (an 11.8% reduction compared to the previous static bridge). Native bridge export symbols dropped from 424 to 7. This build includes all generated runtime bindings in the archive, including private methods.

A separate linker check invokes one SDK binding and then adds one runtime binding. The resulting executables contain only those specific entry points plus the ABI check, and both run successfully.

The archives themselves are 24.6 MB for arm64 and 23.5 MB for amd64, containing 1,708 runtime bindings (including the allocation helper). The linker does not pull unreferenced archive members into the executable. Generation reported 352 unsupported runtime methods on macOS 27.0.1 (arm64).

Add cmd/vzbridgegen to inspect Virtualization.framework Swift declarations and selected private Objective-C runtime methods. The tool generates checked Go and purego bindings alongside precompiled arm64 and amd64 static Swift archives. It also emits a minimal linker stub with verified weak imports to support linking against older SDK versions.
Link precompiled Swift archives through a small cgo adapter and call them with purego. Preserve public Go signatures and tie native lifetimes to Go owners, with releases on the bridge queue and request-ID callbacks. Keep window UI in Objective-C on the main thread and cover native cleanup and callbacks with integration tests.
Explicitly enable CGO in CI configurations to support static linkage. Document consumer Clang and SDK requirements, deployment targets, and bridge regeneration procedures. Add documentation covering public and private API generation rules along with native ownership semantics.
Generate all supported host runtime bindings without a selection list.
Register each Go binding on first use and keep native entry points in
separate archive members. Preserve ownership and ABI checks.
@Code-Hex

Copy link
Copy Markdown
Owner Author

@AkihiroSuda @cfergeau I’ve changed the internal implementation to reduce build times. The public API signatures remain unchanged.

Could you try this with your usual workloads and check that everything still works as expected?
I plan to merge it if no issues come up. Please let me know if you have any other concerns.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This binary blob dependency makes the library harder to use.
Unlikely to be acceptable in package managers such as Homebrew.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’d also have to rebuild it before releasing/signing vfkit

@cfergeau

cfergeau commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

I tried this branch, first I wanted to build example/gui-linux, but this failed with:

go build -o virtualization .
go: downloading [github.com/ebitengine/purego](http://github.com/ebitengine/purego) v0.10.1
go: downloading [golang.org/x/sys](http://golang.org/x/sys) v0.45.0
# [github.com/Code-Hex/vz/example/gui-linux](http://github.com/Code-Hex/vz/example/gui-linux)
/opt/homebrew/Cellar/go/1.27.1/libexec/pkg/tool/darwin_arm64/link:
running cc failed: exit status 1
/usr/bin/cc -arch arm64 -Wl,-headerpad,1144 -o $WORK/b001/exe/a.out
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/go.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000000.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000001.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000002.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000003.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000004.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000005.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000006.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000007.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000008.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000009.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000010.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000011.o
/var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-[3698321559](tel:(369)%20832-1559)/000012.o
-lresolv -O2 -g
/Users/teuf/dev/vz/internal/vzbridge/abi_arm64/libBridge.a
-L/usr/lib/swift -framework Foundation -framework Virtualization
-framework Cocoa
/Users/teuf/dev/vz/internal/vzbridge/abi_arm64/Virtualization.tbd -O2 -g
-framework CoreFoundation -framework Security
ld: multiple errors: tapi error: malformed file
/Library/Developer/CommandLineTools/SDKs/MacOSX27.0.sdk/usr/lib/libresolv.9.tbd:4:20:
error: unknown architecture
                    arm64e.x1-macos, arm64e.x1-maccatalyst ]
                    ^~~~~~~~~~~~~~~
  in
'/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libresolv.tbd';
tapi error: malformed file
/Library/Developer/CommandLineTools/SDKs/MacOSX27.0.sdk/System/Library/Frameworks/Virtualization.framework/Versions/A/Virtualization.tbd:3:48:
error: unknown architecture
targets:         [ x86_64-macos, arm64e-macos, arm64e.x1-macos ]
                                                ^~~~~~~~~~~~~~~~
  in
'/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Virtualization.framework/Virtualization.tbd';
tapi error: malformed file
/Library/Developer/CommandLineTools/SDKs/MacOSX27.0.sdk/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa.tbd:3:48:
error: unknown architecture
targets:         [ x86_64-macos, arm64e-macos, arm64e.x1-macos ]
                                                ^~~~~~~~~~~~~~~~
  in
'/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Cocoa.framework/Cocoa.tbd';
tapi error: malformed file
/Library/Developer/CommandLineTools/SDKs/MacOSX27.0.sdk/usr/lib/libSystem.B.tbd:4:20:
error: unknown architecture
                    arm64e.x1-macos, arm64e.x1-maccatalyst ]
                    ^~~~~~~~~~~~~~~
  in
'/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libSystem.tbd';
tapi error: malformed file
/Library/Developer/CommandLineTools/SDKs/MacOSX27.0.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation.tbd:4:54:
error: unknown architecture
                    arm64e-macos, arm64e-maccatalyst, arm64e.x1-macos,
arm64e.x1-maccatalyst ]
                                                      ^~~~~~~~~~~~~~~
  in
'/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation.tbd';
tapi error: malformed file
/Library/Developer/CommandLineTools/SDKs/MacOSX27.0.sdk/System/Library/Frameworks/Security.framework/Versions/A/Security.tbd:4:20:
error: unknown architecture
                    arm64e.x1-macos, arm64e.x1-maccatalyst ]
                    ^~~~~~~~~~~~~~~
  in
'/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Security.tbd';
tapi error: malformed file
/Library/Developer/CommandLineTools/SDKs/MacOSX27.0.sdk/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation.tbd:4:20:
error: unknown architecture
                    arm64e.x1-macos, arm64e.x1-maccatalyst ]
                    ^~~~~~~~~~~~~~~
  in
'/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Foundation.framework/Foundation.tbd'
clang: error: linker command failed with exit code 1 (use -v to see
invocation)

Updating to xcode 27 fixed this failure. I don’t know if this is expected to fail with older xcode versions.

Then I tried to use the purego-foundation branch with vfkit, but this failed with this error, which may be related to golang/go#26366. The 2 files (libBridge.a and Virtualization.tbd) are indeed missing from the vendor directory

CGO_ENABLED=1 CGO_CFLAGS=-mmacosx-version-min=13.0 GOOS=darwin GOARCH=amd64 go build -ldflags "-X github.com/crc-org/vfkit/pkg/cmdline.gitVersion=v0.6.4-27-gd125afd-dirty" -o out/vfkit-amd64 ./cmd/vfkit
# github.com/crc-org/vfkit/cmd/vfkit
/opt/homebrew/Cellar/go/1.27.1/libexec/pkg/tool/darwin_arm64/link: running cc failed: exit status 1
/usr/bin/cc -arch x86_64 -m64 -Wl,-headerpad,1144 -o $WORK/b001/exe/a.out -Qunused-arguments /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/go.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000000.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000001.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000002.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000003.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000004.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000005.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000006.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000007.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000008.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000009.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000010.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000011.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000012.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000013.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000014.o /var/folders/l0/rh4v8_j54k37h2w320__7d1h0000gn/T/go-link-2281718175/000015.o -O2 -g -framework CoreFoundation -framework IOKit -lresolv -O2 -g /Users/teuf/dev/vfkit/vendor/github.com/Code-Hex/vz/v3/internal/vzbridge/abi_amd64/libBridge.a -L/usr/lib/swift -framework Foundation -framework Virtualization -framework Cocoa /Users/teuf/dev/vfkit/vendor/github.com/Code-Hex/vz/v3/internal/vzbridge/abi_amd64/Virtualization.tbd -O2 -g -lpthread -framework CoreFoundation -framework Security
clang: error: no such file or directory: '/Users/teuf/dev/vfkit/vendor/github.com/Code-Hex/vz/v3/internal/vzbridge/abi_amd64/libBridge.a'
clang: error: no such file or directory: '/Users/teuf/dev/vfkit/vendor/github.com/Code-Hex/vz/v3/internal/vzbridge/abi_amd64/Virtualization.tbd'

make: *** [out/vfkit-amd64] Error 1

@Code-Hex

Code-Hex commented Oct 1, 2026

Copy link
Copy Markdown
Owner Author

Thanks @AkihiroSuda and @cfergeau for your feedback and testing.
In #241, I’ve set Swift aside for now and switched to purego bindings with Objective-C helpers built from source. Removing the prebuilt binaries addresses the distribution concerns around Homebrew and fixes the missing archive issue with go mod vendor.

Could you review #241 and try it with your usual workloads?

@Code-Hex Code-Hex closed this Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Slow cold build times from compiling Objective-C cgo bridge

3 participants