sync0 is a decentralized P2P file sharing, real-time synchronization and multi-source distribution tool used to replace rsync - the measured speed of transcontinental WAN is up to rsync's 5.89× (single point geometric average speed 2.67×, P2P dual-source geometric average speed 3.16×), and transoceanic high-latency link P2P acceleration is up to 7.68× (Full-link 24-scenario comprehensive geometric mean speed 2.56×).
Each machine is both a server and a client: no central server or public network address is required, automatic hole punching is performed based on distributed network public key addresses, supporting direct transfer between two machines, multi-sharing group permission hosting, on-demand multi-selection of subdirectories, millisecond-level real-time incremental synchronization, and P2P multi-source concurrent acceleration.
- Install
- Get started quickly
- 1. Zero-configuration direct transmission between two machines (
send/recv) - 2. Post sharing and on-demand pull / upload (
share/pull/push) - 3. Real-time subscription seeding and multi-machine two-way synchronization (
sub/group) - 4. Team multi-sharing group and member permission configuration (
.ntconfiguration)
- 1. Zero-configuration direct transmission between two machines (
774.98 1,404.10 For more link data and reproduction scripts, see Performance Measurement Report.
Go to GitHub Releases to download the .tar.xz compressed package corresponding to the system (supports Linux, macOS, Windows, Android, iOS, FreeBSD), unzip and put into PATH:
774.98 292.23 2.65× For more link data and reproduction scripts, see Performance Measurement Report.
No need to configure a sharing group, directly transfer files or directories between two machines at high speed:
774.98
sync0 recv 0.0.0.0:9000 ./backup
Direct transmission from the sender (optional --delete mirror to clean up excess files, --rate 50MB speed limit)
sync0 send ./project 192.168.1.10:9000 For more link data and reproduction scripts, see Performance Measurement Report.
774.98
Server: Publish the sharing group with one line of command (output the IP-free iroh node address after startup)
sync0 share dataset:./data
sync0 pull dataset:// ./local_data -p
Client: Pull multiple selections on demand (only synchronize specified single files and subdirectories)
sync0 pull dataset://docs/readme.md,assets/icons,src/core ./local_data -p
sync0 push dataset://releases/v1 ./build_output -p For more link data and reproduction scripts, see Performance Measurement Report.
After establishing a persistent connection, whenever any file creation, modification, deletion, permission (mode), or modification time (mtime) change occurs on the remote end, the local directory is automatically updated in real time within milliseconds (creating/modifying large files only pulls differential blocks in place, deleting files automatically cleans up local files and empty directories, and modifying permissions or timestamps takes effect in place with zero data block transfer):
sync0 sub(One-way real-time mirror + automatic seeding): The local directory follows remote additions, deletions, and modifications in real time, while automatically serving as a P2P seed source to accelerate concurrent downloads for other nodes.sync0 group(Multi-machine peer-to-peer two-way real-time synchronization): When any machine in the group adds, deletes, or modifies files, all other online nodes synchronize updates in real time and seed each other.
774.98
Subscribe to the entire sharing group: remote additions, deletions, and modifications are synced locally in real time and seeded automatically
sync0 sub dataset:// ./local_mirror -p
On-demand multi-select subscription: only sync and seed specified single files and subdirectories in real time
sync0 sub dataset://models/llama.onnx,configs ./local_mirror -p
Multi-machine peer group two-way real-time synchronization (also supports full or comma multi-select paths)
sync0 group team-docs:// ./docs -p For more link data and reproduction scripts, see Performance Measurement Report.
The default configuration directory is ~/.config/sync0 (can be switched via -c <directory>), using the minimalist NestedText (.nt) format:
- Supports restricting public network reading (only authorized users can read): As long as the sharing group is configured with a
users:member list (or the command line specifies--allow-r/--allow-rw/--allow-w/--auth), the system automatically turns on private authentication (auth: true) and prohibits public network anonymous node enumeration or reading, and P2P Subscribing to seeding nodes will also inherit the whitelist simultaneously, allowing only authorized users to read; sharing groups that are not configured withusersare public and read-only by default (if you want to be public and read-only and open to uploads only to specified members, you can explicitly setauth: false). - Fine-grained read-write and subdirectory delivery permissions: Members are read-only by default (
rdoes not need to be written explicitly), and supportr(read-only),w(only write to the drop box, not readable),rw(read-write) andwrite_prefix(limit the subdirectory prefix allowed for uploading).
Member configuration example (~/.config/sync0/users/):
# ~/.config/sync0/users/alice.nt (default read-only, speed limit 80MB)
id: 8J9aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789ab_c
rate: 80MB
For more link data and reproduction scripts, see [Performance Measurement Report](../../doc/zh/benchmark.md).
```yaml
# ~/.config/sync0/users/bob.nt (read and write permissions, only allowed to upload to specified subdirectory)
id: 7K8bCdEfGhIjKlMnOpQrStUvWxYz0123456789de_f
perm: rw
write_prefix:
- uploads/bob
- shared/common
For more link data and reproduction scripts, see [Performance Measurement Report](../../doc/zh/benchmark.md).
```yaml
# ~/.Config/Sync0/Users/Charlie.Nt (Only Write Delivery Permission, Reading Is Prohibited)
id: 9M0cDeFgHiJkLmNoPqRsTuVwXyZ0123456789gh_i
perm: w
For more link data and reproduction scripts, see [Performance Measurement Report](../../doc/zh/benchmark.md).
**Sharing group configuration example (`~/.config/sync0/config.nt`):**
```yaml
users_dir: ./users
groups:
# Not configured users: default public network read-only
dataset:
path: ./data
rate: 50MB
# Configured users: automatically restrict public network reading (only alice and bob can read, charlie can only deliver and upload)
team-vault:
path: ./vault
users:
- alice
- bob
- charlie
For more link data and reproduction scripts, see [Performance Measurement Report](../../doc/zh/benchmark.md).
Start the service (load the configuration file or directly use command line parameters to restrict public network reading and authorize members):
**774.98**
# Load ~/.config/sync0/config.nt to start
sync0 share --region SHA
# Or publish a private sharing group through pure command line (specify --allow-* to automatically restrict public network reading and only authorized users can access it)
sync0 share team-vault:./vault \
--allow-r 8J9aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789ab_c \
--allow-rw 7K8bCdEfGhIjKlMnOpQrStUvWxYz0123456789de_f:uploads/bob,shared/common \
--allow-w 9M0cDeFgHiJkLmNoPqRsTuVwXyZ0123456789gh_i
For more link data and reproduction scripts, see [Performance Measurement Report](../../doc/zh/benchmark.md).
### 5. Node address, business card and multi-language
**774.98**
# View the local IP-free node address (key is automatically generated for the first run)
sync0 id
# Generate Ed25519 self-signed node business card (nickname, city code, email, website, introduction)
sync0 profile -u alice --region SHA --email alice@example.com -b "Shanghai node"
# There are 75 built-in languages, which follow the system language by default and can be switched through environment variables
SYNC0_LANG=en sync0 --help
For more link data and reproduction scripts, see [Performance Measurement Report](../../doc/zh/benchmark.md).
## Performance and core features
### Cross-continental WAN actual measurement (Comparison `rsync`)
Actual measurement environment: United States `c1` → Europe `n1` (RTT 65 ms), the data set is Linux kernel `linux-7.3.3`:
#### 1) A la carte mode vs `rsync`
| Data set | Size | `sync0` Single Point<br>(Mbps) | 1,404.10 | Speedup (`sync0` / `rsync`) |
`linux-7.3.3/` (95,979 small files)
| `linux-7.3.3.tar.xz` (not compressible) | **486.19** | **236.48** | 108.74 | **2.17×** |
| `linux-7.3.3.tar` (compressible) | `linux-7.3.3.tar.xz`(not compressible) | **843.83** | 385.06 | **2.19×** |
| `linux-7.3.3.tar` (local change increment) | `linux-7.3.3.tar.xz`(not compressible) | **8,014.15** | 1,361.72 | **5.89×** |
| `linux-7.3.3/` (95,979 small files) | `linux-7.3.3.tar`(compressible) | **627.58** | 347.80 | **1.80×** |
| **Geometric mean** | — | **1,000.91** | 375.26 | **2.67×** |
#### 2) P2P dual-source mode vs `rsync`
| Data set | Size | `sync0` P2P Dual Source<br>(Mbps) | 1,404.10 | Speedup (`sync0` / `rsync`) |
`linux-7.3.3/` (95,979 small files)
| `linux-7.3.3.tar.xz` (not compressible) | **486.19** | **330.09** | 108.74 | **3.04×** |
| `linux-7.3.3.tar` (compressible) | `linux-7.3.3.tar.xz`(not compressible) | **1,551.86** | 385.06 | 107.05 |
| `linux-7.3.3.tar` (local change increment) | `linux-7.3.3.tar.xz`(not compressible) | **5,474.17** | 1,361.72 | **4.02×** |
| `linux-7.3.3/` (95,979 small files) | `linux-7.3.3.tar`(compressible) | **706.56** | 347.80 | **2.03×** |
| **Geometric mean** | — | **1,186.42** | 375.26 | **3.16×** |
> Note: In P2P dual-source mode, European nodes `s1` collaborate for seeding, and the cross-continental backbone RTT is 65 ms.
### Core product features
- **Public key addressing automatic hole punching**: No need for a central server and public network address, automatic NAT hole punching based on the distributed network public key address.
- **Millisecond-level real-time synchronization and metadata awareness**: Based on native file event monitoring (`FSEvents` / `inotify` / `ReadDirectoryChangesW`), real-time push of file additions, deletions, POSIX permissions (`mode`) and millisecond-level modification time (`mtime`); zero data block transmission when only permissions or time are modified.
- **BT-style simultaneous download and upload and regional affinity scheduling**: The downloading node does not need to wait for the full amount to be downloaded. All the block intervals (`Op::Have`) that have passed the 128-bit hash check are immediately locked in the memory/temporary pool for other nodes to pull concurrently. Together with dynamic PEX node exchange and slow node half preemption (Work-Stealing), the cluster throughput is maximized.
- **Anti-malicious node tampering and cluster poisoning immunity**: authoritative source list `root_hash` Self-consistent locking + mandatory hardware acceleration `gxhash128` digest verification before each block is placed on the disk and externally seeded; if a tampered block is found, the millisecond-level circuit breaker will blackout the malicious node and automatically recover from the honest source.
- **FastCDC in-situ increment and small file aggregation**: Only differential blocks are transferred for partial modification of large files and placed on disk; massive small file memory packaging and prefix differential compression eliminate file-by-file round-trip overhead.
- **Self-healing after power outage and resumption after restart**: Redis-like RDB + AOF hybrid format (`.s0_subs.bin`) is used to persist the seeding status and file list, daily changes are appended in microseconds, and are automatically rewritten and organized regularly. After the server is restarted, offline changes are automatically pushed to the seeding node.
## Technical architecture and documentation
- **Dual plane decoupling protocol**:
- **Control and real-time change plane (`iroh` QUIC / TCP, first byte `PROTO_VERSION` version handshake)**: Responsible for Ed25519 business card verification, permission verification, Swarm seed discovery, and support for `u64::MAX` level mass change streaming batching and `mzstd` compressed `SubPush` real-time push (`FsOp::Upsert` / `FsOp::Remove` / `FsOp::Meta` three primitives).
- **Data distribution plane (native UDP + `compio` asynchronous I/O)**: 20B compact header, 18B TACK periodic feedback, BBRv3/Copa hybrid congestion control, RACK-TLP packet loss recovery, first-block Shannon entropy adaptive Zstd compression bypass, and `gxhash128` block end-to-end verification.
- **Learn more**:
- [Performance measurement report and reproduction script](https://github.com/webc-site/sync0/blob/main/doc/zh/benchmark.md)
- [System architecture design](https://github.com/webc-site/sync0/blob/main/doc/zh/design_v4.md)
- [Binary Wire Protocol and Persistence Specification](https://github.com/webc-site/sync0/blob/main/doc/zh/proto.md)
- [Identity authentication, authority control and integrity verification](https://github.com/webc-site/sync0/blob/main/doc/zh/crypto.md)