duperemove: add new package

[ktgeek]

Maintainer: me
Compile tested: x86_64 / master on 3/18/2025
Run tested: x86_64 / 24.10 on 3/18. tested by deduping live btrfs filesystem

Description:
duperemove is a simple tool for finding duplicated extents and submitting them for deduplication. It
will de-deup results on btrfs.

[GeorgeSapkin]

In the commit description:

- will de-deup results on btrfs.
+ will de-dupe results on btrfs.

[kayhadrin]

@ktgeek, landing this anytime soon? :-)

[GeorgeSapkin]

@ktgeek there were some recent changes to the sqlite3 package (it's listed as a dependency). Does this package still work with them?

@kayhadrin you could help by testing this PR and reporting your findings: target, OpenWrt version, etc.

[kayhadrin]

@GeorgeSapkin I actually don't use OpenWRT directly. I was hoping to use dupermove via Entware since it's based on OpenWRT's package manager. That's assuming that OpenWRT packages are shared with Entware by default, but my cursory search seems to tell me the contrary...

[ktgeek]

@GeorgeSapkin Since I don't have a system running main right now, I cherry-picked the sqlite changes over to my personal 24.10 fork and built and tested at runtime against those. (I'll have a spare system for testing off main starting next week.) Ran without a problem and I specifically made sure I used the runtime flag that i know uses sqlite.

Per @kayhadrin's question above, I thought I was waiting for some else to merge. If there's any action I need to take, let me know. (After I post this, I will rebase to the tip of packages.) Although, looking at the bottom of my PR here, I do have a message of "No applicable reviews submitted by reviewers with write access."

[BKPepe]

This is deprecated SPDX License Identifier.

[ktgeek]

fixed

[GeorgeSapkin]

Still an issue:

• https://spdx.org/licenses/

[ktgeek]

I keep getting build errors before and unrelated to the pull request. That's why I keep rebasing. If someone has a better idea, please let me know. This last build error was a Bus Error on installing libsqlite3, long before it even gets to my changes.

[GeorgeSapkin]

I tried to build and run the main binary in QEMU. I don't have btrfs, but wanted to see if it at least runs. I'm getting this error: sh: lscpu: not found. Is some dependency missing perhaps?

To that end, consider adding a CI test (you can check other packages for an example), so these types of errors can be caught there. Since there are multiple binaries, I'd assume all would need to be tested.

[ktgeek]

I tried to build and run the main binary in QEMU. I don't have btrfs, but wanted to see if it at least runs. I'm getting this error: sh: lscpu: not found. Is some dependency missing perhaps?

Yep, it sure is. I had no idea, as it's not listed on its project page and I tend to have lscpu on my custom builds. Doing a strings on the duperemove binary showed lscpu as a string in the binary and I haven't had the chance to touch the source yet.

Also, just for future reference, it'll still run and find duplicate blocks on non-btrfs (or non-xfs) it just can't do anything about them.

All that said, its not why the build keeps failing, but it is a very good find. Thank you for you that.

To that end, consider adding a CI test (you can check other packages for an example), so these types of errors can be caught there. Since there are multiple binaries, I'd assume all would need to be tested.

Will do!

[ktgeek]

I haven't added the CI tests yet, doing update (and testing) on my local system still. Been busy the past month.

[ktgeek]

This project is still backburned, but one I want to complete as I am using this package. My non-work time has been limited still.

[ktgeek]

Moved this into draft until I can get this off my back burner

[ktgeek]

Finally pulled this off the back burner.

• Confirmed duperemove throws an error if lscpu isn't present, but turns out to be non-fatal. That said, still not good, so lscpu has been added as a dependency.

• Added some basic smoketests for CI.

• I figured out that only the header is used from libxxhash, so that's been moved to a build dependency.

[openwrt-ai]

duperemove: Use the package-specific test.sh
Usage: btrfs-extent-same len file1 loff1 file2 loff2
duperemove: Test failed

The x86_64 runtime test fails right here with no output; the mips_24kc, arm_cortex-a15, aarch64 and i386 jobs also fail. A likely cause is that the CI container's $PWD sits on overlayfs, which also has an anonymous device (major 0), so moving off /tmp does not avoid the check described in the comment above. Drop -q so the log shows why duperemove exits, and pick a scan target that does not need a real block device.

---

Generated by Claude Code

[ktgeek]

-q is gone, and the scan now runs with --debug, so the log shows what duperemove did.

A scan target that doesn't need a real block device isn't possible with duperemove 0.15.2. scan_file() calls check_file(), which calls get_uuid() on the top-level target whatever state the hashfile is in. Off btrfs, get_uuid() rejects any stx_dev_major == 0 filesystem (tmpfs, overlayfs). Otherwise it needs libblkid and /proc/self/mountinfo to resolve a UUID for the backing device, which the unprivileged test container can't do for the /ci bind mount either. A rejected target is skipped silently and duperemove still exits 0, which is why the log showed nothing.

When the files land in the hashfile, test.sh runs the full check. When they don't, it requires duperemove's own refusal message (unsupported filesystem, no blkid UUID, or no mountinfo entry), so a real scan regression still fails.

[openwrt-ai]

Reviewed 1 new commit; no new issues found.

---

Generated by Claude Code