AFUN: Towards an Affordance Foundation Model for Functionality Understanding

Where + How for Functionality Understanding

Zhaoning Wang^1,*  ·  Yi Zhong^1,*  ·  Jiawei Fu²  ·  Henrik I. Christensen²  ·  Jun Gao^1,3

¹University of Michigan    ²UC San Diego    ³NVIDIA     ^*Equal contribution

TL;DR

RGB + text query + depth ──► AFUN model ──► mask + 3D motion curve

From a single RGB observation and a natural-language task description, AFUN predicts a task-conditional functional mask (where to interact) and a 3D post-contact motion curve (how to interact). It pairs frozen pretrained vision-language features with lightweight trainable modules for joint mask + motion prediction.

Currently, this repository provides inference-only code: load a trained AFUN checkpoint and run it on your own images. Depth is handled automatically — Depth-Anything-3 generates metric depth from a single RGB image, or lingbot-depth refines a noisy sensor (e.g. RealSense) depth map if you provide one.

Contents

• 📋 TODO
• 🔧 Installation
• 🚀 Quick Start
• 📦 Dataset & Benchmark
• 🙏 Acknowledgements
• 📚 Citation

---

📋 TODO

• Inference code & demo
• Training Code
• Dataset, Evaluation, and Benchmark — see 📦 Dataset & Benchmark
• Robot Demonstration Code
• Claude Agent md Files

---

🔧 Installation

1. Clone (with submodules)

git clone --recursive https://github.com/EricWang12/AFUN.git
cd AFUN
# (already cloned non-recursively? run: git submodule update --init --recursive)

2. Create the environment

bash install.sh
conda activate afun

NOTE: install.sh defaults to Python 3.11 and CUDA 13 (cu130). For a different CUDA builds, edit the file directly.

3. Get the checkpoint

The AFUN checkpoint afun.pt is hosted on Google Drive. Fetch it automatically — this also pre-caches the Hugging Face backbone weights (e.g. Qwen3-VL-8B, SAM3, Depth-Anything-3, lingbot, Sonata):

python scripts/prefetch_weights.py

Or download it by hand from the Google Drive link and place it at:

checkpoints/afun.pt

The backbone weights (Qwen3-VL-8B, SAM3, Depth-Anything-3, lingbot, Sonata) download automatically from Hugging Face on first run (cached); prefetch_weights.py fetches those too.

---

🚀 Quick Start

Run the demo with an RGB image, a text prompt, and (optionally) a sensor-depth folder:

python demo.py --rgb <path_to_rgb_image> --query <text_prompt> --depth-dir <path_to_depth_map>

--depth-dir would be a folder containing two files:

<depth-dir>/
├── depth.npy    # raw sensor depth — float32, millimeters, shape (H, W) matching the RGB
└── cam_K.txt    # 3×3 camera-intrinsics matrix

By default lingbot refines this depth; pass --no-refine to skip refinement and use the raw sensor depth as-is. Omit --depth-dir entirely and Depth-Anything-3 estimates depth from the RGB instead.

NOTE: Raw sensor depth without refinement generally does not work well with our model, as it is not trained on such data.

Example 1 — sensor depth (lingbot refines the bundled depth via --depth-dir):

python demo.py --rgb       examples/robot_toaster_oven/rgb.png \
               --query     "Open the toaster oven door" \
               --depth-dir examples/robot_toaster_oven

Example 2 — Depth-Anything-3 (no --depth-dir; DA3 estimates depth from the RGB):

python demo.py --rgb examples/kitchen_fridge/rgb.png --query "Open the refrigerator door"

Output (per run, under outputs/):

File	Contents
pred_seg.png	RGB + predicted affordance mask (red) + motion trajectory (yellow→blue, yellow = contact)
pred.npz	mask, motion parameters, intrinsics, confidence
pred_3d.html	interactive 3D point cloud + motion curve (omit with --no-3d)
meta.json / summary.csv / montage.png	run metadata (single / batch)

---

Run all bundled examples (model loads once, then loops over examples/):

python run_examples.py            # add --no-3d to skip the interactive 3D HTML

📦 Dataset & Benchmark

All data is released on Hugging Face under the AFUN-dataset organization (CC BY-NC-SA 4.0). Each data point is one folder: an RGB frame, its metric depth map, the ground-truth affordance mask, the ground-truth 3D motion curve, and a language instruction.

Dataset	What it is	Size
AFUN	Training set: 44,749 data points from SceneFun3D, RoboMIND, DROID, AgiBot World, RH20T, CALVIN, RLBench and VITRA human videos	231 GiB
AFUN_eval	The three evaluation sets of Table 3: afun_test (121), scenefun3d_test (721), robomind2_test (156). The ADE / FDE / CIM protocol is described in its README	9.9 GiB
AFUN_pool	The full 183,657-point pool that AFUN was curated from. Its 106,225 Ego4D-derived points ship as annotations only, with a script that rebuilds the frames from your own Ego4D download	363 GiB

pip install -U huggingface_hub
hf download AFUN-dataset/AFUN_eval --repo-type dataset --local-dir afun_eval    # 9.9 GiB — start here
hf download AFUN-dataset/AFUN      --repo-type dataset --local-dir afun_train   # 231 GiB

Each dataset's README documents the folder layout, the trajectory.json schema, and a loading snippet.

---

🙏 Acknowledgements

This project builds on Qwen3-VL, SAM3, Depth-Anything-3, lingbot-depth, and Sonata. We thank the authors for releasing their code and models.

---

📚 Citation

@article{wang2026afun,
  title   = {{AFUN}: Towards an Affordance Foundation Model for Functionality Understanding},
  author  = {Wang, Zhaoning and Zhong, Yi and Fu, Jiawei and Christensen, Henrik I. and Gao, Jun},
  journal = {arXiv preprint arXiv:2606.02551},
  year    = {2026},
}