[RFC] One file representation for DAV and PHP: node attribute providers

Tip

Help move this idea forward

Use the 👍 reaction to show support for this feature.
Avoid commenting unless you have relevant information to add; unnecessary comments create noise for subscribers.
Subscribe to receive notifications about status changes and new comments.

Is your feature request related to a problem? Please describe.

The only complete representation of a file is the WebDAV one. Its attributes come from Sabre plugins during a PROPFIND: FilesPlugin, SharesPlugin, TagsPlugin, CommentPropertiesPlugin, SystemTagPlugin, files_reminders, files metadata, and whatever apps add on top. The frontend turns that response into a File/Folder with resultToNode() from @nextcloud/files, and apps add their props on that side with registerDavProperty() (server alone registers 12: nc:sharees, oc:share-types, nc:reminder-due-date, nc:system-tags...).

There is no PHP equivalent. Code that returns files without going through DAV builds its own subset, each with its own shape:

apps/files/lib/Helper.php:71 formatFileInfo(): parentId, mtime * 1000, shareOwner, isShareMountPoint, mountType
apps/files_sharing/lib/Controller/ShareAPIController.php:204: storage_id, file_target, item_mtime
apps/files_trashbin/lib/Helper.php:104: a hand-built mimetype, etag, permissions

None of these carry the attributes plugins add, and none can be turned into the same Node the Files app uses. Each app that needs file data either re-implements part of it or sends a PROPFIND:

Talk can't PROPFIND its attachments folder, it is too large. spreed/src/composables/useViewer.js:85 maps its own message data to a viewer file by hand: it converts the permission bitmask to the DAV string (CKGWDR) and derives hasPreview from preview-available === 'yes'.
Assistant sends one PROPFIND per file to get a Node for the Viewer (Properly build the node we pass to the Viewer assistant#675), which doesn't scale for huge folders or accounts with many shares.
Viewer builds 8 nodes by hand, Text 2.

It also costs performance. Fetching a few known files means a DAV SEARCH or one PROPFIND per file, through the full Sabre stack and the per-node prop handlers. While working on #2742 (live refresh from notify_push), a SEARCH for 10 file ids took 2.4 s on a test account with 5,000 incoming shares, mostly spent in searching all mounted storages and in per-node share lookups.

Describe the solution you'd like

One registry that both DAV and PHP read from.

Apps register providers with the attributes they serve, in Clark notation like the DAV props, so the server only loads a provider when one of its attributes is requested:

$context->registerNodeAttributeProvider(SharesAttributeProvider::class, [
    '{http://owncloud.org/ns}share-types',
    '{http://nextcloud.org/ns}sharees',
]);

interface INodeAttributeProvider {
    /** Load what is needed for all the nodes of a response at once */
    public function preload(INodeAttributeContext $context, array $nodes, array $attributes): void;
    /** @return array<string, mixed> scalars, or NodeAttributeList for lists */
    public function getAttributes(INodeAttributeContext $context, Node $node, array $attributes): array;
}

A public serializer, INodeSerializer::serialize(INodeAttributeContext $context, array $nodes, ?array $attributes = null): array, returns the NodeData shape @nextcloud/files already uses, attributes keyed by local name like resultToNode() does today. Lists use the same shape the webdav client produces ({ 'share-type': [0, 3] }).
A generic Sabre plugin serves registered attributes on propFind when no other plugin handled them, and calls preload() on preloadCollection and preloadProperties. Existing plugins keep working and move over one by one, and SEARCH results get batched loading for every migrated attribute.
The registered attribute names reach the frontend through initial state, so @nextcloud/files requests them without registerDavProperty(). A counterpart to resultToNode(), for example dataToNode(), builds a Node from the serializer output.
Public pages are a requirement, not a follow-up. Public share pages have no user, only a share token, so providers get an INodeAttributeContext holding either the user or the share, and the serializer and DAV adapter work the same on remote.php and public.php.
Providers load everything in preload(), and getAttributes() doesn't query per node. A test helper asserts the same number of queries for 1 and 100 nodes, so the N+1 we see in SEARCH today (about 13 queries per node) doesn't come back.
Rollout:
- Phase 1: the registry, the serializer and the DAV adapter, with the five most used attributes: favorite, etag, has-preview, share-attributes and share-types. share-attributes is in on purpose, so the DAV/PHP parity tests for permission related attributes exist from the start.
- Phase 2: the remaining server plugins.
- Phase 3: props registered by apps (files_lock, Photos, Text).

Describe alternatives you've considered

Keep DAV as the only source and fetch with SEARCH or PROPFIND. That is what we do today, with the cost above.
Make formatFileInfo() the shared format. It doesn't know about plugin attributes, so the gap stays.

Additional context

What to migrate first. fileid, permissions, size, getlastmodified, getcontenttype, owner-id and displayname become top-level Node fields in resultToNode() and are needed by every view, so they are the core set. Among the other attributes, ranked by reads of node.attributes.* in server and in Photos, Text, Deck, Talk, Viewer, Office, Collectives, Activity, Mail, Recognize, files_lock, Forms and files_pdfviewer:

Attribute	Reads	Where	Used for
`favorite`	37	files, files_sharing, photos	star, favorites-first sorting, favorite action
`etag`	17	core, files, files_sharing, files_versions, photos, viewer	preview cache-busting, change detection
`has-preview`	12	core, files, files_sharing, viewer, photos, mail; Talk maps its own	thumbnails everywhere
`share-attributes`	12	files, files_sharing, files_versions, mail, richdocuments	download restrictions on shared files
`share-types`	12	files, files_sharing, spreed, activity	sharing icons and status
`mount-type`	9	files, files_sharing, files_versions, photos	which actions apply to shared, external or groupfolder nodes
`sharees`	8	files_sharing	"Shared with" avatars
`owner-display-name`	6	files, files_sharing	owner of shared files
`hide-download`	3	files; Talk's backend sends its own (`'yes'`/`'no'`)	hiding downloads on restricted shares
`is-mount-root`	2	files	rename, move and delete rules on mount roots
`e2ee-is-encrypted`	2	text, viewer	end-to-end encrypted folders

Apps also register their own props with registerDavProperty(), and would become providers too: files_lock (nc:lock, nc:lock-owner, nc:lock-owner-displayname, nc:lock-owner-editor, nc:lock-owner-type, nc:lock-time), Photos (nc:metadata-photos-*, nc:metadata-blurhash, nc:metadata-files-live-photo), Text (nc:rich-workspace-flat, nc:rich-workspace-file-flat).

share-attributes, hide-download and share-permissions drive download and permission decisions, and Talk's backend already sends its own hide-download in message parameters (spreed/src/composables/useMessageInfo.ts:115). A mismatch between the DAV and PHP output there is a security issue, not a display bug, so they need tests comparing both outputs.

Open questions:

Attribute values: XML props are parsed by the webdav client into nested shapes (for example share-types becomes { 'share-type': [0, 3] }). The serializer either reproduces them, or both sides move to one clean shape, which touches every attribute consumer.
JSON keys are local names, as in resultToNode(), so two apps registering the same local name in different namespaces would collide. The registry would refuse the second one.
Permissions, share permissions and hide-download depend on the user's mount or on the public share, so the serializer has to work on user or share scoped nodes only, never on raw cache entries.
Existing registerDavProperty() apps need a transition period where both keep working.

@icewind1991

@icewind1991 @susnux @artonge @salmart-dev @cristianscheid @CarlSchwan: you know the filesystem, DAV, search and @nextcloud/files code this touches. A review of the proposal would help 🙏

@julien-nc @Antreesy @juliusknorr @mejo-: Assistant, Talk, Text, files_lock, Deck and Collectives build file objects by hand or send a PROPFIND per file today. Your expertise is also highly appreciated 🙇

I think allowing us to move away from webdav entirely for some specific use cases would be very effective and increase our performances. Like getting a specific file by its id without having to do a PROPFIND or a SEARCH

Note only, as not yet fully read:

SEARCH for 10 file ids took 2.4 s

Its likely far cheaper to instead do a PROPSTAT per file (usually only a low number of files change for a user).
This does not need the server to do a full search but only need to setup the filesystem for that id (will is currently being improved performance-wise with partial share setup we work on for 36).

@susnux

@susnux right! I tried it on my heavy test account (70k files in the root, 5k incoming shares, MariaDB).
A Depth 0 PROPFIND on a single file takes ~180 ms, and 10 of them in parallel ~630 ms. The SEARCH for the same 10 ids takes 2.4 s. So yes, way cheaper. Yet, even 180ms is a bit too much for my taste. And it's only one file on a not-so-busy test instance...

There are two things I'm not sure how to handle yet:

notify_push only gives us ids. For files we already list that's fine, we know their path. But for a new file or a rename we only have the id, and there's no DAV route by id. So we'd still need some cheap lookup by id for those.
And for incoming share roots, Depth 0 and the folder listing don't return the same share-types/sharees. Depth 0 includes the incoming share, the listing doesn't. So refreshing a share root that way would change its sharing icon.

@susnux

I dug into this a bit more, so here are some numbers.

The question: when a few files change, what's the fastest way for the Files app to refresh just those? @susnux suggested one PROPFIND per file instead of a SEARCH. I also tried a new endpoint, the Nodes API, that returns all the changed files in a single request, built on the serializer from this RFC.

The setup: a heavy account (70k files, 5.5k shares received), the same data for all three options, median of 3 runs.

Changed files	SEARCH	PROPFIND per file	Nodes API
1	377 ms	186 ms	184 ms
5	2,359 ms	390 ms	238 ms
10	2,386 ms	621 ms	202 ms
20	2,525 ms	1,231 ms	217 ms

What I take from it:

SEARCH is slow as soon as you're in a folder with many shares below it. In a plain subfolder it's fine.
PROPFIND per file is good for one or two files, then grows with every file.
The Nodes API stays around 200 ms whatever the count. It also returned the same data as a PROPFIND on 13 of the 14 files I compared. The 14th file shows a known DAV quirk.

So I think a "get these files by id" endpoint on top of this RFC is worth having. The next part is up as a draft: #65400.

In addition to Talk, Assistant, Viewer and Text listed above, I looked for other places that build their own file objects today. These would switch to the serializer right away:

Recommendations (dashboard widget and the "Recommended files" section): the backend returns its own shape (RecommendedFile::jsonSerialize()), and the frontend works out previews and links by itself.
Files sharing views (shared with you, with others, pending, deleted): files are rebuilt from the share API in ocsEntryToNode(), sharees and preview flags included.
External storage: mount roots are built from the mounts API in entryToFolder().
Deck: card attachments get their file data from FilesAppService, with its own key names.

Each one has its own mapping to maintain today. With the serializer they would all get the same file data the Files app gets, from the same code.

@icewind1991

btw @icewind1991, I noticed that every Basic auth request spends ~126 ms in the postLogin setupForUser() on my heavy test account (5.5k received shares): a Depth 0 PROPFIND on one file takes 177 ms, 51 ms with that call disabled. Desktop and mobile clients pay it on every request. Is that covered by the partial share setup work for 36, or worth its own issue?

While working on #2742 (live refresh from notify_push),

For that use case, I would suggest just keeping it simple and look when one of the changed fileids is either the current folder or one of the files inside it, and then reload the folder with a PROPFIND.

(This proposed feature might still be good to have regardless ofc)

@icewind1991

14. then reload the folder with a PROPFIND

that is not possible @icewind1991, some customers have very expensive setups where the propfind is already on degraded performances and takes a few seconds. Until we have a somewhat really speedy dav response time, I don't wanna do that.
(even on our cloud, my home takes 1.5s to complete the propfind)

But thats done in the background, so the user does not notice there is something loading and thus does not know something is slow.
E.g. if you sync from desktop client and a file is uploaded then it might take 10s instead of 5s for the file to appear in the browser window thats open, no?

But thats done in the background, so the user does not notice there is something loading and thus does not know something is slow.
E.g. if you sync from desktop client and a file is uploaded then it might take 10s instead of 5s for the file to appear in the browser window thats open, no?

Well yes, but if changes takes 10s to come, it's still not useful. It's called LIVE folder sync, not DELAYED folder sync 😁😁😁

And from a server performance view, blocking a php thread because us engineers deemed the performance bad but not noticeable, isn't really a good solution from an ethical pov. We should always aim for better, not being satisfied with mediocre 👀

This RFC is beyond this single scope of folder sync though, so please let's not focus on this 😉

for a new file or a rename we only have the id, and there's no DAV route by id. So we'd still need some cheap lookup by id

I know at least integration_openproject that would benefit from an endpoint to get a file representation from its ID. It's been a while they are using a custom endpoint: https://github.com/nextcloud/integration_openproject/blob/743c1b34cead5148acc2b4a879344800082fe033/lib/Controller/FilesController.php#L114

I don't know the filesystem internals enough to judge on the proposal. If that introduces a more convenient and more performant way to fetch nodes data, I'm all for it.

I think allowing us to move away from webdav entirely for some specific use cases would be very effective and increase our performances. Like getting a specific file by its id without having to do a PROPFIND or a SEARCH

Do you even think it would make sense to avoid using WebDav in the Files app's frontend in some cases? Like refreshing the data of one specific file item.

For the assistant's use case, we can have multiple image results for one task. So it would be nice to give a list of nodes to the viewer so the user could cycle though them inside the viewer. It is indeed an issue that we would have to perform one potentially expensive PROPFIND per file.

20. Do you even think it would make sense to avoid using WebDav in the Files app's frontend in some cases? Like refreshing the data of one specific file item.

I kind of think it does. Webadv is nice because of how well known and developed the standard is. But for our custom apps that doesn't have need to have such a versatile api, it's perfectly fine. As long as we're consistent between the data. I think content parity is paramount here, which is why I think this RFC makes sense :)

I also think such an PHP API makes sense, I am not sure about the serializer though. Because in my opinion there must be only one format used, otherwise we will have big problem in the future maintaining two different APIs.
So from my point of view it makes sense to have such an API in the backend, for working with files.
Meaning extending the existing Node API we have in the backend.

But it must not be a second frontend facing API, so only WebDAV should stay the single point of truth for frontend related queries.
Maybe we will in the future have a look a JMAP, but as we already discussed many times in the past another frontend facing API is nothing we want to have and maintain.

As said for the backend itself to properly handle metadata in it such an API makes sense and we can plug that into Sabre.
But it must be well planned and thought trough, it will be something that has to be stable for many years, so a proper planning process has to be done.

But it must not be a second frontend facing API, so only WebDAV should stay the single point of truth for frontend related queries.

This is where I disagree..we already have many apps doing their own serving of files through ocs Apis.
What do you make of those ?

@nickvergessen

Assistant, Talk, Text, files_lock, Deck and Collectives build file objects by hand or send a PROPFIND per file today

From Talk frontend side:

PROPFIND is still there for a legacy or feature opt-out support;
Rebuilding objects for Viewer - would be nice, if it would be a viewer compatible node object, with RFC + viewer library it would still be present in some form, i'm afraid
And we would still need to support legacy formats anyway for Talk Desktop client

From backend side:

there's a known performance limitation with preparing chat messages causing to mount filesystem and thus can be slow.
Should be improved with subfolders feature already, but can't tell if RFC will boost it more. cc @nickvergessen

@susnux

@susnux gave it bit more thoughts, I think we can meet in the middle.

The serializer doesn't add a new format: it returns the NodeData shape @nextcloud/files already builds from every PROPFIND, from the same providers that now serve the DAV properties. I compared both on a heavy account (Recent, Favorites, listings, single files, public links): identical for all 31 Files props. So the maintenance risk of two outputs drifting apart is covered by the providers being the only source.

For the Files frontend, I'm fine keeping WebDAV the single entry point. Fetching by id can live in DAV: a REPORT (or SEARCH) that resolves the ids directly, served by the same providers. The speed comes from resolving ids and loading attributes in batch, not from OCS, so we'd keep the gain. I'd move the /api/v1/nodes endpoint of #65400 to that.
Basically, we'll keep using WebDAV for usual global handling. But for very specific apps or apis (recommendations, attachments in comments, etc etc) it makes no sense to go through webDAV. Having a clean source of truth to use in php is required I'm afraid. And I'd rather have a proper API than every dev to their own cooking 😉

The serializer would stay a PHP API for apps that already return files from their own OCS endpoints (OpenProject, Deck, Recommendations, Talk...), so they return the same thing Files gets instead of their own mapping. That standardises existing APIs rather than adding one.

Talking about proper planing, well, this is it lol.
Thatere's a reaosn I drafted such a complete RFC with a proper implementation example to look at here #65397

@Antreesy for the Viewer, that's the goal: one node shape for everyone. And the mount cost you mention is the same one I found on every Basic auth request (~126 ms on a heavy account), asked icewind about it above.

Navigation Menu

Repository navigation

[RFC] One file representation for DAV and PHP: node attribute providers #65395

Help move this idea forward

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[RFC] One file representation for DAV and PHP: node attribute providers #65395

Description

Help move this idea forward

Activity

skjnldsv commented on Oct 11, 2026

susnux commented on Oct 11, 2026

skjnldsv commented on Oct 11, 2026

skjnldsv commented on Oct 11, 2026

skjnldsv commented on Oct 11, 2026

skjnldsv commented on Oct 12, 2026

icewind1991 commented on Oct 12, 2026

skjnldsv commented on Oct 12, 2026

susnux commented on Oct 12, 2026

skjnldsv commented on Oct 13, 2026

julien-nc commented on Oct 16, 2026

skjnldsv commented on Oct 16, 2026

susnux commented on Oct 16, 2026

skjnldsv commented on Oct 16, 2026

Antreesy commented on Oct 16, 2026

skjnldsv commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions