Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,9 +182,9 @@ WAVEFLOW_METADATA_LASTFM_KEY=… # your own Last.fm API application's key

| Variable | Default | What it does |
| --- | --- | --- |
| `WAVEFLOW_METADATA_PROVIDERS` | empty | The switch: an ordered, comma-separated list. Empty is off. Today the only provider is `lastfm` |
| `WAVEFLOW_METADATA_PROVIDERS` | empty | The switch: an ordered, comma-separated list. Empty is off. The providers are `lastfm` (biographies and similar artists) and `deezer` (artist portraits, no key) |
| `WAVEFLOW_METADATA_LASTFM_KEY` | — | Required when `lastfm` is listed; the server refuses to start without it. No key ships in the binary |
| `WAVEFLOW_METADATA_IMAGES` | on | `off` keeps the lookups and refuses artist image downloads |
| `WAVEFLOW_METADATA_IMAGES` | on | `off` keeps the lookups and refuses artist image downloads; `deezer` is then not asked at all, since portraits are all it is asked for |
| `WAVEFLOW_METADATA_TTL_DAYS` | 30 | How long an answer with content is kept before it is asked again |
| `WAVEFLOW_METADATA_MISS_TTL_DAYS` | 7 | How long "the provider knows nothing about this artist" is kept |
| `WAVEFLOW_METADATA_RATE_PER_SEC` | 4 | Requests per second, per provider |
Expand All @@ -201,9 +201,22 @@ request carries the artist's name and your key, and no redirect is followed.
Requests are spaced to the rate above, and simultaneous visits to one artist
share one request.

With `deezer` listed, also `api.deezer.com`, which needs no key, and then the
picture it names — but only from Deezer's image hosts
(`e-cdns-images.dzcdn.net`, `cdn-images.dzcdn.net`), only over https, only at
a public address (the address checked is the one connected to), only a JPEG,
PNG or WebP by its own bytes, and at most 10 MiB. A picture that fails any of
these is not fetched and the artist is remembered as having none until the
short TTL runs out.

**What it stores.** In `waveflow.db`, for each provider and each artist name
asked about: the biography, the ranked list of similar artists by name, whether
the provider knew the artist, and when to ask again. Nothing in it belongs to an
asked about: the biography, the ranked list of similar artists by name, the
portrait's hash, whether the provider knew the artist, and when to ask again.
The portrait itself goes in the artwork store beside the covers, and every
artist with that name shows it through the `coverArt` it already carries —
**back it up with the rest of `artwork_dir`**: unlike a cover, it cannot be
extracted again from your files, only downloaded again once the cache entry
expires. Nothing in it belongs to an
account, and nothing records who asked. A failed request stores nothing, so the
next visit asks again.

Expand Down
12 changes: 12 additions & 0 deletions docs/rfcs/RFC-007-library-event-stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,18 @@ over this server and **written nowhere**. An artist photo cannot change because
there never is one. The question reopens the day something writes it, and on
that day this is the line to re-read.

> **Amended 2026-10-16 (#268): an artist now emits one, for its portrait only.**
> [RFC-011](RFC-011-metadata-enrichment.md) decision 8 writes
> `artist.artwork_hash` — a portrait fetched from a metadata provider — so the
> case the desktop reserved exists. The rule is the album's: an `upsert` with an
> empty payload, emitted **only when the hash actually moved**, by the same
> `UPDATE … WHERE artwork_hash IS NOT ? RETURNING` pattern, so a refresh that
> fetched the same picture says nothing. Artist rows are per library, so a
> portrait shared by a name emits one event per row, each in its own library.
> `library_event.entity_type` admitted `'artist'` from the start, so no
> migration was needed. The counters stay derived and still emit nothing: the
> reasoning above holds for everything but the photo.

## Decision 7 — two bounds, because one alone is always wrong

> **Built.** See the *Implemented by* line in the header.
Expand Down
8 changes: 5 additions & 3 deletions docs/rfcs/RFC-011-metadata-enrichment.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@
InstaZDLL/WaveFlow#855 (the core's clients accept a client, an address and a
key), #270 (the pin moved onto them), #271 (the cache and the engine that
fills it), #272 (`getArtistInfo`, `getArtistInfo2` and the native artist
page read it), and the pull request closing #267 (`getSimilarSongs` and
`getSimilarSongs2` draw from it). Portraits are still to come. The *Status* field above never flips in this
project; this line is the one that says what is running.
page read it), #273 and #274 (`getSimilarSongs` and `getSimilarSongs2` draw
from it), and the pull request closing #268 (portraits from Deezer). The web
client and the compatibility campaign are still to come. The *Status* field
above never flips in this project; this line is the one that says what is
running.
- **Date**: 2026-10-14
- **Revised**: 2026-10-15, while implementing the cache (#265). Decisions 4, 6
and 10 rested on attachment points the code does not hold; each correction
Expand Down
10 changes: 10 additions & 0 deletions migrations-v2/20261016000000_metadata_artist_image.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
-- RFC-011 decision 8: the portrait a provider pointed at, once fetched.
--
-- The hash is kept on the provider's cache row as well as on `artist`, so an
-- artist re-identified by a PID spec change is linked again without a second
-- download. `ON DELETE SET NULL` is what makes that safe: the sweeper counts
-- `artist`, `album` and `track` as references and not this column, so an
-- artwork row nobody shows may go, and the cache row then forgets it rather
-- than naming a file that no longer exists. The next refresh fetches it again.
ALTER TABLE metadata_artist
ADD COLUMN image_hash TEXT REFERENCES artwork(hash) ON DELETE SET NULL;
12 changes: 11 additions & 1 deletion src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,8 @@ pub struct ScrobbleDestination {
#[derive(Clone)]
pub struct MetadataSettings {
/// `WAVEFLOW_METADATA_PROVIDERS`: an ordered, comma-separated list, empty
/// by default. The order is the order fields are merged in when read.
/// by default, of `lastfm` and `deezer`. The order is the order fields are
/// merged in when read.
pub providers: Vec<crate::enrichment::Provider>,
/// `WAVEFLOW_METADATA_IMAGES`, on by default, `off` to keep the lookups and
/// refuse decision 8's image fetch. A narrowing only: it switches nothing
Expand Down Expand Up @@ -1246,6 +1247,15 @@ mod tests {
assert!(parse_metadata_providers("lastfm", false).is_err());
assert!(parse_metadata_providers("lastfm,lastfm", true).is_err());
assert!(parse_metadata_providers("last.fm", true).is_err());
// Deezer asks for no key, so none is required of it.
assert_eq!(
parse_metadata_providers("lastfm, deezer", true).unwrap(),
[Provider::LastFm, Provider::Deezer]
);
assert_eq!(
parse_metadata_providers("deezer", false).unwrap(),
[Provider::Deezer]
);
}

#[test]
Expand Down
197 changes: 188 additions & 9 deletions src/database.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1917,7 +1917,7 @@ impl Database {
name_canonical: &str,
) -> Result<Vec<crate::enrichment::CachedArtist>, sqlx::Error> {
let rows = sqlx::query(
"SELECT provider, outcome, bio_summary, bio_full, fetched_at, expires_at \
"SELECT provider, outcome, bio_summary, bio_full, image_hash, fetched_at, expires_at \
FROM metadata_artist WHERE name_canonical = ?",
)
.bind(name_canonical)
Expand All @@ -1938,6 +1938,7 @@ impl Database {
outcome,
bio_summary: row.try_get("bio_summary")?,
bio_full: row.try_get("bio_full")?,
image_hash: row.try_get("image_hash")?,
fetched_at: row.try_get("fetched_at")?,
expires_at: row.try_get("expires_at")?,
});
Expand Down Expand Up @@ -1970,12 +1971,51 @@ impl Database {
.collect()
}

/// Records what a provider answered: its row and its whole neighbour list,
/// Links the portrait `hash` to every artist called `name_canonical` that
/// does not show it yet, emitting one artist event per row it moved.
///
/// Read first without the writer gate, since this runs on the read path
/// and almost always finds nothing to do. The artwork row is checked in
/// the same statement as the link: the sweeper may have taken it since
/// the cache row was read.
pub(crate) async fn link_artist_portrait(
&self,
name_canonical: &str,
hash: &str,
) -> Result<(), sqlx::Error> {
let unlinked: bool = sqlx::query_scalar(
"SELECT EXISTS (SELECT 1 FROM artist \
WHERE canonical_name = ? AND artwork_hash IS NOT ?)",
)
.bind(name_canonical)
.bind(hash)
.fetch_one(&self.pool)
.await?;
if !unlinked {
return Ok(());
}
let _writer = self.writer_guard().await;
let mut tx = self.pool.begin().await?;
link_portrait_in(
&mut tx,
name_canonical,
hash,
crate::authentication::now_ms(),
)
.await?;
tx.commit().await
}

/// Records what a provider answered: its row, its whole neighbour list,
/// and its portrait — the artwork row and every artist it belongs to —
/// in one transaction.
///
/// The writer gate is taken here and nowhere earlier — the request that
/// produced `answer` is over before this is called, so no network I/O ever
/// runs under it.
/// One transaction is what keeps the sweeper from ever seeing the artwork
/// row unreferenced: it deletes under this same gate, so it runs wholly
/// before or wholly after. The writer gate is taken here and nowhere
/// earlier — the requests that produced `answer` are over, and the
/// portrait's file is already in the store, so no network or file I/O
/// ever runs under it.
pub(crate) async fn write_metadata_artist(
&self,
provider: crate::enrichment::Provider,
Expand All @@ -1986,24 +2026,42 @@ impl Database {
) -> Result<(), sqlx::Error> {
let _writer = self.writer_guard().await;
let mut tx = self.pool.begin().await?;
if let Some(image) = &answer.image {
sqlx::query(
"INSERT INTO artwork (hash, format, source, byte_size, created_at) \
VALUES (?, ?, ?, ?, ?) ON CONFLICT (hash) DO NOTHING",
)
.bind(&image.hash)
.bind(image.format)
.bind(provider.as_str())
.bind(image.byte_size)
.bind(fetched_at)
.execute(&mut *tx)
.await?;
}
sqlx::query(
"INSERT INTO metadata_artist \
(provider, name_canonical, outcome, bio_summary, bio_full, fetched_at, expires_at) \
VALUES (?, ?, ?, ?, ?, ?, ?) \
(provider, name_canonical, outcome, bio_summary, bio_full, image_hash, \
fetched_at, expires_at) \
VALUES (?, ?, ?, ?, ?, ?, ?, ?) \
ON CONFLICT (provider, name_canonical) DO UPDATE SET \
outcome = excluded.outcome, bio_summary = excluded.bio_summary, \
bio_full = excluded.bio_full, fetched_at = excluded.fetched_at, \
expires_at = excluded.expires_at",
bio_full = excluded.bio_full, image_hash = excluded.image_hash, \
fetched_at = excluded.fetched_at, expires_at = excluded.expires_at",
)
.bind(provider.as_str())
.bind(name_canonical)
.bind(answer.outcome().as_str())
.bind(answer.bio_summary.as_deref())
.bind(answer.bio_full.as_deref())
.bind(answer.image.as_ref().map(|image| image.hash.as_str()))
.bind(fetched_at)
.bind(expires_at)
.execute(&mut *tx)
.await?;
if let Some(image) = &answer.image {
link_portrait_in(&mut tx, name_canonical, &image.hash, fetched_at).await?;
}
sqlx::query(
"DELETE FROM metadata_artist_similar WHERE provider = ? AND name_canonical = ?",
)
Expand All @@ -2030,6 +2088,61 @@ impl Database {
}
}

/// Points every artist called `name_canonical` at the portrait `hash`, and
/// records one artist event for each row that actually moved — RFC-007
/// decision 6's rule for albums, which an artist now follows: a refresh that
/// fetched the same picture changes nothing and says nothing.
///
/// Artist rows are per library, and so are events, so each moved row carries
/// its own library. A row whose artwork row is gone is not linked: the cache
/// may name a hash the sweeper has since taken. Nor is a hash the cache no
/// longer names: a read that saw the old row, then repairs after a fill
/// replaced it, would put the old portrait back and announce it.
async fn link_portrait_in(
tx: &mut sqlx::Transaction<'_, sqlx::Sqlite>,
name_canonical: &str,
hash: &str,
now: i64,
) -> Result<(), sqlx::Error> {
let moved = sqlx::query(
"UPDATE artist SET artwork_hash = ?, updated_at = ? \
WHERE canonical_name = ? AND artwork_hash IS NOT ? \
AND EXISTS (SELECT 1 FROM artwork WHERE artwork.hash = ?) \
AND EXISTS (SELECT 1 FROM metadata_artist \
WHERE name_canonical = ? AND image_hash = ?) \
RETURNING id, library_id",
)
.bind(hash)
.bind(now)
.bind(name_canonical)
.bind(hash)
.bind(hash)
.bind(name_canonical)
.bind(hash)
.fetch_all(&mut **tx)
.await?;
for row in moved {
let id: String = row.try_get("id")?;
let library_id: String = row.try_get("library_id")?;
let parse =
|value: &str| Uuid::parse_str(value).map_err(|error| sqlx::Error::Decode(error.into()));
crate::catalog::record_library_event(
tx,
crate::catalog::LibraryChange {
library_id: parse(&library_id)?,
entity_type: "artist",
entity_id: parse(&id)?,
action: "upsert",
payload: serde_json::json!({}),
changed_at: now,
origin_device_id: None,
},
)
.await?;
}
Ok(())
}

#[cfg(test)]
mod tests {
use super::*;
Expand Down Expand Up @@ -2541,4 +2654,70 @@ mod tests {
);
}
}

/// A read takes the cache row without the gate, so a fill can replace and
/// link a newer portrait before the read's repair runs. That repair must
/// not put the hash it read back over the newer one, nor announce it.
#[tokio::test]
async fn a_repair_does_not_restore_a_portrait_the_cache_replaced() {
let (_temp, database) = migrated_database().await;
const ACCOUNT: &str = "6f1d1b6e-0f1a-4a3d-9c21-000000000021";
const LIBRARY: &str = "6f1d1b6e-0f1a-4a3d-9c21-000000000022";
const ARTIST: &str = "6f1d1b6e-0f1a-4a3d-9c21-000000000023";
let read = "a".repeat(64);
let replaced = "b".repeat(64);
for (statement, binds) in [
(
"INSERT INTO account (id, username, password_hash, role, created_at, updated_at) \
VALUES (?, 'owner', 'this-account-never-authenticates', 'admin', 0, 0)",
vec![ACCOUNT],
),
(
"INSERT INTO library (id, owner_user_id, name, root_path, visibility, \
created_at, updated_at) \
VALUES (?, ?, 'Library', '/music', 'private', 0, 0)",
vec![LIBRARY, ACCOUNT],
),
(
"INSERT INTO artwork (hash, format, source, byte_size, created_at) \
VALUES (?, 'jpg', 'deezer', 1, 0), (?, 'jpg', 'deezer', 1, 0)",
vec![read.as_str(), replaced.as_str()],
),
(
"INSERT INTO artist (id, library_id, name, canonical_name, artwork_hash, \
created_at, updated_at) \
VALUES (?, ?, 'Nova Kern', 'nova kern', ?, 0, 0)",
vec![ARTIST, LIBRARY, replaced.as_str()],
),
(
"INSERT INTO metadata_artist (provider, name_canonical, outcome, image_hash, \
fetched_at, expires_at) \
VALUES ('deezer', 'nova kern', 'hit', ?, 0, 0)",
vec![replaced.as_str()],
),
] {
let mut query = sqlx::query(statement);
for bind in binds {
query = query.bind(bind.to_owned());
}
query.execute(&database.pool).await.expect("seed row");
}

database
.link_artist_portrait("nova kern", &read)
.await
.expect("repair");

let linked: String = sqlx::query_scalar("SELECT artwork_hash FROM artist WHERE id = ?")
.bind(ARTIST)
.fetch_one(&database.pool)
.await
.expect("artist row");
assert!(linked == replaced, "the newer portrait was replaced");
let events: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM library_event")
.fetch_one(&database.pool)
.await
.expect("event count");
assert_eq!(events, 0);
}
}
Loading
Loading