diff --git a/README.md b/README.md
index 41fa7c7..742163a 100644
--- a/README.md
+++ b/README.md
@@ -3,7 +3,7 @@
**An SEO content engine that uses RSS as raw material.**
Aggregate It imports RSS/Atom feeds, faithfully rewrites the content with AI (no
-fact changes — news stays news), deduplicates stories semantically, and maintains
+fact changes, news stays news), deduplicates stories semantically, and maintains
*living* topic pages that stay up to date as new sources arrive. It automatically
extracts entities (companies, people, products…), resolves them against your
custom post types, creates researched hub pages, and builds an internal-link graph
@@ -15,30 +15,30 @@ fuel happens to be RSS.
- **License:** GPL-2.0-or-later
- **Distribution:** GitHub releases + Plugin Update Checker self-update (matches `crm-connect`); WordPress.org listing optional later. Freemium add-ons via provider seams.
- **Slug:** `aggregate-it`
-- **Conventions:** follows the `crm-connect` house style — see [`docs/CONVENTIONS.md`](docs/CONVENTIONS.md)
-- **Status:** planning / pre-implementation
+- **Conventions:** follows the `crm-connect` house style, see [`docs/CONVENTIONS.md`](docs/CONVENTIONS.md)
+- **Status:** implemented (phases 0-4 built, see [`docs/PLAN.md`](docs/PLAN.md))
## What makes it different
| Capability | Why it matters for SEO |
|---|---|
-| **Living canonical posts** — one authoritative URL per story, updated over time | Concentrates link equity + sends `dateModified` freshness signals instead of splitting equity across duplicates |
-| **Faithful rewrite** — reword + strip promo junk, never alter facts | Original prose without the hallucination/legal risk; a documented selling point |
-| **Novelty gate** — only update when genuinely new facts arrive | Avoids the thin-rewrite pattern that triggers Google's Scaled Content Abuse policy |
-| **Entity hub engine** — auto-match/create CPT entities + contextual backlinks | Builds hub-and-spoke topical authority + machine-readable `about`/`sameAs` schema |
-| **Native-rich schema** — Article/NewsArticle + entity + citation graph | The durable moat thin-rewrite competitors can't replicate |
+| **Living canonical posts**: one authoritative URL per story, updated over time | Concentrates link equity + sends `dateModified` freshness signals instead of splitting equity across duplicates |
+| **Faithful rewrite**: reword + strip promo junk, never alter facts | Original prose without the hallucination/legal risk; a documented selling point |
+| **Novelty gate**: only update when genuinely new facts arrive | Avoids the thin-rewrite pattern that triggers Google's Scaled Content Abuse policy |
+| **Entity hub engine**: auto-match/create CPT entities + contextual backlinks | Builds hub-and-spoke topical authority + machine-readable `about`/`sameAs` schema |
+| **Native-rich schema**: Article/NewsArticle + entity + citation graph | The durable moat thin-rewrite competitors can't replicate |
## Documentation
-- [`docs/DECISIONS.md`](docs/DECISIONS.md) — the resolved design decisions and their rationale
-- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — data model, pipeline, subsystems, config schema, extension API
-- [`docs/CONVENTIONS.md`](docs/CONVENTIONS.md) — engineering house style, ported from `crm-connect`
-- [`docs/EXTENDING.md`](docs/EXTENDING.md) — provider/hook seams for premium add-ons
-- [`docs/PLAN.md`](docs/PLAN.md) — phased implementation plan (phases 0–4 built)
+- [`docs/DECISIONS.md`](docs/DECISIONS.md): the resolved design decisions and their rationale
+- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md): data model, pipeline, subsystems, config schema, extension API
+- [`docs/CONVENTIONS.md`](docs/CONVENTIONS.md): engineering house style, ported from `crm-connect`
+- [`docs/EXTENDING.md`](docs/EXTENDING.md): provider/hook seams for premium add-ons
+- [`docs/PLAN.md`](docs/PLAN.md): phased implementation plan (phases 0-4 built)
## Operating profile (chosen constraints)
-- **Scale:** single site, moderate volume — Action Scheduler, no external worker
+- **Scale:** single site, moderate volume. A claim-based database queue driven by WP-Cron, no external worker
- **Editorial:** fully automatic publish; manual edits after the fact
- **AI:** cheapest-viable models behind a swappable, bring-your-own-key adapter
- **Content stance:** full faithful rewrite, facts preserved, promo stripped
diff --git a/aggregate-it.php b/aggregate-it.php
index 85d913f..b74c64d 100644
--- a/aggregate-it.php
+++ b/aggregate-it.php
@@ -2,12 +2,12 @@
/**
* Plugin Name: Aggregate It
* Plugin URI: https://github.com/nmbrthirteen/aggregate-it
- * Description: An SEO content engine that uses RSS as raw material — it imports feeds, faithfully rewrites them with AI, deduplicates stories into living topic pages, and auto-builds entity hubs with an internal-link graph.
+ * Description: An SEO content engine that uses RSS as raw material. It imports feeds, faithfully rewrites them with AI, deduplicates stories into living topic pages, and auto-builds entity hubs with an internal-link graph.
*
* Features:
* - Import RSS/Atom and JSON feeds on a per-feed schedule, with categories, tags, include/exclude keywords, and per-feed length and publish overrides.
* - Bulk-add feeds, OPML import (paste or upload), and a global keyword/domain blacklist.
- * - Faithful AI rewrite that keeps the facts and strips promo — choose Google Gemini, OpenAI, or Claude with your own key.
+ * - Faithful AI rewrite that keeps the facts and strips promo, choose Google Gemini, OpenAI, or Claude with your own key.
* - Model, article-length, and writing-style controls, a daily spend cap, and a one-click connection test.
* - Semantic deduplication into one canonical post per story, a novelty gate, and dated "In the news" timelines.
* - Automatic entity hub pages with schema.org types, sameAs links, optional Wikipedia enrichment, an internal-link graph, and related articles.
diff --git a/assets/css/admin.css b/assets/css/admin.css
index 3ee6b07..020c091 100644
--- a/assets/css/admin.css
+++ b/assets/css/admin.css
@@ -438,8 +438,6 @@
display: block;
color: #646970;
font-size: 11px;
- text-transform: uppercase;
- letter-spacing: .02em;
}
.aggregate-it .ai-hub-feed-link {
diff --git a/assets/css/hub.css b/assets/css/hub.css
index 4cff221..f51ac43 100644
--- a/assets/css/hub.css
+++ b/assets/css/hub.css
@@ -1,4 +1,4 @@
-/* Aggregate It — entity hub front-end styles. Neutral + theme-agnostic. */
+/* Aggregate It entity hub front-end styles. Neutral + theme-agnostic. */
.aggregate-it-details {
border-collapse: collapse;
width: 100%;
diff --git a/assets/js/admin.js b/assets/js/admin.js
index 4de7440..92957ce 100644
--- a/assets/js/admin.js
+++ b/assets/js/admin.js
@@ -143,7 +143,7 @@
}
tbody.innerHTML = '';
if ( ! rows.length ) {
- tbody.innerHTML = '
No articles yet — add a feed to get started.
';
+ tbody.innerHTML = '
No articles yet. Add a feed to get started.
';
return;
}
rows.forEach( function ( r ) {
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index b9be356..8f9230d 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -3,7 +3,7 @@
How Aggregate It is put together: the pipeline, the data model, the subsystems, the
configuration schema, and the extension API that the freemium seams depend on.
-Read [`DECISIONS.md`](DECISIONS.md) first — this document assumes those choices.
+Read [`DECISIONS.md`](DECISIONS.md) first; this document assumes those choices.
---
@@ -32,7 +32,7 @@ Read [`DECISIONS.md`](DECISIONS.md) first — this document assumes those choice
Every stage is a job on the custom claim-based queue (the crm-connect pattern).
```
-The plugin is the runtime *and* the CMS — no external worker. A **custom claim-based DB
+The plugin is the runtime *and* the CMS: no external worker. A **custom claim-based DB
queue** (see [`CONVENTIONS.md`](CONVENTIONS.md) §5) drives every stage: atomic
`claim_token` claiming, stale-claim recovery, exponential backoff, dead-letter,
auto-pause, and a non-blocking "nudge" for low latency. The only outbound calls are: AI
@@ -42,7 +42,7 @@ provider, optional research/keyword providers, source-site fetches, and IndexNow
## 2. The pipeline (item state machine)
-Each row in `ai_items` advances through explicit, **idempotent, resumable** states —
+Each row in `ai_items` advances through explicit, **idempotent, resumable** states,
**one stage per queue claim**, never the whole pipeline in a single request. A
monolithic "import + rewrite" call is the anti-pattern we are avoiding.
@@ -56,14 +56,14 @@ monolithic "import + rewrite" call is the anti-pattern we are avoiding.
| ↳ `updated` | Processing (matched cluster) | Novelty gate; if new facts → append dated section, bump `dateModified` | cheap |
| `entity_linked` | Entity | Resolve/create entities, inject first-mention links, write relationships | optional paid |
| `published` | Publish | Slug, meta via adapter, JSON-LD, IndexNow ping | free |
-| `failed` | — | Retried w/ exponential backoff; permanent failures → dead-letter + log | — |
+| `failed` | - | Retried w/ exponential backoff; permanent failures → dead-letter + log | - |
**Idempotency:** every stage keys off `content_hash`; re-running a stage is safe.
**Re-runs:** a post can be re-rewritten (new prompt/model) without re-importing.
**Pause:** when the daily spend cap trips, the queue's transient auto-pause halts paid
stages (mirrors crm-connect's auto-pause); free stages continue.
**Batch API:** the rewrite stage may submit a batch job and set `next_attempt_at` to a
-poll time — the claim loop polls until ready, then advances. No long blocking calls.
+poll time; the claim loop polls until ready, then advances. No long blocking calls.
---
@@ -172,7 +172,7 @@ CREATE TABLE {prefix}_ai_log (
At single-site moderate volume the live vector set is thousands, not millions.
Cosine over a few thousand packed float32 rows in PHP is sub-millisecond-per-compare
and runs inside a queue job, not a web request. External vector DBs
-(Pinecone/Qdrant) are explicitly avoided — they'd break "activate and it just works"
+(Pinecone/Qdrant) are explicitly avoided; they'd break "activate and it just works"
and the OSS no-paid-dependency rule. A `pgvector`/scale path can be added behind the
same `VectorStore` interface later.
@@ -184,7 +184,7 @@ same `VectorStore` interface later.
- Feed parsing via WordPress core `SimplePie` (already bundled) for RSS/Atom; JSON
Feed handled separately.
-- **Full-content extraction:** `ContentExtractor` — feed-content-if-full-enough, else
+- **Full-content extraction:** `ContentExtractor`: feed-content-if-full-enough, else
fetch + GPL-compatible readability port (no JS). `HttpFetcher` enforces `robots.txt`,
per-domain rate limit, honest UA, response caching.
- Syntactic dedup by `(source_id, guid)` unique key + `content_hash`.
@@ -192,43 +192,43 @@ same `VectorStore` interface later.
### 4.2 Processing
- `Embedder` (provider adapter) → `ai_vectors`.
-- `Clusterer` — 3-signal: cosine ≥ threshold **∧** shared primary entity **∧** within
+- `Clusterer`: 3-signal: cosine ≥ threshold **∧** shared primary entity **∧** within
`window_until`. Borderline (near threshold) → log, treat as new cluster.
-- `Rewriter` — the **single structured call**. Strict JSON schema output. Prompt is
+- `Rewriter`: the **single structured call**. Strict JSON schema output. Prompt is
template-driven (per-feed overridable), enforces faithful-rewrite + strip-promo +
keyword-natural rules.
-- `FactsGuard` — extract numbers/dates/proper-nouns from input & output; any in output
+- `FactsGuard`: extract numbers/dates/proper-nouns from input & output; any in output
not in input → flag on the item (publishes, but visible in the dashboard).
-- `NoveltyGate` — diff new item's facts against `cluster.fact_set`; no new facts →
+- `NoveltyGate`: diff new item's facts against `cluster.fact_set`; no new facts →
suppress; new facts → append + merge into `fact_set`.
### 4.3 Entity engine
-- `EntityExtractor` — comes from the rewrite call's `entities[]` (no separate NER call).
-- `EntityResolver` — normalize → exact/alias → type-constrained fuzzy+embedding →
+- `EntityExtractor`: comes from the rewrite call's `entities[]` (no separate NER call).
+- `EntityResolver`: normalize → exact/alias → type-constrained fuzzy+embedding →
confidence band (link / create / skip-and-log). Batch-dedup guard via a per-run
pending-creation set.
-- `EntityResearcher` — pluggable. Default: stub from in-article context (zero deps).
+- `EntityResearcher`: pluggable. Default: stub from in-article context (zero deps).
With a research provider: cited external facts + `sameAs`.
-- `EntityLinker` — first-mention contextual link, capped per post, writes the
+- `EntityLinker`: first-mention contextual link, capped per post, writes the
relationship taxonomy.
-- `HubRenderer` — entity CPT template: description + cited facts + `sameAs` +
+- `HubRenderer`: entity CPT template: description + cited facts + `sameAs` +
auto-growing related-posts list + Organization/Person/Product schema.
-- `EntityMerge` — admin tool: merge B→A, repoint links/aliases/relationships/schema.
+- `EntityMerge`: admin tool: merge B→A, repoint links/aliases/relationships/schema.
### 4.4 Publish / SEO
-- `SlugGenerator` — from locked primary keyword.
-- `SeoAdapter` — strategy per active plugin (Yoast / Rank Math / SEOPress / native
+- `SlugGenerator`: from locked primary keyword.
+- `SeoAdapter`: strategy per active plugin (Yoast / Rank Math / SEOPress / native
fallback): writes title + meta + focus keyword; **suppresses the plugin's schema** on
our objects.
-- `SchemaGraph` — builds the native JSON-LD `@graph`.
-- `IndexNow` — pings on publish and on novelty update; bumps `dateModified`.
+- `SchemaGraph`: builds the native JSON-LD `@graph`.
+- `IndexNow`: pings on publish and on novelty update; bumps `dateModified`.
### 4.5 Cost & observability
-- `CostMeter` — per call/stage tokens + USD → `ai_log`; aggregates per feed/day.
-- `SpendCap` — hard daily ceiling; trips → paid stages pause, admin notice.
+- `CostMeter`: per call/stage tokens + USD → `ai_log`; aggregates per feed/day.
+- `SpendCap`: hard daily ceiling; trips → paid stages pause, admin notice.
- Dashboard: pipeline funnel, per-feed costs, facts-guard flags, ambiguous-entity
queue, original-vs-rewrite diff, feed-health.
@@ -280,7 +280,7 @@ Adding support for a new post type = adding a rule. No code.
## 6. Extension API (the freemium seams)
Free core must expose clean hooks so premium add-ons (and the community) extend without
-forking. Retrofitting these later is painful — they go in from phase 1.
+forking. Retrofitting these later is painful; they go in from phase 1.
**Provider interfaces** (premium add-ons register implementations):
@@ -304,11 +304,11 @@ interface Seo_Adapter { // one per SEO plugin
**Key filters/actions** (illustrative):
- `aggregate_it_register_ai_provider`, `…_research_provider`, `…_keyword_provider`
-- `aggregate_it_rewrite_prompt` — mutate the structured-call prompt/schema
-- `aggregate_it_cluster_match` — override/augment the 3-signal decision
-- `aggregate_it_entity_resolve` — inject custom resolution
-- `aggregate_it_before_publish` / `…_after_update` — hook the lifecycle
-- `aggregate_it_schema_graph` — mutate the JSON-LD before output
+- `aggregate_it_rewrite_prompt`: mutate the structured-call prompt/schema
+- `aggregate_it_cluster_match`: override/augment the 3-signal decision
+- `aggregate_it_entity_resolve`: inject custom resolution
+- `aggregate_it_before_publish` / `…_after_update`: hook the lifecycle
+- `aggregate_it_schema_graph`: mutate the JSON-LD before output
Premium add-ons (keyword-data provider, research provider, headless extraction,
multilingual, strategic-mode packs) are pure consumers of these seams.
@@ -317,7 +317,7 @@ multilingual, strategic-mode packs) are pure consumers of these seams.
## 7. Failure & safety posture
-- **Scaled Content Abuse:** mitigated structurally — living posts + novelty gate +
+- **Scaled Content Abuse:** mitigated structurally: living posts + novelty gate +
entity value + citations = genuine added value, not thin rewrites.
- **Hallucination:** faithful-rewrite prompt + deterministic facts-guard; strict cited
entity stubs; entities degrade to in-article-only without a research provider.
diff --git a/docs/CONVENTIONS.md b/docs/CONVENTIONS.md
index 709f1a5..43b123c 100644
--- a/docs/CONVENTIONS.md
+++ b/docs/CONVENTIONS.md
@@ -41,7 +41,7 @@ Mirror `crm-connect.php` exactly:
## 2. Plugin container (`src/Plugin.php`)
- `final class Plugin` **singleton** (`instance()`), private constructor.
-- **Manual dependency injection** — no DI container. Services are constructed in the
+- **Manual dependency injection**: no DI container. Services are constructed in the
constructor; shared collaborators are passed in explicitly.
- `boot()` calls `->register()` on each service, loads textdomain, runs
`Schema::maybe_upgrade()`, and fires `do_action( 'aggregate_it_booted', $this )`.
@@ -72,7 +72,7 @@ final class Importer {
## 4. Database (`src/Database/Schema.php`)
- One `Schema` class. **Status/state values are class constants** (`STATE_FETCHED`,
- `STATE_EXTRACTED`, …) — never bare strings in queries.
+ `STATE_EXTRACTED`, …), never bare strings in queries.
- `table( string $name )` → `$wpdb->prefix . 'aggregate_it_' . $name`.
- `install()` builds tables with **`dbDelta`** + `$wpdb->get_charset_collate()`, then
`update_option( 'aggregate_it_db_version', AGGREGATE_IT_VERSION )`.
@@ -80,7 +80,7 @@ final class Importer {
- `uninstall()` drops tables + deletes options (called from `uninstall.php`).
- All timestamps stored UTC via `gmdate( 'Y-m-d H:i:s' )`. Always `$wpdb->prepare`.
-## 5. The queue — follow crm-connect's custom claim-based queue (NOT Action Scheduler)
+## 5. The queue: follow crm-connect's custom claim-based queue (NOT Action Scheduler)
This **supersedes the earlier Action Scheduler decision.** crm-connect's queue is a
proven, dependency-free pattern that solves the WP-Cron reliability complaints, and
@@ -97,10 +97,10 @@ matching it keeps one mental model across both plugins. Reuse it verbatim in sha
- **The "nudge":** a recurring **1-minute** custom cron interval is the floor, but a
non-blocking `wp_remote_post` to `admin-ajax.php` (guarded by a `run_token` +
`hash_equals`) kicks the worker *immediately* after enqueue for low latency. Reuse
- this — it's why the cron-backed queue feels real-time.
+ this; it's why the cron-backed queue feels real-time.
**Adaptations for the heavier AI pipeline** (the one place AI work stresses the
-pattern — LLM calls take seconds, not the sub-second HTTP of a CRM upsert):
+pattern, LLM calls take seconds, not the sub-second HTTP of a CRM upsert):
- The worker advances each `ai_items` row **one stage per claim**, not the whole
pipeline in one request.
@@ -116,14 +116,14 @@ pattern — LLM calls take seconds, not the sub-second HTTP of a CRM upsert):
- **Typed accessor methods per setting** with sane defaults
(`daily_spend_cap_usd(): float`, `target_post_type(): string`, …).
- **Secrets (AI/provider API keys) stored encrypted** via `Support\Crypto`
- (`set_api_key()` encrypts, `api_key()` decrypts) — never plaintext in the DB.
+ (`set_api_key()` encrypts, `api_key()` decrypts), never plaintext in the DB.
## 7. Support utilities (`src/Support/`)
Port these from crm-connect; they're directly reusable:
-- **`Crypto`** — `aes-256-cbc`, key derived from `AUTH_KEY` via sha256, base64 envelope,
+- **`Crypto`**: `aes-256-cbc`, key derived from `AUTH_KEY` via sha256, base64 envelope,
graceful fallback when openssl is absent. For all stored API keys.
-- **`EventLog`** — bounded ring buffer (max ~200) in an option, `error/warning/info`.
+- **`EventLog`**: bounded ring buffer (max ~200) in an option, `error/warning/info`.
Use for admin-facing notices (facts-guard flags, ambiguous entities, paused-queue).
Per-stage **cost** accounting goes in the queryable `ai_log` *table*, not here.
- **`Json`**, **`Url`**, plus new ones as needed (`Html`/readability glue, `Vector`
@@ -145,7 +145,7 @@ public function get(): Ai_Provider {
```
Same shape for `Research_Provider`, `Keyword_Provider`, and the `Seo_Adapter`
-(resolved by detecting the active SEO plugin). This **is** the freemium seam — premium
+(resolved by detecting the active SEO plugin). This **is** the freemium seam; premium
add-ons register via the filter.
## 9. Admin (`src/Admin/`)
@@ -161,7 +161,7 @@ add-ons register via the filter.
## 10. Distribution & release
-**Primary channel: GitHub releases + PUC self-update** (matches crm-connect) — *not*
+**Primary channel: GitHub releases + PUC self-update** (matches crm-connect), *not*
WordPress.org for v1.
- `bin/release.sh X.Y.Z`: validates semver, bumps the version in the header + the
@@ -171,8 +171,8 @@ WordPress.org for v1.
install, exclude dev files) and publish the GitHub release asset; PUC consumes it.
> **If a WordPress.org listing is added later** (worth it for the SEO-discoverability of
-> the plugin itself): PUC must be **conditionally disabled** in wp.org builds — the
-> directory forbids third-party update sources — and external-AI-API usage must be
+> the plugin itself): PUC must be **conditionally disabled** in wp.org builds; the
+> directory forbids third-party update sources, and external-AI-API usage must be
> disclosed in `readme.txt` with BYO-key opt-in. Keep the two build targets in mind but
> don't carry the cost now.
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index 5568700..3f5c170 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -8,50 +8,50 @@ considered, and why. These are the load-bearing choices; everything in
- **Scale / runtime:** single WordPress site, moderate volume (dozens of feeds,
hundreds of items/day). → A **custom claim-based DB queue** (the `crm-connect`
- pattern); no external worker service. *(Supersedes an earlier Action Scheduler lean —
+ pattern); no external worker service. *(Supersedes an earlier Action Scheduler lean,
see D14.)*
- **Editorial model:** fully automatic publish. The owner edits anything off after
the fact. → No human review queue, but rich observability so problems are findable.
- **AI / budget:** cheapest viable. → Small/cheap default models behind a
provider-agnostic, bring-your-own-key adapter; batch + caching everywhere.
-- **Content stance:** full **faithful** rewrite. News stays news — no fact changes.
+- **Content stance:** full **faithful** rewrite. News stays news: no fact changes.
Strip promotional/boilerplate junk from the source.
- **Overriding goal:** SEO. Every decision below is filtered through "does this rank?"
-## D1 — Content model: living canonical posts (B), built via hybrid path
+## D1: Content model: living canonical posts (B), built via hybrid path
-One authoritative post per *story/topic*, updated over time — not one post per feed
+One authoritative post per *story/topic*, updated over time, not one post per feed
item. Built incrementally: every item still becomes a post in phase 1, but each
carries a `topic_cluster_id` from day one, so the "update existing instead of
duplicate" behavior layers in later with **no migration**.
-- *Rejected:* item-per-post (classic aggregator) — splits link equity + freshness
+- *Rejected:* item-per-post (classic aggregator): splits link equity + freshness
across duplicates; the opposite of what we want for SEO.
- *Why:* one living URL concentrates equity and emits `dateModified` freshness.
-## D2 — Updates: append timeline-style + novelty gate
+## D2: Updates: append timeline-style + novelty gate
-When new info arrives for an existing story, append a dated section ("Update —
+When new info arrives for an existing story, append a dated section ("Update:
June 25: …"). Never rewrite the whole post. **And** only update when the new item
-carries genuinely new facts (novelty gate) — restated wire copy is suppressed.
+carries genuinely new facts (novelty gate); restated wire copy is suppressed.
- *Rejected:* wholesale re-synthesis (expensive, churns SEO, mangle risk on a cheap
model); structured section-merge (highest complexity + mangle risk).
- *Why:* cheapest, lowest blast radius under full-auto, preserves facts and URL
stability; novelty gate is what keeps it from becoming thin-content spam.
-## D3 — Clustering: 3-signal conservative match
+## D3: Clustering: 3-signal conservative match
Two items are "the same story" only if **all three** hold: semantic similarity
(embedding cosine over a configurable threshold) **and** shared primary entity
**and** within a rolling time window (default ~7 days, per-feed configurable).
-- *Rejected:* similarity-only — merges different stories about the same subject.
+- *Rejected:* similarity-only: merges different stories about the same subject.
- *Why:* under full-auto a **false merge corrupts a live post**; a false split just
leaves a stray duplicate. Bias hard toward not merging. Borderline cases are
logged, not silently merged.
-## D4 — URLs & post type: flat keyword slugs, configurable target
+## D4: URLs & post type: flat keyword slugs, configurable target
Canonical posts go to a **configurable target post type (default `post`)**. URLs are
**flat, keyword-rich slugs with no dates** (`/ai-generated-keyword-slug/`), generated
@@ -62,18 +62,18 @@ comes from the entity internal-link graph, not deep URL nesting.
keyword. Google News/Discover targeting deferred (compatible, but extra
requirements not taken on now).
-## D5 — SEO metadata: hybrid adapter
+## D5: SEO metadata: hybrid adapter
Detect the active SEO plugin (Yoast / Rank Math / SEOPress) and write the
AI-generated **target keyword, SEO title, meta description** into *its* fields. If
none is installed, a **minimal native fallback** outputs title + meta only.
-- *Rejected:* fully native SEO module — duplicate/competing meta tags when the user
+- *Rejected:* fully native SEO module: duplicate/competing meta tags when the user
also runs Yoast (they will).
- *Principle:* an SEO engine generates the *inputs*; let the user's chosen plugin
render the ``.
-## D6 — Structured data: native-rich JSON-LD
+## D6: Structured data: native-rich JSON-LD
Schema is the one place we go native (plugins can't build it from our data). One
clean JSON-LD `@graph`: `Article`/`NewsArticle` (per-source toggle), always
@@ -85,19 +85,19 @@ adapter **suppresses the plugin's competing Article schema** on our objects.
- *Why:* the entity + citation graph is the durable SEO moat and the "added value"
proof against Scaled Content Abuse.
-## D7 — Keyword targeting: 3 layers, auto-infer default
+## D7: Keyword targeting: 3 layers, auto-infer default
-1. **Default — AI-inferred** primary keyword per cluster (volume mode: publish all).
-2. **Optional — keyword strategy list** (global/per-feed) with a **strategic-mode
+1. **Default: AI-inferred** primary keyword per cluster (volume mode: publish all).
+2. **Optional: keyword strategy list** (global/per-feed) with a **strategic-mode
toggle** (only publish clusters that match a target keyword).
-3. **Optional/premium — pluggable keyword-data provider** (DataForSEO/Semrush/Ahrefs,
+3. **Optional/premium: pluggable keyword-data provider** (DataForSEO/Semrush/Ahrefs,
BYO key) for volume + difficulty, to prioritize/skip clusters.
-Guardrails: **keyword locked at cluster creation** (never recomputed — appends must
+Guardrails: **keyword locked at cluster creation** (never recomputed, appends must
not churn the slug); rewrite prompt treats the keyword as "include naturally, never
stuff."
-## D8 — Full-content extraction: hybrid + polite
+## D8: Full-content extraction: hybrid + polite
Use feed content when it's full enough (length heuristic); otherwise fetch the source
URL and run **server-side readability extraction (no JS)**. **Polite by default:**
@@ -109,7 +109,7 @@ bar**, else skip + log. **Never auto-publish a thin stub.**
protects the user's server IP and the plugin's reputation. Headless rendering for
JS-heavy/paywalled sites is a deferred premium enhancement.
-## D9 — Entity engine: conservative resolution + graceful degradation
+## D9: Entity engine: conservative resolution + graceful degradation
Per extracted mention, **type-constrained** matching within the target CPT:
- **Strong match** (exact/alias or high score) → link.
@@ -120,53 +120,53 @@ Per extracted mention, **type-constrained** matching within the target CPT:
Creation degrades gracefully: a stub is built from **in-article context** with **zero
external dependency**; rich research (cited external facts + `sameAs`) is a **pluggable
-BYO-key provider** (premium). Stubs are strict — only cited fields, no hallucinated
+BYO-key provider** (premium). Stubs are strict, only cited fields, no hallucinated
fills. Ship an **entity merge tool** (repoints links/aliases/schema) as the cleanup valve.
- *Why:* a clean sparse graph outranks a dense corrupted one. Duplicate entities are
recoverable (merge); wrong links corrupt published posts + schema.
-## D10 — Entity linking & hubs
+## D10: Entity linking & hubs
Contextual in-body link on the **first mention only**, natural-name anchor, **capped
links per post**. Bidirectional relationship (postmeta/taxonomy) powers schema +
hub. **Hubs are pillar pages:** AI description + cited facts + `sameAs` + an
-**auto-growing related-posts list** — every new post about an entity strengthens its
+**auto-growing related-posts list**, every new post about an entity strengthens its
hub (the topical-authority + freshness flywheel).
-## D11 — Freshness & indexing
+## D11: Freshness & indexing
Sitemaps **delegated to the SEO plugin** (ensure entity CPTs included). **IndexNow**
-built in (free) — ping on publish and on a real novelty update. Freshness loop: novelty
+built in (free), ping on publish and on a real novelty update. Freshness loop: novelty
append → bump `dateModified` → update sitemap `lastmod` → fire IndexNow. No faked
instant-index API for normal content.
-## D12 — Cost strategy
+## D12: Cost strategy
**One structured AI call per item** returns `{rewritten_body, seo_title,
-meta_description, slug, primary_keyword, entities[], facts[]}` — collapses
+meta_description, slug, primary_keyword, entities[], facts[]}`, collapses
rewrite+SEO+NER+fact-extraction into one round trip. Embeddings are a separate cheap
call. **Batch API** for non-urgent work (~50% off). **Content-hash + prompt caching.**
**Per-feed/per-stage cost tracking** with a **hard daily spend cap** that pauses the
-queue. **Deterministic facts-guard** (numbers/dates/proper-noun subset check — no
+queue. **Deterministic facts-guard** (numbers/dates/proper-noun subset check, no
extra AI call) flags invented facts.
-## D13 — Images & EEAT
+## D13: Images & EEAT
Featured image **downloaded into the media library** (never hotlinked); import is
**configurable** (off / import / import-if-licensed), default conservative due to
copyright. **AI-generated alt text** (image SEO + a11y). **Author/EEAT:** configurable
-real author with bio + `author` schema — **no fabricated human personas**. Optional
+real author with bio + `author` schema, **no fabricated human personas**. Optional
"AI-assisted" disclosure toggle.
-## D14 — Engineering conventions: match `crm-connect`
+## D14: Engineering conventions: match `crm-connect`
The author's existing `crm-connect` plugin sets the house style; Aggregate It mirrors
it for consistency. Full detail in [`CONVENTIONS.md`](CONVENTIONS.md). The two choices
that *changed* prior decisions:
- **Queue: custom claim-based DB queue, not Action Scheduler.** crm-connect runs a
- proven dependency-free queue — atomic `claim_token` claiming, stale-claim recovery,
+ proven dependency-free queue, atomic `claim_token` claiming, stale-claim recovery,
exponential backoff, dead-letter, transient auto-pause, and a non-blocking
"nudge" (admin-ajax self-POST) that makes a 1-minute cron feel real-time. This
answers the WP-Cron reliability concern *without* Action Scheduler, and keeps one
diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md
index b1b1912..8133cf2 100644
--- a/docs/EXTENDING.md
+++ b/docs/EXTENDING.md
@@ -1,25 +1,25 @@
# Extending Aggregate It
Aggregate It's core is free and provider-agnostic. Every paid/external capability is a
-seam you (or a premium add-on) plug into via a filter — no core fork required. This is
+seam you (or a premium add-on) plug into via a filter, no core fork required. This is
also the freemium boundary: the free plugin ships the engine; add-ons ship providers.
## AI provider (rewrite + embeddings)
-The one required capability. Four providers ship in core — pick one in Settings, no
+The one required capability. Four providers ship in core; pick one in Settings, no
filter needed:
-- **`MockProvider`** — deterministic, free, the default. Good for testing the pipeline.
-- **`GeminiProvider`** — cheapest live option. `generateContent` JSON mode + native
+- **`MockProvider`**: deterministic and free, for the test suite only. It echoes the source text, so publishing refuses to run with it; the default is "not configured" until a real provider and key are saved.
+- **`GeminiProvider`**: cheapest live option. `generateContent` JSON mode + native
`text-embedding-004`. Default model `gemini-2.0-flash-lite`.
-- **`OpenAiProvider`** — Chat Completions JSON mode + native `text-embedding-3-small`.
+- **`OpenAiProvider`**: Chat Completions JSON mode + native `text-embedding-3-small`.
Default model `gpt-4o-mini`.
-- **`AnthropicProvider`** — Claude Messages API (structured outputs). Default Haiku 4.5.
+- **`AnthropicProvider`**: Claude Messages API (structured outputs). Default Haiku 4.5.
Anthropic has no embeddings endpoint, so it uses Voyage AI when a Voyage key is set,
else a local lexical fallback.
All call their HTTP APIs via the WP HTTP layer (no SDK bundling). The model is a free
-Settings field — blank uses the provider's cheapest default.
+Settings field; blank uses the provider's cheapest default.
To register a *different* provider (a local model, Azure, etc.):
@@ -44,14 +44,14 @@ interface AiProvider {
}
```
-The structured schema is the one from `Ai\Rewriter::schema()` —
+The structured schema is the one from `Ai\Rewriter::schema()`,
`rewritten_body, seo_title, meta_description, slug, primary_keyword, entities[], facts[]`.
Report real `tokens`/`cost_usd` so the spend cap and dashboard work. Use the
Anthropic Batch API for the ~50% discount where latency allows.
## Research provider (entity enrichment, optional)
-Absent by default — entities are built from in-article context only. Register one to add
+Absent by default; entities are built from in-article context only. Register one to add
cited external facts + `sameAs`:
```php
@@ -72,21 +72,21 @@ Implement `AggregateIt\Keyword\KeywordProvider::metrics( array $keywords )` →
## Content / prompt / schema hooks
-- `aggregate_it_rewrite_prompt` ( $prompt, $title, $content, $target_keyword ) — mutate the rewrite instruction.
-- `aggregate_it_rewrite_schema` ( $schema ) — extend the structured-output contract.
-- `aggregate_it_extract_html` ( null, $html ) — return a string to override the built-in readability heuristic with a real library.
-- `aggregate_it_schema_graph` ( $graph, $post_id ) — mutate the JSON-LD before output.
-- `aggregate_it_entity_post_types` ( $types ) — declare extra entity CPTs for the dashboard count.
+- `aggregate_it_rewrite_prompt` ( $prompt, $title, $content, $target_keyword ): mutate the rewrite instruction.
+- `aggregate_it_rewrite_schema` ( $schema ): extend the structured-output contract.
+- `aggregate_it_extract_html` ( null, $html ): return a string to override the built-in readability heuristic with a real library.
+- `aggregate_it_schema_graph` ( $graph, $post_id ): mutate the JSON-LD before output.
+- `aggregate_it_entity_post_types` ( $types ): declare extra entity CPTs for the dashboard count.
## Lifecycle
-- `aggregate_it_publish_ping` ( $post_id ) — fired when a post is published or updated (IndexNow listens here).
-- `aggregate_it_spend_cap_reached` ( $cap ) — daily spend ceiling hit.
-- `aggregate_it_dead_letter` ( $item, $exception ) — an item failed permanently.
-- `aggregate_it_booted` ( $plugin ) — container ready.
+- `aggregate_it_publish_ping` ( $post_id ): fired when a post is published or updated (IndexNow listens here).
+- `aggregate_it_spend_cap_reached` ( $cap ): daily spend ceiling hit.
+- `aggregate_it_dead_letter` ( $item, $exception ): an item failed permanently.
+- `aggregate_it_booted` ( $plugin ): container ready.
## Notes for premium add-ons
- Ship as a separate plugin that registers the filters above on `plugins_loaded`/`init`.
-- The free core never hard-depends on any add-on — every seam degrades gracefully.
+- The free core never hard-depends on any add-on: every seam degrades gracefully.
- Keep GPL-2.0-or-later for WordPress.org compatibility.
diff --git a/docs/PLAN-custom-sources-activity-hubs.md b/docs/PLAN-custom-sources-activity-hubs.md
index cd18bdb..02f9a8b 100644
--- a/docs/PLAN-custom-sources-activity-hubs.md
+++ b/docs/PLAN-custom-sources-activity-hubs.md
@@ -2,34 +2,34 @@
Three independent but mutually reinforcing features. Build order below is by
dependency, not by the order they were requested. Everything here is
-config-driven — no site-specific code, no hardcoded selectors or post types.
+config-driven, no site-specific code, no hardcoded selectors or post types.
> **STATUS: complete.** Phase 0 (v0.1.37), Phase 1 + scraper follow-ups (v0.1.38,
-> v0.1.39), Phase 2 (v0.1.40), and final polish — live extraction preview + stub-hub
-> review queue — all shipped. PRs #1–#5 merged.
+> v0.1.39), Phase 2 (v0.1.40), and final polish, live extraction preview + stub-hub
+> review queue, all shipped. PRs #1-#5 merged.
Guiding constraints (from project memory + this request):
- Generalize everything; nothing hardcoded for any specific site.
-- Observability over silent fallbacks — every automated decision is recorded.
+- Observability over silent fallbacks: every automated decision is recorded.
- Clean/minimal UI, no decorative code comments.
---
-## Verified ground-truth (2nd research pass) — corrections that change the build
+## Verified ground-truth (2nd research pass): corrections that change the build
These were checked against the actual source; several invalidate the obvious-but-wrong
design:
1. **Live pipeline order is `fetched → extracted → embedded → clustered →
entity_linked → published`** (`Pipeline::default_order()`). `STATE_REWRITTEN`
- exists in `Schema` but is **not in the active order** — the AI rewrite happens
+ exists in `Schema` but is **not in the active order**; the AI rewrite happens
*inside* the `clustered` stage (`ComposeStage`), which is the real publish step.
2. **`PassthroughStage` is a no-op Phase-0 placeholder** (advances state, does
- nothing). It is NOT a "publish structured data verbatim" mechanism — do not
+ nothing). It is NOT a "publish structured data verbatim" mechanism; do not
reuse it for passthrough scraping.
3. **`PostFactory::create()` is hard-coupled to AI `$structured` output**
(`seo_title`, `slug`, `rewritten_body`, `facts`) and to the **global**
- `Settings::target_post_type()`. There is **no per-source post type** today — it
+ `Settings::target_post_type()`. There is **no per-source post type** today; it
is a genuine gap a scraper must fill (add `PostFactory::create_mapped()`).
4. **Robots.txt + throttle will self-sabotage a naive scraper.** `HttpFetcher::fetch()`
*throws* on a robots.txt disallow, and `throttle()` *throws* `RateLimited` if the
@@ -40,11 +40,11 @@ design:
deferred to a per-item pipeline stage** where `RateLimited → defer` already works.
This mirrors how RSS already splits lightweight import from heavy `ExtractStage`.
There is an SSRF override filter (`aggregate_it_allow_private_hosts`) but **no
- robots override** — a per-source robots toggle (with a visible ToS warning) must
+ robots override**, a per-source robots toggle (with a visible ToS warning) must
be added.
5. **No CSS-selector or HTML5 library is present**; runtime deps are zero (only
`php >=8.0`), and the plugin autoloads from `src/` when `vendor/` is absent
- (`aggregate-it.php:33-51`). Recommended: stay dependency-free — use
+ (`aggregate-it.php:33-51`). Recommended: stay dependency-free, use
`DOMDocument`/`DOMXPath` (already used in `ContentExtractor`) and accept **XPath**
plus a small CSS→XPath converter for the common subset. Adding
`symfony/dom-crawler` is the fallback only if the release workflow is confirmed to
@@ -54,12 +54,12 @@ Reusable as-is: **`EntityRegistrar::register_post_types()` is the exact template
for registering a scraper's target CPT from config; **`HttpFetcher`** gives SSRF
guard, robots, per-host throttle, UA, and caching; the **`Entry` format**
(`{guid,url,title,content,image,date}`) and `Importer::ingest()`'s guid/hash dedup
-work unchanged — but custom scraped fields (location, venue, end_date) need a new
+work unchanged, but custom scraped fields (location, venue, end_date) need a new
`fields` payload threaded through `ItemStore::enqueue()` into `Item->flags`.
---
-## Phase 0 — Activity Log (foundation) — IMPLEMENTED
+## Phase 0: Activity Log (foundation), IMPLEMENTED
Rationale: a durable "what changed from what" record is the substrate that makes
both the scraper and Topic Hubs legible. Build it first; the other phases emit
@@ -76,13 +76,13 @@ deletes only zero-cost rows so spend history survives. New `ActivityLogTest`; fu
suite 64 green.
Migration: gated on a new `Schema::DB_VERSION` (not the plugin version), so the
-dbDelta column-add runs on the next boot of any existing install — independent of a
+dbDelta column-add runs on the next boot of any existing install, independent of a
release. Hardened after an adversarial review pass: activity writes that fail
`error_log` once instead of vanishing silently; `wp_json_encode` failures store NULL;
the 5s live refresh pauses while a row is expanded or text is selected; the Live
toggle disables on paged views; JS-built labels are i18n'd. Note (volume): the old
`EventLog` capped at 200 rows; the activity log now writes ~1 info row per extracted
-item, pruned by `Retention` at `retention_days` (default 90) — deliberate trade for
+item, pruned by `Retention` at `retention_days` (default 90), deliberate trade for
durability/observability.
### Problem with today's logging
@@ -110,19 +110,19 @@ New table `wp_aggregate_it_activity` (migration in `src/Database/Schema.php`):
Index on (created_at), (item_id), (source_id), (type).
### Code
-- `src/Support/ActivityLog.php` — `record(type, summary, detail, ctx)`,
+- `src/Support/ActivityLog.php`: `record(type, summary, detail, ctx)`,
`query(filters, paging)`, `prune(keep_days)`. Keep `EventLog` as a thin
back-compat shim that forwards to ActivityLog, or fold its call sites over.
- Instrument transitions at the single choke point `ItemStore::advance()` plus
each stage where a transformation has a meaningful before/after:
- - `ExtractStage` — feed length → extracted length, image source chosen/missed.
- - `ClusterStage`/`Clusterer` — new story vs merged into #N, similarity score,
+ - `ExtractStage`: feed length → extracted length, image source chosen/missed.
+ - `ClusterStage`/`Clusterer`: new story vs merged into #N, similarity score,
shared fact (the "close call" data already exists).
- - `ComposeStage` — original feed title → SEO title; raw → rewritten length;
+ - `ComposeStage`: original feed title → SEO title; raw → rewritten length;
**suppression + reason** (thin / keyword / strategic) which is currently
silent.
- - `EntityStage` — entities detected → linked / created / skipped + score.
- - `ImageImporter` — which candidate won.
+ - `EntityStage`: entities detected → linked / created / skipped + score.
+ - `ImageImporter`: which candidate won.
- Retention: hook into `src/Maintenance/Retention.php` to prune by age/row cap.
### UI
@@ -135,7 +135,7 @@ Index on (created_at), (item_id), (source_id), (type).
---
-## Phase 1 — Custom Website Sources (generalized scraper)
+## Phase 1: Custom Website Sources (generalized scraper)
**Feature-complete on `feature/custom-website-sources`** (PR #2):
- Engine: `CssToXpath`, `FieldExtractor`, `ScraperParser` (one-fetch list + sitemap
@@ -159,7 +159,7 @@ per-source robots-override toggle; a live extraction preview in the form.
The centerpiece. Lets a user point at any HTML site (e.g. igamingcalendar.com),
extract repeating items, and map fields onto a chosen post type. igaming is
-server-rendered HTML with a sitemap but **no RSS / JSON / JSON-LD** — so an HTML
+server-rendered HTML with a sitemap but **no RSS / JSON / JSON-LD**, so an HTML
scraper is required, and feeds/structured-data are an accuracy bonus when present.
### 1a. Source type + parser factory
@@ -171,30 +171,30 @@ scraper is required, and feeds/structured-data are an accuracy bonus when presen
- Add `src/Source/Parser/ScraperParser.php`.
- `Importer::parse()` becomes a dispatch on `source_type` (the existing seam).
-### 1b. Discovery — how items are found (config)
+### 1b. Discovery: how items are found (config)
Stored in `Source::settings['scrape']['discovery']`:
- `mode: "list" | "sitemap"`
-- list mode: `list_urls[]`, `item_selector` (the repeating row), `detail` (bool —
+- list mode: `list_urls[]`, `item_selector` (the repeating row), `detail` (bool,
follow each row's link to a detail page for richer fields).
-- sitemap mode: `sitemap_url`, `url_filter` (regex) — each matching URL is an item.
+- sitemap mode: `sitemap_url`, `url_filter` (regex): each matching URL is an item.
`src/Source/Scrape/Discovery.php` resolves a discovery config into a list of
item handles (inline row DOM and/or detail URL). **Runs at import time with a single
-`HttpFetcher::fetch()`** (the listing or sitemap page) — per finding #4, detail-page
+`HttpFetcher::fetch()`** (the listing or sitemap page), per finding #4, detail-page
fetches must NOT happen here. Each handle becomes an `Entry` whose `url` is the
detail link and whose inline row fields ride in a new `fields` payload. The detail
fetch + richer extraction is deferred to `ExtractStage` (per item, throttle-safe).
Scraping a path blocked by robots.txt requires the per-source robots-override toggle
-(finding #4) — surfaced in the UI with a ToS warning, implemented as an
+(finding #4), surfaced in the UI with a ToS warning, implemented as an
`aggregate_it_respect_robots` filter check keyed by source.
-### 1c. Extraction — accuracy cascade
+### 1c. Extraction: accuracy cascade
`src/Source/Scrape/FieldExtractor.php`. For each item, resolve every configured
field by trying, in order of accuracy:
-1. **Structured data** — schema.org JSON-LD / microdata on the page (most
+1. **Structured data**: schema.org JSON-LD / microdata on the page (most
accurate when present).
-2. **Feed/sitemap metadata** — when the discovery source already carries it.
-3. **CSS/XPath selectors** — `{ selector, attr (text/html/href/src/datetime/any
+2. **Feed/sitemap metadata**: when the discovery source already carries it.
+3. **CSS/XPath selectors**: `{ selector, attr (text/html/href/src/datetime/any
attr), regex?, transform? (date-parse, trim) }`. Add a small CSS→XPath
converter (or accept XPath directly) over `DOMDocument`/`DOMXPath`.
@@ -202,7 +202,7 @@ Field set is open-ended: `title, url, guid, date, image, content` + arbitrary
custom fields (`location, venue, end_date, category`, …). Nothing event-specific
is hardcoded.
-### 1d. Selector configuration — AI-assisted + validated (most accurate practical path)
+### 1d. Selector configuration: AI-assisted + validated (most accurate practical path)
`src/Source/Scrape/SelectorAssistant.php`:
- User pastes a listing URL (and optionally a detail URL).
- Fetch sample page(s) via `HttpFetcher` (already has SSRF guard, robots,
@@ -213,7 +213,7 @@ is hardcoded.
and show the extracted preview; user confirms/edits. Manual selector entry is
always available as the floor.
- On each scheduled scrape, re-validate; selector drift (rows suddenly empty)
- emits a `warning` activity row — the activity log becomes the early-warning
+ emits a `warning` activity row; the activity log becomes the early-warning
system for site changes.
### 1e. Mapping to a post type ("post type connection")
@@ -232,18 +232,18 @@ Generic CPT/taxonomy registration from config (reuse the existing
`DelegationRules` / `EntityRegistrar` pattern) so a user can define an "Events"
post type without code.
-### 1f. Processing mode — passthrough by default
+### 1f. Processing mode: passthrough by default
- Per-source `processing: "passthrough" | "rewrite"`, set at enqueue as
`Item->flags['passthrough']`. Scraped structured data defaults to **passthrough**:
fields map in verbatim, no AI rewrite, so dates/venues can't be hallucinated.
News-style sources opt into `rewrite`.
-- Mechanism (corrected — the pipeline is one-stage-per-state and stages route via
+- Mechanism (corrected, the pipeline is one-stage-per-state and stages route via
their return value): add a cheap `if ($item->flags['passthrough'])` guard to each
AI/cost stage so passthrough items skip the expensive work but still traverse the
state machine:
- `ExtractStage` (fetched): if passthrough, optionally fetch the detail page
(throttle-safe; `RateLimited → defer`) and run `FieldExtractor` into
- `flags['fields']`; advance — no readability/AI.
+ `flags['fields']`; advance; no readability/AI.
- `EmbedStage` / `ClusterStage`: if passthrough, skip embedding/clustering; advance.
- `ComposeStage` (clustered): if passthrough, call **`PostFactory::create_mapped()`**
(no `Rewriter`, no `CostMeter`) using the per-source post type + field/meta/
@@ -268,12 +268,12 @@ post type without code.
---
-## Phase 2 — Reimagine "Linked Pages" → legible Topic Hubs — SHIPPED (engine reused)
+## Phase 2: Reimagine "Linked Pages" → legible Topic Hubs, SHIPPED (engine reused)
Landed on `feature/topic-hubs`: renamed "Linked Pages" → **Topic Hubs** (menu, dashboard
card, page heading) with a plain-language explainer; `EntityStage` now records every
-decision into the Activity log — linked + **created** + the previously-silent **skipped**
-(with reason: ambiguous match / no hub for type) — as structured detail; the post editor
+decision into the Activity log, linked + **created** + the previously-silent **skipped**
+(with reason: ambiguous match / no hub for type), as structured detail; the post editor
metabox lists the topic hubs linked from each article. Thresholds were already per-rule
config (link 92 / ambiguous 75). Follow-up left: a stub-page review queue.
@@ -303,9 +303,9 @@ Same entity-hub engine (`EntityStage`, `DelegationRules`, `EntityRepository`,
---
## Suggested sequencing
-1. Phase 0 (Activity log) — small, foundational, immediately useful.
-2. Phase 1 (Custom sources) — the headline feature; emits into the log.
-3. Phase 2 (Topic Hubs legibility) — mostly UI + surfacing existing data into the log.
+1. Phase 0 (Activity log): small, foundational, immediately useful.
+2. Phase 1 (Custom sources): the headline feature; emits into the log.
+3. Phase 2 (Topic Hubs legibility): mostly UI + surfacing existing data into the log.
Each phase ships independently and is backward compatible (new `source_type`
defaults to `rss`; activity table is additive; Topic Hubs reuses the existing
diff --git a/docs/PLAN.md b/docs/PLAN.md
index 90b55d4..e217e52 100644
--- a/docs/PLAN.md
+++ b/docs/PLAN.md
@@ -5,9 +5,9 @@ a working aggregator first, then the AI+SEO core that makes it rank, then the en
moat, then polish + premium seams.
- **Stack:** PHP 8.0+, WordPress 6.4+, custom claim-based queue (the `crm-connect`
- pattern — see [`CONVENTIONS.md`](CONVENTIONS.md)), Composer for dev deps.
+ pattern, see [`CONVENTIONS.md`](CONVENTIONS.md)), Composer for dev deps.
- **Standard:** WordPress Coding Standards (PHPCS), GPL-2.0+ only.
-- **Conventions:** match `crm-connect` — namespace `AggregateIt\`, singleton `Plugin`
+- **Conventions:** match `crm-connect`: namespace `AggregateIt\`, singleton `Plugin`
with `register()` services, `Schema`/`dbDelta`, encrypted `Settings`, provider
factory-with-filter, GitHub + PUC distribution. Details in `CONVENTIONS.md`.
- **Testing:** PHPUnit + WP test suite; integration tests against canned feed fixtures
@@ -15,13 +15,13 @@ moat, then polish + premium seams.
- **Definition of done per phase:** feature works end-to-end on a real feed, has tests,
and the relevant hooks/filters from the extension API are in place.
-> The extension-API seams (`ARCHITECTURE.md` §6) are NOT a final phase — each phase
+> The extension-API seams (`ARCHITECTURE.md` §6) are NOT a final phase; each phase
> wires its own hooks as it lands. Retrofitting them is the one thing that forces a
> rewrite later.
---
-## Phase 0 — Scaffold (foundation)
+## Phase 0: Scaffold (foundation)
- Plugin bootstrap (mirror `crm-connect.php`): header w/ GitHub `Plugin URI`, constants,
composer autoload + spl fallback, **PUC GitHub updater**, activation→`Schema::install`,
@@ -47,7 +47,7 @@ moat, then polish + premium seams.
---
-## Phase 1 — Ingestion engine (a working aggregator) — ✅ built
+## Phase 1: Ingestion engine (a working aggregator), ✅ built
**Goal:** add feeds, import items, deduplicate, land them as posts. No AI yet.
@@ -58,7 +58,7 @@ moat, then polish + premium seams.
> `aggregate_it_extract_html`), `Pipeline\ExtractStage` (replaces the `fetched`
> passthrough; flags thin items), GUID + content-hash dedup in `ItemStore`, and the
> Sources admin screen. Items currently flow through to `published` via passthroughs
-> (no rewrite yet — that's Phase 2).
+> (no rewrite yet; that's Phase 2).
- **Sources admin:** CRUD for feeds, per-feed settings, schedule, status, health view.
- **Fetch + parse:** SimplePie (RSS/Atom) + JSON Feed; map items → `ai_items`.
@@ -71,7 +71,7 @@ moat, then polish + premium seams.
plus the nudge (NOT raw WP-Cron for the work itself).
- Items publish as plain posts (rewrite is phase 2) so the pipeline is observable.
-*Ships:* a polite, reliable RSS aggregator with full-content extraction — already on
+*Ships:* a polite, reliable RSS aggregator with full-content extraction, already on
par with the baseline tools, minus AI.
**Exit criteria:** a real feed imports on schedule; truncated feeds get full content;
@@ -79,13 +79,13 @@ duplicates are suppressed; rude-scraping safeguards verified.
---
-## Phase 2 — AI + SEO core (the ranking engine) — ✅ built
+## Phase 2: AI + SEO core (the ranking engine), ✅ built
**Goal:** faithful rewrite, semantic dedup, living canonical posts, full SEO output.
> Status: implemented. `Vector\VectorStore` (brute-force cosine), `Pipeline\EmbedStage`
> (paid), `Cluster\Clusterer` + `ClusterStage` (3-signal: similarity ∧ time window ∧
-> shared salient fact), `Ai\Rewriter` (single structured call), `Ai\FactsGuard`
+> two shared salient facts), `Ai\Rewriter` (single structured call), `Ai\FactsGuard`
> (invented-fact + novelty detection), `Keyword\KeywordStrategy` (auto-infer + list +
> strategic skip), `Publish\PostFactory` (living posts + dated append), `Seo\Seo`
> (Yoast/RankMath/SEOPress meta adapter + schema suppression + native fallback),
@@ -94,27 +94,27 @@ duplicates are suppressed; rude-scraping safeguards verified.
> cap, and the original-vs-rewrite editor meta box. Verified with the MockProvider.
- **`Embedder`** → `ai_vectors`; `VectorStore` with brute-force cosine.
-- **`Clusterer`** — 3-signal match (similarity ∧ shared entity ∧ time window);
+- **`Clusterer`**: 3-signal match (similarity ∧ shared entity ∧ time window);
borderline → log + new cluster. Assign `topic_cluster_id`.
-- **`Rewriter`** — the **single structured AI call** → `{body, seo_title,
+- **`Rewriter`**: the **single structured AI call** → `{body, seo_title,
meta_description, slug, primary_keyword, entities[], facts[]}`. Template-driven
prompt (per-feed overridable): faithful rewrite, strip promo, keyword-natural.
-- **`FactsGuard`** — deterministic invented-fact detector → flags item.
+- **`FactsGuard`**: deterministic invented-fact detector → flags item.
- **Living posts (B):** new cluster → create canonical post; matched cluster →
**`NoveltyGate`** → append dated section + bump `dateModified` + merge `fact_set`.
- **Keyword layer 1** (auto-infer) + **layer 2** scaffold (keyword list + strategic-mode
toggle). Keyword **locked at cluster creation**.
- **`SlugGenerator`** (flat keyword slug, no dates).
-- **`SeoAdapter`** — Yoast / Rank Math / SEOPress strategies + native fallback (title +
+- **`SeoAdapter`**: Yoast / Rank Math / SEOPress strategies + native fallback (title +
meta); **suppress plugin schema** on our objects.
-- **`SchemaGraph`** — native JSON-LD: Article/NewsArticle, `datePublished` +
+- **`SchemaGraph`**: native JSON-LD: Article/NewsArticle, `datePublished` +
`dateModified`, author, publisher, image, `mainEntityOfPage`, `citation` → sources.
(`about`/`mentions`/entity hubs arrive with phase 3.)
- **Cost controls live:** batch API path, content-hash + prompt caching, per-feed cost
tracking, hard daily spend cap pauses paid stages.
- **Diff view:** original vs rewrite in the admin.
-*Ships:* an SEO content engine — original faithful posts, one living URL per story,
+*Ships:* an SEO content engine: original faithful posts, one living URL per story,
freshness signals, correct meta + schema, bounded cost.
**Exit criteria:** two near-duplicate feed items produce ONE living post; a genuine
@@ -123,7 +123,7 @@ the active SEO plugin with no duplicate schema; spend cap halts paid work on tri
---
-## Phase 3 — Entity engine (the moat) — ✅ built
+## Phase 3: Entity engine (the moat), ✅ built
**Goal:** auto-build the entity hub graph + internal links + entity schema.
@@ -139,19 +139,19 @@ the active SEO plugin with no duplicate schema; spend cap halts paid work on tri
- **Delegation-rule config** (`ARCHITECTURE.md` §5) + admin UI: map entity type → CPT →
field map → match/research/linking settings.
-- **`EntityResolver`** — normalize → exact/alias → type-constrained fuzzy+embedding →
+- **`EntityResolver`**: normalize → exact/alias → type-constrained fuzzy+embedding →
band (link / create / skip-and-log); **batch-dedup guard**.
-- **`EntityResearcher`** — default in-article stub (zero deps); pluggable provider path
+- **`EntityResearcher`**: default in-article stub (zero deps); pluggable provider path
for cited external facts + `sameAs`. Strict cited stubs.
-- **`EntityLinker`** — first-mention contextual links, capped per post; relationship
+- **`EntityLinker`**: first-mention contextual links, capped per post; relationship
taxonomy (bidirectional).
-- **`HubRenderer`** — pillar template: description + cited facts + `sameAs` +
+- **`HubRenderer`**: pillar template: description + cited facts + `sameAs` +
auto-growing related-posts list; Organization/Person/Product schema.
- **Schema upgrade:** add `about`/`mentions` to post graph now that hubs exist.
-- **`EntityMerge`** admin tool — merge B→A, repoint everything.
+- **`EntityMerge`** admin tool: merge B→A, repoint everything.
- **Ambiguous-entity queue** in the dashboard.
-*Ships:* the durable SEO advantage — topical authority via auto hub-and-spoke linking +
+*Ships:* the durable SEO advantage: topical authority via auto hub-and-spoke linking +
machine-readable entity/citation graph.
**Exit criteria:** a mentioned company links to an existing hub when present, creates a
@@ -161,7 +161,7 @@ related list.
---
-## Phase 4 — Polish + premium seams — ✅ built
+## Phase 4: Polish + premium seams, ✅ built
**Goal:** the indexing/freshness loop, media, EEAT, ops maturity, and the paid hooks.
@@ -171,9 +171,9 @@ related list.
> tracking in `Importer` (consecutive-fail → dead) + admin notice, a full **Settings**
> screen (provider/key, spend cap, post type, author, images, IndexNow, clustering,
> keywords, feeds), and `docs/EXTENDING.md` documenting every provider/hook seam. All
-> phases 0–4 complete.
+> phases 0-4 complete.
-- **`IndexNow`** — ping on publish + novelty update; sitemap `lastmod` via SEO plugin.
+- **`IndexNow`**: ping on publish + novelty update; sitemap `lastmod` via SEO plugin.
- **Images:** featured-image import (configurable off/import/if-licensed, default
conservative) + **AI alt text**.
- **EEAT:** configurable author (bio + `author` schema); optional AI-assisted
@@ -182,7 +182,7 @@ related list.
- **Observability dashboard:** pipeline funnel, per-feed cost, facts-guard flags,
ambiguous-entity queue, feed health.
- **Premium provider seams finalized & documented:** keyword-data provider (layer 3),
- research provider, headless extraction, multilingual, strategic-mode packs — all as
+ research provider, headless extraction, multilingual, strategic-mode packs, all as
consumers of the phase-0…3 interfaces.
*Ships:* production-grade ops + fast indexing + the freemium boundary.
@@ -200,7 +200,7 @@ provider registers through the public interface with zero core changes.
- **Cost discipline:** no feature added without its `CostMeter` accounting.
- **WordPress.org readiness:** external-API disclosure in `readme.txt`, BYO-key opt-in,
no obfuscation, sanitize/escape/nonce everywhere, GPL-compatible deps only.
-- **Docs:** keep `DECISIONS.md` authoritative — if a decision changes during build,
+- **Docs:** keep `DECISIONS.md` authoritative: if a decision changes during build,
amend it there first, then the code.
---
@@ -208,10 +208,10 @@ provider registers through the public interface with zero core changes.
## Open items to verify at build time
1. **Readability library:** confirm a maintained, **GPL-compatible** PHP readability
- port (license check is a hard gate — Apache-2.0 is compatible with GPLv3 but not
+ port (license check is a hard gate; Apache-2.0 is compatible with GPLv3 but not
GPLv2-only; pick accordingly given GPL-2.0-**or-later**).
2. **Cheapest viable model IDs + batch availability** for the default `Ai_Provider`
- (decided at implementation, behind the adapter — not baked into core).
+ (decided at implementation, behind the adapter, not baked into core).
3. **Embedding model + dims** (sets `ai_vectors.dims` and the cosine packing format).
4. **SEO-plugin meta keys** for Yoast / Rank Math / SEOPress current versions (adapter
strategies); confirm each exposes a schema-suppression filter.
diff --git a/src/Admin/Admin.php b/src/Admin/Admin.php
index cfc40a9..f88f6c6 100644
--- a/src/Admin/Admin.php
+++ b/src/Admin/Admin.php
@@ -2,6 +2,7 @@
namespace AggregateIt\Admin;
+use AggregateIt\Ai\ProviderFactory;
use AggregateIt\Database\Schema;
use AggregateIt\Pipeline\Pipeline;
use AggregateIt\Plugin;
@@ -49,6 +50,47 @@ public function register(): void {
add_action( 'admin_post_aggregate_it_clear_logs', [ $this, 'handle_clear_logs' ] );
add_action( 'admin_post_aggregate_it_reset', [ $this, 'handle_reset' ] );
add_action( 'admin_notices', [ $this, 'feed_health_notice' ] );
+ add_action( 'admin_notices', [ $this, 'provider_notice' ] );
+ add_action( 'admin_notices', [ $this, 'embedding_notice' ] );
+ }
+
+ public function provider_notice(): void {
+ if ( ! current_user_can( 'manage_options' ) ) {
+ return;
+ }
+ $reason = get_transient( ProviderFactory::BLOCKED_TRANSIENT );
+ if ( ! $reason ) {
+ return;
+ }
+ $message = $reason === 'undecryptable'
+ ? __( 'Aggregate It: the saved AI API key cannot be decrypted, so articles are on hold. Enter the key again in Settings.', 'aggregate-it' )
+ : __( 'Aggregate It: no AI service is configured, so articles are on hold. Choose a service and add its API key in Settings.', 'aggregate-it' );
+ printf(
+ '