FetcherBolt is over 1100 lines now, FetcherThread alone is ~430 and mixes robots, sitemaps, crawl-delay, protocol calls and status emission. #867 and #504 would add more on top.
Plan: characterization tests first, then extract the queue classes as-is (same package, package-private), then thin down FetcherThread step by step. No behaviour change — anything fishy found along the way gets its own issue (first one: #2058). No new public API, extensibility stays on the streams.
Maybe later a common fetch kernel with SimpleFetcherBolt, but that's a separate discussion.
WDYT?
FetcherBolt is over 1100 lines now, FetcherThread alone is ~430 and mixes robots, sitemaps, crawl-delay, protocol calls and status emission. #867 and #504 would add more on top.
Plan: characterization tests first, then extract the queue classes as-is (same package, package-private), then thin down FetcherThread step by step. No behaviour change — anything fishy found along the way gets its own issue (first one: #2058). No new public API, extensibility stays on the streams.
Maybe later a common fetch kernel with SimpleFetcherBolt, but that's a separate discussion.
WDYT?