fix: only treat the base domain and its subdomains as internal

[eeshsaxena]

Summary

is_external_url decided whether a link belongs to the site being crawled with a bare suffix check:

return not url_domain.endswith(base)

Any host whose name merely ends with the base domain string passed that check and was classified as internal:

is_external_url("http://malicious-example.com", "example.com")  # False (treated as internal)
is_external_url("http://notexample.com/path", "example.com")    # False (treated as internal)

base comes from get_base_domain(), so it is a bare registrable domain and these are genuinely different sites an attacker can register. The result feeds link classification in content_scraping_strategy.py and the deep-crawl scope check in domain_mapper.py, so a crawl scoped to one domain could follow links onto look-alike domains.

The fix matches what the rest of the codebase already does (deep_crawling/filters.py, domain_mapper.py): internal means equal to the base domain, or a dot-separated subdomain of it (domain == base or domain.endswith(f".{base}")). Real subdomains (blog.example.com) and relative URLs are unchanged. The www. strip also moves from .replace("www.", "") to .removeprefix("www."), so a host containing www. somewhere other than the front is not silently rewritten.

No existing issue filed — happy to open one first if you'd prefer it tracked separately.

List of files changed and why

• crawl4ai/utils.py — rewrite is_external_url's membership test from endswith(base) to == base or endswith("." + base), and switch the www. strip to removeprefix.
• tests/regression/test_reg_utils.py — add TestIsExternalUrl covering the base domain, a subdomain, www., the two look-alike cases (notexample.com, malicious-example.com), the base domain as a prefix, an unrelated domain, a relative URL, and a mailto: link.

How Has This Been Tested?

Ran pytest tests/regression/test_reg_utils.py. Against the current code, the two look-alike cases in the new TestIsExternalUrl fail (demonstrating the bug); with the fix applied the full file passes.

Note: tests/test_link_extractor.py::test_link_extractor fails on a clean checkout as well (the async test isn't picked up by the configured asyncio plugin), so it's unrelated to this change.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes