Keep the web scanner out of a store's endless link spaces

Wapiti's crawler was spending its whole four hour budget on one site because an
online store builds a fresh URL for the same product on every page it appears
on: the add to cart link carries an encoded copy of the page you came from, the
review form has an address per product, and the login link carries a referer
the same way.  A populated catalog turns that into thousands of near identical
links the crawler never reaches the end of, so it ran to its hard limit while
the same scan on a site without the catalog loaded finished in minutes.

The engine now tells wapiti to skip those spaces, and the add to cart, review
post, login referer, logout, section load, wishlist and product compare paths,
by passing each as an excluded address rooted at the target.  A discovered path
that lands in one of them is also dropped before it is handed back as a place to
start crawling.  None of these is real attack surface and every ordinary page is
still crawled, so coverage is unchanged while a big store now finishes in the
same minutes as a small one.  A site that is not a store serves none of these
paths, so the exclusions match nothing and are safe to send every time.

Author: hackrange

app/core/dast_engines.py

@@ -263,10 +263,24 @@ def command(tool: str, url: str, host: str, profile: str, workdir: Path,
                 "--max-scan-time", str(max(600, hard - 300)), "-A", _ua(),
                 "--store-session", str(workdir / "wapiti-session"),
                 "--store-config", str(workdir / "wapiti-config")]
+        # Keep the crawler out of the endless, low value URL spaces an online
+        # store generates, so it spends its time on real pages instead.
+        excludes = _crawl_excludes(url)
+        for bad in excludes:
+            argv += ["-x", bad]
         # Routes from the code and paths found by ffuf, as extra places to
-        # start crawling from.  The crawler cannot reach an unlinked route.
-        for extra in _same_site(url, paths)[:60]:
+        # start crawling from.  The crawler cannot reach an unlinked route.  A
+        # discovered path that is itself one of the excluded traps is dropped,
+        # so it is never handed back in as a starting point.
+        trap_prefixes = [bad[:-1] for bad in excludes]  # drop the trailing '*'
+        seeded = 0
+        for extra in _same_site(url, paths):
+            if any(extra.startswith(px) for px in trap_prefixes):
+                continue
             argv += ["-s", extra]
+            seeded += 1
+            if seeded >= 60:
+                break
         return argv, report
     if tool == "sqlmap":
         out = workdir / "sqlmap"
@@ -310,6 +324,40 @@ def _same_site(url: str, paths) -> list:
     return out
 
 
+# URL spaces an online store, Magento above all, builds one of per product, so
+# a crawler never reaches the end of them, together with the ones that throw
+# the scanner's own session away.  None is real attack surface: the add to cart
+# and login "referer" links only re encode the page you came from, so the same
+# product turns into a different URL on every page it is listed on; the review
+# post address needs a signed in session; and logout just ends the session the
+# scan is using.  Left in, wapiti spends its whole budget walking thousands of
+# near identical links.  Measured against a populated Magento catalog that was
+# four hours, cut to minutes once these were excluded, while every real page is
+# still crawled.  A site that is not Magento serves none of these paths, so the
+# patterns match nothing and are safe to pass every time.  Each is a full
+# address ending in a wildcard, because wapiti matches an exclude against the
+# whole URL from start to end.
+_CRAWL_TRAPS = (
+    "/checkout/cart/add/*",
+    "/checkout/cart/delete/*",
+    "/review/product/post/*",
+    "/customer/account/login/referer/*",
+    "/customer/account/logout*",
+    "/customer/section/load*",
+    "/wishlist/*",
+    "/catalog/product_compare/*",
+)
+
+
+def _crawl_excludes(url: str) -> list:
+    """Absolute wildcard URLs for the crawl traps above, rooted at the target."""
+    from urllib.parse import urlparse
+
+    p = urlparse(url)
+    base = f"{p.scheme}://{p.netloc}"
+    return [base + tail for tail in _CRAWL_TRAPS]
+
+
 def links_in(url: str, body: str) -> list:
     """Same site links in a page, the ones with a query string first."""
     from urllib.parse import urljoin