4.1.0: modern engine picks the best client IP, with exhaustive tests

.github/workflows/ci.yml

@@ -81,7 +81,7 @@ jobs:
         run: python -m build
       - name: Check distribution metadata
         run: python -m twine check dist/*
-      - uses: actions/upload-artifact@v4
+      - uses: actions/upload-artifact@v7
         with:
           name: artifacts
           path: dist/*
@@ -98,7 +98,7 @@ jobs:
       - build
       - ci
     steps:
-      - uses: actions/download-artifact@v4
+      - uses: actions/download-artifact@v8
         with:
           name: artifacts
           path: dist

CHANGELOG.md

@@ -1,3 +1,41 @@
+## 4.1.0
+
+Best match (modern engine only; legacy is unchanged). Results can differ from 4.0.0 on well-formed input,
+always toward a better address; use `algorithm="legacy"` for exact v3 results:
+- Addresses are ranked public > private > link-local > loopback. Unspecified (`0.0.0.0`, `::`),
+  multicast, broadcast, and reserved addresses are never returned. Python reports multicast as
+  `is_global`, so v3 could return `224.0.0.1` as the client.
+- Without `proxy_count` / `proxy_list`, the first public hop of a chain wins, not only the first hop:
+  `10.0.0.1, 177.139.233.139` now yields `177.139.233.139`. With `leftmost=False` the scan runs from the
+  right. With proxy settings, the client position is fixed exactly as before.
+- `trusted_route` is `True` for any address resolved through a matching proxy config, including private
+  clients; v3 reported `False` for them.
+- New exhaustive tests: every 1–3 hop chain over ten address kinds, in every leftmost / strict / proxy
+  configuration, checked against an independent reference model; every header assignment and dict order;
+  every spelling of a hop; never-worse-than-legacy across the whole matrix; seeded fuzzing.
+
+Enhance (modern engine only; legacy is unchanged):
+- Parse RFC 7239 `Forwarded` elements by their `for=` value, including quoted, bracketed IPv6 with a port.
+  Previously `Forwarded` never produced an IP, so when it is present it can now resolve at its existing
+  precedence slot. Obfuscated hops (`for=unknown`, `for=_hidden`) count as invalid tokens.
+- New default headers, added only between the 4.0.0 entries and `REMOTE_ADDR`, so none outranks a header
+  that resolved requests before: Azure Front Door `X-Azure-ClientIP`, DigitalOcean `DO-Connecting-IP`,
+  Envoy/Istio `X-Envoy-External-Address`, plus the missing `HTTP_X_CLIENT_IP` and raw `X-AppEngine-User-IP`
+  forms of headers already on the list.
+- Header names match case-insensitively (`-` and `_` equivalent), so lowercase keys such as AWS Lambda's
+  work. Exact keys still take priority.
+
+Harden (modern engine only):
+- Reject malformed tokens instead of truncating them: unclosed brackets (`[::1`), text after a bracket
+  (`[::1]junk`), and non-numeric, empty, or out-of-range ports (`1.2.3.4:abc`, `1.2.3.4:70000`). Note v3
+  accepted `1.2.3.4:abc` as `1.2.3.4`.
+- Non-string header values (`None`, bytes) are skipped instead of raising `AttributeError`.
+- `proxy_list` entries are stripped of whitespace. An empty entry now raises `ValueError`: it used to match
+  every address and mark any spoofed chain as trusted.
+
+CI:
+- Bump `actions/upload-artifact` to v7 and `actions/download-artifact` to v8, which run on Node 24.
+
 ## 4.0.0
 
 Community (thank you!):

README.md

@@ -97,12 +97,29 @@ ip, trusted_route = ipw.get_client_ip(meta, strict=False)
 | Output | Description |
 | --- | --- |
 | `ip` | `IPv4Address`, `IPv6Address`, or `None` |
-| `trusted_route` | `True` when `proxy_count` or `proxy_list` was configured and matched |
+| `trusted_route` | `True` when `proxy_count` or `proxy_list` was configured and matched, for any returned IP (v3 only set it for public IPs) |
 
 ### Selection rules
 
-Headers are checked in precedence order. The first **public** IP found wins; otherwise the first
-**private** IP; otherwise the first **loopback** IP; otherwise `None`.
+Headers are checked in precedence order. Every address is ranked:
+
+| Rank | Addresses |
+| --- | --- |
+| 1. public | globally routable |
+| 2. private | RFC 1918, IPv6 ULA, CGNAT `100.64.0.0/10`, documentation ranges |
+| 3. link-local | `169.254.0.0/16`, `fe80::/10` |
+| 4. loopback | `127.0.0.0/8`, `::1` |
+| never returned | `0.0.0.0`, `::`, multicast, broadcast, reserved |
+
+The first **public** IP wins. If none is found, the best-ranked IP wins, and the earlier header
+wins a tie.
+
+Within one header, the client entry depends on your proxy settings:
+
+- **`proxy_count` / `proxy_list` set:** the entry just before your trusted proxies, exactly as in v3.
+- **Neither set:** the first public entry in the chain, not only the first entry. So
+  `10.0.0.1, 177.139.233.139` yields `177.139.233.139` (v3 returned `10.0.0.1`). With
+  `leftmost=False` the chain is scanned from the right.
 
 ```mermaid
 flowchart TD
@@ -115,13 +132,19 @@ flowchart TD
     E -->|yes| F["Pick the client entry"]
     F --> G{"Public IP?"}
     G -->|yes| H["Return (ip, trusted_route)"]
-    G -->|no| I["Keep as private or loopback fallback"]
+    G -->|no| I["Keep if it outranks the current fallback"]
     I --> B
-    B -->|no headers left| J["Return first private, else loopback, else None"]
+    B -->|no headers left| J["Return the best fallback, else None"]
 ```
 
+The legacy engine keeps v3's rules. A combination suite checks that the modern engine never returns a
+worse address than legacy for the same input.
+
 Ports are stripped (`1.2.3.4:8080`, `[2001:db8::1]:443`) and IPv4-mapped IPv6 addresses
-(`::ffff:1.2.3.4`) are returned as plain IPv4.
+(`::ffff:1.2.3.4`) are returned as plain IPv4. RFC 7239 `Forwarded` elements are read by their
+`for=` value (`for="[2001:db8::1]:4711";proto=https`). Malformed tokens such as `[::1`,
+`[::1]junk`, or `1.2.3.4:abc` are rejected rather than truncated. Header names match
+case-insensitively, so lowercase keys (AWS Lambda / API Gateway v2) work too.
 
 ## Default header precedence
 
@@ -133,7 +156,7 @@ Ports are stripped (`1.2.3.4:8080`, `[2001:db8::1]:443`) and IPv4-mapped IPv6 ad
     "HTTP_X_REAL_IP",
     "HTTP_X_FORWARDED",          # Squid
     "HTTP_X_CLUSTER_CLIENT_IP",  # Rackspace LB, Riverbed Stingray
-    "HTTP_FORWARDED_FOR",        # RFC 7239
+    "HTTP_FORWARDED_FOR",        # de facto variant
     "HTTP_FORWARDED",            # RFC 7239
     "HTTP_CF_CONNECTING_IP",     # Cloudflare
     "HTTP_TRUE_CLIENT_IP",       # Cloudflare Enterprise, Akamai
@@ -151,10 +174,22 @@ Ports are stripped (`1.2.3.4:8080`, `[2001:db8::1]:443`) and IPv4-mapped IPv6 ad
     "FLY-CLIENT-IP",
     "FORWARDED",
     "CLIENT-IP",
+    # added after 4.0.0 — always below every earlier entry, above REMOTE_ADDR
+    "HTTP_X_CLIENT_IP",          # Microsoft Azure (Django/WSGI form)
+    "X-APPENGINE-USER-IP",       # Google App Engine (raw form)
+    "HTTP_X_AZURE_CLIENTIP",     # Azure Front Door
+    "X-AZURE-CLIENTIP",
+    "HTTP_DO_CONNECTING_IP",     # DigitalOcean App Platform
+    "DO-CONNECTING-IP",
+    "HTTP_X_ENVOY_EXTERNAL_ADDRESS",  # Envoy / Istio
+    "X-ENVOY-EXTERNAL-ADDRESS",
     "REMOTE_ADDR",               # direct connection
 )
 ```
 
+Headers released earlier never move. New ones are added only just above `REMOTE_ADDR`, so an
+upgrade can never let a new header outrank one that already resolved your requests.
+
 Narrow it to what your infrastructure actually sets:
 
 ```python

python_ipware/__version__.py

@@ -1 +1 @@
-__version__ = "4.0.0"
+__version__ = "4.1.0"

python_ipware/modern/defaults.py

@@ -1,8 +1,17 @@
 """Default header precedence for the modern engine.
 
-Superset of the v3 list, adding widely-deployed CDN/edge headers. Order is
-most-to-least trustworthy for a typical deployment; the first header that
-yields a usable IP wins.
+Superset of the v3 list, adding widely-deployed CDN/edge headers. Headers are
+scanned in order; the first one that yields a globally routable IP wins, with
+private and then loopback addresses as fallbacks.
+
+Ordering rule: existing entries never move. New headers are added only in the
+block just above ``REMOTE_ADDR``, so a new header can outrank the raw socket
+address but never a header that already resolved a request in an earlier
+release.
+
+Every default here can be forged by a client that reaches the app directly.
+If all traffic arrives through one known edge, pass an explicit
+``precedence`` naming only that edge's header.
 """
 
 DEFAULT_PRECEDENCE: tuple[str, ...] = (
@@ -12,8 +21,8 @@
     "HTTP_X_REAL_IP",
     "HTTP_X_FORWARDED",
     "HTTP_X_CLUSTER_CLIENT_IP",
-    "HTTP_FORWARDED_FOR",
-    "HTTP_FORWARDED",
+    "HTTP_FORWARDED_FOR",  # de facto variant; not defined by RFC 7239
+    "HTTP_FORWARDED",  # RFC 7239 (for=...;proto=...), parsed per element
     "HTTP_CF_CONNECTING_IP",  # Cloudflare
     "HTTP_TRUE_CLIENT_IP",  # Cloudflare Enterprise / Akamai
     "HTTP_FASTLY_CLIENT_IP",  # Fastly / Firebase
@@ -30,5 +39,14 @@
     "FLY-CLIENT-IP",
     "FORWARDED",
     "CLIENT-IP",
-    "REMOTE_ADDR",
+    # --- added after 4.0.0: below every earlier entry, above REMOTE_ADDR ---
+    "HTTP_X_CLIENT_IP",  # Azure X-Client-IP in Django/WSGI form
+    "X-APPENGINE-USER-IP",  # Google App Engine, raw header form
+    "HTTP_X_AZURE_CLIENTIP",  # Azure Front Door
+    "X-AZURE-CLIENTIP",
+    "HTTP_DO_CONNECTING_IP",  # DigitalOcean App Platform
+    "DO-CONNECTING-IP",
+    "HTTP_X_ENVOY_EXTERNAL_ADDRESS",  # Envoy / Istio
+    "X-ENVOY-EXTERNAL-ADDRESS",
+    "REMOTE_ADDR",  # direct connection; always last
 )

python_ipware/modern/engine.py

@@ -1,23 +1,28 @@
 """The modern python-ipware engine.
 
-Behavior-compatible with the v3 algorithm on the essentials, but cleaner and
-hardened:
-
-* Superset header precedence (adds True-Client-IP, Fastly, App Engine, Azure).
-* Robust IPv6 / bracketed-port / IPv4-mapped parsing.
-* Same "best IP" fallback ladder: prefer a globally routable address; else the
-  first private; else loopback.
+Same inputs, outputs and proxy semantics as v3, with a better best-match:
+
+* Superset header precedence (Forwarded parsing, more CDN / edge headers).
+* Robust IPv6 / bracketed-port / IPv4-mapped / RFC 7239 parsing.
+* Explicit ranking: global > private > link-local > loopback. Unspecified,
+  multicast, broadcast and reserved addresses are never returned.
+* Without trusted-proxy config, the first *globally routable* hop of a chain
+  wins, not just the first hop, so ``10.0.0.1, 177.139.233.139`` yields the
+  public address. With ``proxy_count`` / ``proxy_list`` the client position is
+  fixed by the config, exactly as in v3.
 * Same ``strict`` semantics for proxy_count / proxy_list validation.
 * Trusted-proxy matching anchored to the end of the chain. Each ``proxy_list``
   entry is either a CIDR network (``"100.64.0.0/10"``, ``"fd7a:115c:a1e0::/48"``)
   matched by real network membership, or a plain string prefix (``"10.1."``).
+* ``trusted_route`` is True whenever the returned IP came from a chain that
+  passed the configured proxy validation, whatever its tier.
 """
 
 import ipaddress
 from typing import Optional, Union
 
 from .defaults import DEFAULT_PRECEDENCE
-from .parsers import IpAddressType, split_proxy_chain
+from .parsers import TIER_GLOBAL, TIER_REJECT, IpAddressType, ip_tier, split_proxy_chain
 
 OptionalIp = Optional[IpAddressType]
 IpNetworkType = Union[ipaddress.IPv4Network, ipaddress.IPv6Network]
@@ -55,23 +60,52 @@ def __init__(
             raise ValueError("proxy_count must be non-negative")
         if proxy_list is not None and not all(isinstance(p, str) for p in proxy_list):
             raise ValueError("All elements in the proxy list must be strings.")
+        proxy_list = [p.strip() for p in proxy_list or []]
+        # An empty prefix matches every address, which would mark any spoofed
+        # chain as trusted. It is always a misconfiguration (e.g. a trailing
+        # comma in an env var), so fail loudly instead.
+        if any(not p for p in proxy_list):
+            raise ValueError("proxy_list entries must not be empty.")
 
         self.precedence = precedence or DEFAULT_PRECEDENCE
         self.leftmost = leftmost
         self.proxy_count = proxy_count
-        self.proxy_list = list(proxy_list or [])
+        self.proxy_list = proxy_list
         self._proxy_matchers = [_compile_proxy_matcher(p) for p in self.proxy_list]
 
     # -- meta access --------------------------------------------------------
 
-    def _get_meta_value(self, meta: dict[str, str], key: str) -> str:
+    @staticmethod
+    def _fold(key: str) -> str:
+        return key.upper().replace("-", "_")
+
+    def _get_meta_value(
+        self,
+        meta: dict[str, str],
+        key: str,
+        folded: Optional[dict[str, object]] = None,
+    ) -> str:
         meta = meta or {}
-        return meta.get(key, meta.get(key.replace("_", "-"), "")).strip()
+        value = meta.get(key)
+        if value is None:
+            value = meta.get(key.replace("_", "-"))
+        # Exact keys win; the folded view only fills gaps, so lowercase keys
+        # (AWS Lambda / API Gateway v2, raw ASGI dicts) still match.
+        if value is None and folded is not None:
+            value = folded.get(self._fold(key))
+        # Header values are text; anything else (None, bytes, lists from a
+        # misbehaving adapter) is ignored rather than crashing the lookup.
+        return value.strip() if isinstance(value, str) else ""
 
     def _get_meta_values(self, meta: dict[str, str]) -> list[str]:
+        meta = meta or {}
+        folded: dict[str, object] = {}
+        for k, v in meta.items():
+            if isinstance(k, str):
+                folded.setdefault(self._fold(k), v)
         values: list[str] = []
         for key in self.precedence:
-            value = self._get_meta_value(meta, key)
+            value = self._get_meta_value(meta, key, folded)
             if value:
                 values.append(value)
         return values
@@ -103,22 +137,32 @@ def _proxy_list_valid(self, chain: list[IpAddressType], strict: bool) -> bool:
     # -- selection ----------------------------------------------------------
 
     def _best_from_chain(self, chain: list[IpAddressType]) -> tuple[OptionalIp, bool]:
-        # ``chain`` is already client-first (see get_client_ip).
-        if not chain:
-            return None, False
+        # ``chain`` is already client-first (see get_client_ip) and non-empty.
         if self.proxy_list:
             return chain[-(len(self.proxy_list) + 1)], True
         if self.proxy_count is not None:
             return chain[-(self.proxy_count + 1)], True
-        return chain[0], False
+        # No trusted-proxy config, so no position in the chain is verified.
+        # Take the first globally routable hop; otherwise the best-ranked hop,
+        # earliest on ties. This never picks a worse address than v3's chain[0].
+        best: OptionalIp = None
+        best_tier = TIER_REJECT
+        for ip in chain:
+            tier = ip_tier(ip)
+            if tier == TIER_GLOBAL:
+                return ip, False
+            if tier > best_tier:
+                best, best_tier = ip, tier
+        return best, False
 
     # -- public API ---------------------------------------------------------
 
-    def get_client_ip(
-        self, meta: dict[str, str], strict: bool = False
-    ) -> tuple[OptionalIp, bool]:
-        loopback: list[IpAddressType] = []
-        private: list[IpAddressType] = []
+    def get_client_ip(self, meta: dict[str, str], strict: bool = False) -> tuple[OptionalIp, bool]:
+        # Best non-global candidate so far. Strictly-greater comparison keeps
+        # the earliest header on ties, preserving header precedence.
+        fallback: OptionalIp = None
+        fallback_tier = TIER_REJECT
+        fallback_trusted = False
 
         for raw in self._get_meta_values(meta):
             chain = split_proxy_chain(raw, strict)
@@ -136,15 +180,10 @@ def get_client_ip(
             ip, trusted = self._best_from_chain(chain)
             if ip is None:
                 continue
-            if ip.is_global:
+            tier = ip_tier(ip)
+            if tier == TIER_GLOBAL:
                 return ip, trusted
-            if ip.is_loopback:
-                loopback.append(ip)
-            else:
-                private.append(ip)
-
-        if private:
-            return private[0], False
-        if loopback:
-            return loopback[0], False
-        return None, False
+            if tier > fallback_tier:
+                fallback, fallback_tier, fallback_trusted = ip, tier, trusted
+
+        return fallback, fallback_trusted