LDAPUtils.newDN rejects DNs with a non-leading '#' in an attribute value (RFC 4514) #1162

Describe the bug

LDAPUtils.newDN() / isDN() run a dnRule regex pre-check before DN.valueOf(). Its unquoted attribute value class, [^,=\\+<>#;\\\\\"], excludes # at every position of the value. RFC 4514 forbids an unescaped # only as the first character, where it starts a hexstring. So a valid DN with # inside a value fails the pre-check. newDN() then silently returns DN.rootDN() and isDN() returns false, while DN.valueOf() accepts the same string.

This is the same class of problem as #1160 (= inside a value), fixed in #1161. The regex follows the RFC 2253 parse grammar, which listed both = and # as special characters.

There is also a round-trip problem: OpenDJ's own DN.toString() writes a non-leading # unescaped. A DN that OpenAM builds with the SDK and serialises back is therefore rejected by newDN(). The escaped input ou=…\#v1,dc=x is accepted, but DN.valueOf(...).toString() turns it into ou=…#v1,dc=x, which the pre-check rejects.

To Reproduce

String dn = "ou=https://idp.example.com/metadata#v1,dc=openam,dc=org";
LDAPUtils.isDN(dn);   // false
LDAPUtils.newDN(dn);  // DN.rootDN()
DN.valueOf(dn);       // succeeds, 3 RDNs

String escaped = "ou=https://idp.example.com/metadata\\#v1,dc=openam,dc=org";
LDAPUtils.isDN(escaped);                            // true
LDAPUtils.isDN(DN.valueOf(escaped).toString());     // false: toString() drops the escape

A realistic trigger is a SAML/WS-Fed entity ID with a URL fragment, which ends up as an RDN value of an SMS DN, the same path as the Google Workspace entity ID in #1160.

Checked against opendj-core 5.1.2 (the version OpenAM pins):

Expected behavior

dnRule accepts an unescaped # at any position except the first character of an unquoted value, as RFC 4514 allows. A leading # stays reserved for the #hexstring form.

RFCs

• RFC 4514 §2.4: a # must be escaped only when it is "occurring at the beginning of the string".
• RFC 4514 §3, ABNF:

  string   = [ ( leadchar / pair ) [ *( stringchar / pair ) ( trailchar / pair ) ] ]
  leadchar = LUTF1 / UTFMB
  LUTF1    = %x01-1F / %x21 / %x24-2A / %x2D-3A / %x3D / %x3F-5B / %x5D-7F
  trailchar  = TUTF1 / UTFMB
  TUTF1    = %x01-1F / %x21 / %x23-2A / %x2D-3A / %x3D / %x3F-5B / %x5D-7F
  stringchar = SUTF1 / UTFMB
  SUTF1    = %x01-21 / %x23-2A / %x2D-3A / %x3D / %x3F-5B / %x5D-7F
  hexstring = SHARP 1*hexpair
  

  # is %x23. LUTF1 skips it (%x21 then %x24), while TUTF1 and SUTF1 include it (%x23-2A). So # is invalid only as the leading character, where it would be ambiguous with hexstring.
• RFC 4514 Appendix B (changes since RFC 2253) lists this explicitly. The Section 3 ABNF "did not require escaping of equals sign ('=' U+003D) characters" and "did not require escaping of non-leading number sign ('#' U+0023) characters".
• RFC 2253 §3, obsoleted by RFC 4514: stringchar = <any character except one of special, "\" or QUOTATION> with special = "," / "=" / "+" / "<" / ">" / "#" / ";". That is the origin of the current character class. Even RFC 2253 §2.4 only required escaping a # "occurring at the beginning of the string" when generating DNs.

Suggested fix

In every copy of the unquoted-value alternative in dnRule, exclude # only from the first character. On top of #1161 that would look like:

(?:(?:[^,+<>#;\\\\\"]|<escape>|<hexpair>)(?:[^,+<>;\\\\\"]|<escape>|<hexpair>)*)?

The first character still cannot be #, so this alternative stays disjoint from the #(?:[\\dA-Fa-f]{2})+ hexstring alternative, with no extra backtracking. Add tests for a non-leading #, for ou=#x (still rejected), and for a round trip through DN.valueOf(...).toString().

Desktop

• Version: OpenAM 14.6.5 and later, including master / 16.x. The pre-check was introduced in FIX check isDN by regular expression #436; before that, newDN() only checked contains("=").

Additional context

Found while reviewing #1161.