Check unique elementIds and device functions in the HTML view

README.md

@@ -103,14 +103,33 @@ complete and are not a legal assessment.
 - Criteria with `check.type: links` check every `RelatedResource` of the
   passport for the required attributes and send a HEAD request to its URL
   (at most 20 URLs per passport). A URL that does not answer gives a warning.
-- Criteria with `check.type: html` (DPP-SEC-014) request the product
-  identifier with `Accept: text/html` and list the URLs a browser loads without
-  user action while it displays the HTML view (scripts, style sheets, fonts,
-  images, frames, preloads, CSS `url()` and `@import`, meta refresh). With
-  `external_resources: forbidden` every such URL on another host than the
-  passport's fails the criterion. Without an HTML view the result is
-  `skipped` (`no_evidence`). Resources added by scripts at run time are not
-  seen.
+- Criteria with `check.type: html` (DPP-SEC-014, DPP-SEC-021) request the
+  product identifier with `Accept: text/html`. Without an HTML view the result
+  is `skipped` (`no_evidence`).
+  - With `external_resources: forbidden` dpplint lists the URLs a browser loads
+    without user action while it displays the HTML view (scripts, style
+    sheets, fonts, images, frames, preloads, CSS `url()` and `@import`, meta
+    refresh); every such URL on another host than the passport's fails the
+    criterion. Resources added by scripts at run time are not seen.
+  - With `device_features: forbidden` dpplint searches, without running any
+    script, the inline scripts, event handler attributes and `javascript:`
+    URLs of the HTML view and the scripts it loads from its own host (at most
+    10, the first 2 MB of each) for browser interfaces that use functions of
+    the reader's device (location, camera, microphone, notifications,
+    contacts, sensors, network information and others), and the markup for
+    frames granted such permissions, file inputs with `capture` and permission
+    elements. Every finding fails the criterion. Code assembled at run time,
+    obfuscated or loaded by a script, and scripts from other hosts, are not
+    seen; an interface name in a comment or string counts.
+- Criteria with `check.type: element_ids` (DPP-DAT-017, DPP-DAT-021) check
+  that elementIds are unique. In the expanded full serialization dpplint
+  compares the elementIds within every array of data elements (`unique:
+  level`, a repeated elementId fails) or across the whole passport (`unique:
+  passport`, an elementId at two places gives a warning; the items of a
+  MultiValuedDataElement count as one place). In the compressed serialization
+  `level` looks for a member name repeated within one JSON object of the
+  passport as delivered (the response, or the posted body), and `passport` is
+  `skipped`.
 - Criteria whose condition does not hold are reported as `skipped`.
 - Every `skipped` criterion carries a `reason` and a `reason_code`
   (`not_applicable`, `no_evidence`, `not_evaluated`, … as defined under

app/controllers/api/v1/validations_controller.rb

@@ -24,7 +24,7 @@ def create
         passport = JSON.parse(request.raw_post)
         return render(json: { error: "body is not a JSON object" }, status: :unprocessable_entity) unless passport.is_a?(Hash)
 
-        render json: PassportLinter.new.run(passport: passport)
+        render json: PassportLinter.new.run(passport: passport, source: request.raw_post)
       rescue JSON::ParserError => e
         render json: { error: "invalid JSON: #{e.message}" }, status: :unprocessable_entity
       end

app/services/element_ids_check.rb

@@ -0,0 +1,137 @@
+# Runs a criterion with check.type element_ids: whether the elementIds of the
+# passport are unique (EN 18223 4.1.2.3, Table 2; "Check types",
+# element_ids, in CRITERIA-FORMAT.md of dpp-criteria).
+#
+# In the expanded full serialization (EN 18223 Annex A; an `elements` array
+# at the top level) the data elements are the objects with an elementId in
+# the `elements` arrays of the passport, of its DataElementCollections and
+# MultiValuedDataElements, and in the `value` array of a MultiValuedDataElement
+# whose items are such objects.
+#
+# unique: level - every elementId once per array of data elements (a
+#   violation for each one repeated). In the compressed serialization the
+#   elementIds are JSON member names, so no JSON object of the passport as
+#   delivered (`source`, the bytes of the response or of the posted body) may
+#   contain a name twice; without those bytes the criterion is skipped.
+# unique: passport - every elementId at one place of the passport (a warning
+#   for each one used at several places). The items of a MultiValuedDataElement
+#   count as one place, because they repeat one structure. Only the expanded
+#   full serialization.
+class ElementIdsCheck
+  Outcome = Struct.new(:skipped, :code, :messages, keyword_init: true)
+
+  Element = Struct.new(:id, :place, keyword_init: true)
+
+  MAX_LISTED = 10
+
+  def initialize(check, passport, source: nil)
+    @check = check
+    @passport = passport
+    @source = source
+  end
+
+  def outcome
+    if full?
+      @check["unique"] == "passport" ? Outcome.new(messages: passport_messages) : Outcome.new(messages: level_messages)
+    elsif @check["unique"] == "passport"
+      Outcome.new(skipped: "the passport is in the compressed serialization, where elementIds cannot be told apart from " \
+                           "attribute names without the data dictionary", code: "not_evaluated")
+    elsif @source.nil?
+      Outcome.new(skipped: "the passport is in the compressed serialization and is not available as delivered, so " \
+                           "repeated member names cannot be seen", code: "not_evaluated")
+    else
+      compressed_level_outcome
+    end
+  end
+
+  # The full serialization has an `elements` array at the top level.
+  def full? = @passport.is_a?(Hash) && @passport["elements"].is_a?(Array)
+
+  private
+
+  def level_messages
+    levels = []
+    walk(@passport["elements"], [], [], levels: levels)
+    repeated = levels.flat_map do |parent, ids|
+      ids.tally.select { |_, n| n > 1 }.map { |id, n| "#{id} #{n} times #{where(parent)}" }
+    end
+    return [] if repeated.empty?
+
+    [{ severity: "violation", message: "elementId repeated within one level: #{listed(repeated)}" }]
+  end
+
+  def passport_messages
+    elements = []
+    walk(@passport["elements"], [], [], elements: elements)
+    places = elements.group_by(&:id).transform_values { |list| list.map(&:place).uniq }
+    shared = places.select { |_, list| list.size > 1 }
+    return [] if shared.empty?
+
+    texts = shared.map { |id, list| "#{id} at #{list.size} places (#{list.map { |p| p.join('/') }.join(', ')})" }
+    [{ severity: "warning", message: "elementId used at more than one place of the passport: #{listed(texts)}" }]
+  end
+
+  # Walks one array of data elements. path: the elementIds above it; place:
+  # the same with "*" for the items of a MultiValuedDataElement.
+  def walk(array, path, place, levels: nil, elements: nil)
+    items = array.select { |e| e.is_a?(Hash) && e["elementId"].is_a?(String) }
+    levels&.push([path, items.map { |e| e["elementId"] }])
+    items.each do |e|
+      id = e["elementId"]
+      elements&.push(Element.new(id: id, place: place + [id]))
+      multi = e["objectType"] == "MultiValuedDataElement"
+      children(e).each do |list|
+        if multi
+          walk_items(list, path + [id], place + [id], levels: levels, elements: elements)
+        else
+          walk(list, path + [id], place + [id], levels: levels, elements: elements)
+        end
+      end
+    end
+  end
+
+  # The items of a MultiValuedDataElement: one level for the repeated-id check;
+  # below them, every item counts as the same place ("*").
+  def walk_items(array, path, place, levels:, elements:)
+    items = array.select { |e| e.is_a?(Hash) && e["elementId"].is_a?(String) }
+    levels&.push([path, items.map { |e| e["elementId"] }])
+    items.each do |e|
+      id = e["elementId"]
+      elements&.push(Element.new(id: id, place: place + ["*"]))
+      multi = e["objectType"] == "MultiValuedDataElement"
+      children(e).each do |list|
+        if multi
+          walk_items(list, path + [id], place + ["*"], levels: levels, elements: elements)
+        else
+          walk(list, path + [id], place + ["*"], levels: levels, elements: elements)
+        end
+      end
+    end
+  end
+
+  # Arrays of data elements below an element: `elements`, and `value` if it
+  # holds objects with an elementId (MultiValuedDataElement items, Annex A).
+  def children(element)
+    %w[elements value].filter_map do |key|
+      list = element[key]
+      list if list.is_a?(Array) && list.any? { |i| i.is_a?(Hash) && i["elementId"].is_a?(String) }
+    end
+  end
+
+  def compressed_level_outcome
+    duplicates = JsonDuplicates.find(@source)
+    return Outcome.new(messages: []) if duplicates.empty?
+
+    texts = duplicates.map { |d| "#{d[:name]} #{d[:count]} times in the object at #{d[:path]}" }
+    Outcome.new(messages: [{ severity: "violation", message: "member name repeated within one JSON object: #{listed(texts)}" }])
+  rescue JsonDuplicates::Invalid => e
+    Outcome.new(skipped: "the passport as delivered cannot be read: #{e.message}", code: "not_evaluated")
+  end
+
+  def where(parent) = parent.empty? ? "at the top level of the passport" : "in #{parent.join('/')}"
+
+  def listed(texts)
+    more = texts.size > MAX_LISTED ? " and #{texts.size - MAX_LISTED} more" : ""
+    "#{texts.first(MAX_LISTED).join('; ')}#{more}"
+  end
+end

app/services/html_check.rb

@@ -12,9 +12,39 @@
 # differs from the host of the final response URL. Links the user has to
 # follow are not counted; resources added by scripts at run time cannot be
 # seen in the document.
+#
+# device_features: forbidden looks, without running any script, for browser
+# interfaces that use functions of the reader's device (DEVICE_INTERFACES) in
+# the inline scripts, event handler attributes and javascript: URLs of the
+# document and in the scripts it loads from its own host (at most MAX_SCRIPTS,
+# the first MAX_SCRIPT_BYTES of each; a script that cannot be retrieved is not
+# checked), and in the markup for frames granted device permissions
+# (DEVICE_PERMISSIONS), file inputs with capture and permission elements.
 class HtmlCheck
   Outcome = Struct.new(:skipped, :code, :messages, keyword_init: true)
 
+  DEVICE_INTERFACES = [
+    ["location", /\.geolocation\b|\bgetCurrentPosition\b|\bwatchPosition\b/],
+    ["camera, microphone or screen", /\bgetUserMedia\b|\bgetDisplayMedia\b|\.mediaDevices\b/],
+    ["notifications", /\bNotification\s*\.\s*requestPermission\b|\bnew\s+Notification\s*\(|\.showNotification\s*\(|\b[Pp]ushManager\b/],
+    ["contacts", /\bnavigator\s*\.\s*contacts\b/],
+    ["Bluetooth, USB, serial, HID or NFC", /\bnavigator\s*\.\s*(?:bluetooth|usb|serial|hid)\b|\bNDEFReader\b/],
+    ["motion or orientation sensors",
+     /\b(?:on)?(?:devicemotion|deviceorientation(?:absolute)?)\b|\bDevice(?:Motion|Orientation)Event\s*\.\s*requestPermission\b|\bnew\s+(?:Accelerometer|Gyroscope|Magnetometer|AbsoluteOrientationSensor|RelativeOrientationSensor|LinearAccelerationSensor|GravitySensor|AmbientLightSensor)\s*\(/],
+    ["context or network information", /\bnavigator\s*\.\s*connection\b|\bgetBattery\s*\(|\bIdleDetector\b/],
+    ["one-time codes from SMS", /\bOTPCredential\b|\btransport\s*:\s*\[\s*["']sms["']/],
+    ["app installation", /\bbeforeinstallprompt\b|\bgetInstalledRelatedApps\b/],
+    ["device behaviour", /\bnavigator\s*\.\s*(?:wakeLock|vibrate)\b/]
+  ].freeze
+  DEVICE_PERMISSIONS = %w[camera microphone geolocation display-capture bluetooth usb serial hid accelerometer gyroscope
+                          magnetometer ambient-light-sensor idle-detection screen-wake-lock].freeze
+  JS_TYPES = %w[text/javascript application/javascript application/ecmascript text/ecmascript module
+                application/x-javascript text/jscript text/x-javascript].freeze
+  URL_ATTRIBUTES = %w[href src action formaction].freeze
+  MAX_SCRIPTS = 10
+  MAX_SCRIPT_BYTES = 2 * 1024 * 1024
+  MAX_FINDINGS = 10
+
   SRC_ELEMENTS = %w[script img iframe frame embed audio video source track].freeze
   SRCSET_ELEMENTS = %w[img source].freeze
   LINK_RELS = %w[stylesheet icon apple-touch-icon mask-icon manifest preload modulepreload
@@ -37,9 +67,23 @@ def outcome
 
     messages = []
     messages << external_message(res) if @check["external_resources"] == "forbidden"
+    messages << device_message(res) if @check["device_features"] == "forbidden"
     Outcome.new(messages: messages.compact)
   end
 
+  # Device interfaces found in a piece of script: [[label, matched text], …].
+  def self.device_uses(code)
+    DEVICE_INTERFACES.filter_map do |label, pattern|
+      match = code.to_s.match(pattern)
+      [label, match[0].gsub(/\s+/, " ")] if match
+    end
+  end
+
+  def self.javascript_type?(type)
+    type = type.to_s.split(";").first.to_s.strip.downcase
+    type.empty? || JS_TYPES.include?(type)
+  end
+
   # The URLs a browser loads without user action, resolved, with the element
   # or construct they come from: [[url, source], …].
   def self.loaded_urls(html, page_url)
@@ -121,6 +165,61 @@ def external_message(res)
       message: "the HTML view loads #{external.size} resource(s) from other hosts than #{own}: #{listed.join(', ')}#{more}" }
   end
 
+  def device_message(res)
+    doc = Nokogiri::HTML5(res.body.to_s)
+    findings = markup_findings(doc) + inline_findings(doc) + script_findings(doc, res.url.to_s)
+    return nil if findings.empty?
+
+    listed = findings.first(MAX_FINDINGS)
+    more = findings.size > MAX_FINDINGS ? " and #{findings.size - MAX_FINDINGS} more" : ""
+    { severity: "violation", message: "the HTML view uses functions of the reader's device: #{listed.join('; ')}#{more}" }
+  end
+
+  def markup_findings(doc)
+    found = []
+    doc.css("iframe[allow]").each do |e|
+      granted = e["allow"].to_s.split(";").map { |d| d.strip.split(/\s+/).first.to_s.downcase } & DEVICE_PERMISSIONS
+      found << "iframe allow grants #{granted.join(', ')}" if granted.any?
+    end
+    found << "file input with capture" if doc.css("input[capture]").any?
+    %w[permission geolocation].each { |name| found << "#{name} element" if doc.css(name).any? }
+    found
+  end
+
+  def inline_findings(doc)
+    found = []
+    doc.css("script").each_with_index do |e, i|
+      next if e["src"] || !self.class.javascript_type?(e["type"])
+
+      self.class.device_uses(e.content).each { |label, use| found << "#{label} (#{use} in inline script #{i + 1})" }
+    end
+    doc.xpath("//@*").each do |a|
+      name = a.name.downcase
+      code = if name.start_with?("on") then a.value
+             elsif URL_ATTRIBUTES.include?(name) && a.value.to_s.strip.downcase.start_with?("javascript:") then a.value
+             end
+      next unless code
+
+      self.class.device_uses(code).each { |label, use| found << "#{label} (#{use} in attribute #{name} of #{a.parent.name})" }
+    end
+    found
+  end
+
+  # Scripts loaded from the host of the page, fetched as text and searched.
+  def script_findings(doc, page_url)
+    own = self.class.web_host(page_url)
+    base = self.class.absolute(doc.at_css("base[href]")&.[]("href"), page_url) || page_url
+    urls = doc.css("script[src]").select { |e| self.class.javascript_type?(e["type"]) }.map { |e| e["src"] }
+    urls += doc.css("link[href]").select { |e| e["rel"].to_s.downcase.split.include?("modulepreload") }.map { |e| e["href"] }
+    urls = urls.filter_map { |u| self.class.absolute(u, base) }.uniq.select { |u| self.class.web_host(u) == own }
+    urls.first(MAX_SCRIPTS).flat_map do |url|
+      res = @resolver.get(url, accept: "*/*")
+      next [] unless res.success?
+
+      self.class.device_uses(res.body.to_s.byteslice(0, MAX_SCRIPT_BYTES).to_s.dup.force_encoding(Encoding::UTF_8).scrub).map { |label, use| "#{label} (#{use} in #{url})" }
+    end
+  end
+
   def describe(res)
     return res.error if res.error