diff --git a/README.md b/README.md
index 84cd0be..92e77e5 100644
--- a/README.md
+++ b/README.md
@@ -111,16 +111,23 @@ complete and are not a legal assessment.
sheets, fonts, images, frames, preloads, CSS `url()` and `@import`, meta
refresh); every such URL on another host than the passport's fails the
criterion. Resources added by scripts at run time are not seen.
- - With `device_features: forbidden` dpplint searches, without running any
+ - With `device_features: forbidden` dpplint reads, without running any
script, the inline scripts, event handler attributes and `javascript:`
URLs of the HTML view and the scripts it loads from its own host (at most
- 10, the first 2 MB of each) for browser interfaces that use functions of
- the reader's device (location, camera, microphone, notifications,
- contacts, sensors, network information and others), and the markup for
- frames granted such permissions, file inputs with `capture` and permission
- elements. Every finding fails the criterion. Code assembled at run time,
- obfuscated or loaded by a script, and scripts from other hosts, are not
- seen; an interface name in a comment or string counts.
+ 10, the first 2 MB of each) as JavaScript tokens and looks for uses of
+ functions of the reader's device (location, camera, microphone,
+ notifications, contacts, sensors, network information and others): calls
+ of their browser interfaces, permission requests and listeners for device
+ events; in the markup it looks for frames granted such permissions, file
+ inputs with `capture` and permission elements. Every use fails the
+ criterion. Names in comments and strings, testing whether an interface
+ exists, and hooking into it without a call, as polyfills such as zone.js
+ do, are not a use. A reference after which a use may follow that only
+ running the script would show (an interface stored under another name, a
+ device method read without a call) gives a warning if there is no use;
+ so does a script that cannot be read as JavaScript but names a device
+ interface. Code assembled at run time, obfuscated or loaded by a script,
+ and scripts from other hosts, are not seen.
- With `markup_errors: forbidden` dpplint posts the HTML view, as received
and with the media type and charset of the response, to the Nu Html
Checker that it runs itself (`NU_CHECKER_URL`; see "Deployment"). It never
diff --git a/app/services/device_uses.rb b/app/services/device_uses.rb
new file mode 100644
index 0000000..be56a32
--- /dev/null
+++ b/app/services/device_uses.rb
@@ -0,0 +1,355 @@
+# Where a piece of JavaScript uses functions of the reader's device
+# (device_features in CRITERIA-FORMAT.md of dpp-criteria, DPP-SEC-021).
+#
+# Only code counts, read with JsTokens: comments, and names inside strings,
+# template literals and regular expressions, are not code. A use is
+# - a call of a device method (METHODS, navigator.vibrate) on any object, also
+# as obj["name"](…), .call(…) or .apply(…);
+# - a call of a method of a device interface (INTERFACES: mediaDevices,
+# pushManager, navigator.contacts, .bluetooth, .usb, .serial, .hid,
+# .wakeLock), or reading a property of navigator.connection;
+# - a permission request: requestPermission of Notification,
+# DeviceMotionEvent, DeviceOrientationEvent or IdleDetector;
+# - new with Notification, NDEFReader, IdleDetector or a sensor class;
+# - a listener for devicemotion, deviceorientation, deviceorientationabsolute
+# or beforeinstallprompt, registered with addEventListener or on, or by
+# assigning the on… property;
+# - transport: ["sms"] (a one-time code from an SMS).
+#
+# Hooking into an interface is not a use. Polyfills such as zone.js test
+# whether the interface exists, pass it to a function together with the
+# names of its methods and replace those methods with wrappers; feature
+# detection only tests whether it exists. None of this calls a device method.
+#
+# A doubt is a reference after which a use may follow that cannot be seen
+# without running the script:
+# - a device method, a member of a device interface or requestPermission read
+# without being called or replaced (it may be called later under another
+# name);
+# - a device interface stored or passed on in another way than above (its
+# members may be used under another name);
+# - one of the event names passed as an argument to another function than
+# addEventListener or on (frameworks register listeners that way).
+#
+# If the text cannot be read as JavaScript (JsTokens::Error), the names of
+# the device interfaces are searched for as text (MENTIONS) and every one
+# found is a doubt.
+class DeviceUses
+ Finding = Struct.new(:kind, :label, :text)
+
+ LOCATION = "location".freeze
+ MEDIA = "camera, microphone or screen".freeze
+ NOTIFICATIONS = "notifications".freeze
+ CONTACTS = "contacts".freeze
+ DEVICES = "Bluetooth, USB, serial, HID or NFC".freeze
+ SENSORS = "motion or orientation sensors".freeze
+ CONTEXT = "context or network information".freeze
+ OTP = "one-time codes from SMS".freeze
+ APPS = "app installation".freeze
+ BEHAVIOUR = "device behaviour".freeze
+ LABELS = [LOCATION, MEDIA, NOTIFICATIONS, CONTACTS, DEVICES, SENSORS, CONTEXT, OTP, APPS, BEHAVIOUR].freeze
+
+ # Device methods: calling one on any object is a use.
+ METHODS = {
+ "getCurrentPosition" => LOCATION, "watchPosition" => LOCATION, "getUserMedia" => MEDIA, "getDisplayMedia" => MEDIA,
+ "showNotification" => NOTIFICATIONS, "getBattery" => CONTEXT, "getInstalledRelatedApps" => APPS
+ }.freeze
+ # Device methods of navigator.
+ NAVIGATOR_METHODS = { "vibrate" => BEHAVIOUR }.freeze
+ # Device interfaces: name => [label, object they are read from (nil: any),
+ # their members]. The members of geolocation that use the device are
+ # METHODS; clearWatch only ends a use.
+ INTERFACES = {
+ "geolocation" => [LOCATION, nil, %w[getCurrentPosition watchPosition clearWatch]],
+ "mediaDevices" => [MEDIA, nil, %w[getUserMedia getDisplayMedia enumerateDevices getSupportedConstraints selectAudioOutput]],
+ "pushManager" => [NOTIFICATIONS, nil, %w[subscribe getSubscription permissionState]],
+ "contacts" => [CONTACTS, "navigator", %w[select getProperties]],
+ "bluetooth" => [DEVICES, "navigator", %w[requestDevice getDevices getAvailability]],
+ "usb" => [DEVICES, "navigator", %w[requestDevice getDevices]],
+ "serial" => [DEVICES, "navigator", %w[requestPort getPorts]],
+ "hid" => [DEVICES, "navigator", %w[requestDevice getDevices]],
+ "wakeLock" => [BEHAVIOUR, "navigator", %w[request]],
+ "connection" => [CONTEXT, "navigator", %w[effectiveType downlink downlinkMax rtt saveData type]]
+ }.freeze
+ # Classes whose requestPermission asks for a device permission.
+ PERMISSION_CLASSES = {
+ "Notification" => NOTIFICATIONS, "DeviceMotionEvent" => SENSORS, "DeviceOrientationEvent" => SENSORS,
+ "IdleDetector" => CONTEXT
+ }.freeze
+ # Classes whose construction uses a device function.
+ CONSTRUCTORS = {
+ "Notification" => NOTIFICATIONS, "NDEFReader" => DEVICES, "IdleDetector" => CONTEXT
+ }.merge(%w[Accelerometer Gyroscope Magnetometer AbsoluteOrientationSensor RelativeOrientationSensor
+ LinearAccelerationSensor GravitySensor AmbientLightSensor].to_h { |c| [c, SENSORS] }).freeze
+ # Events whose listeners use a device function.
+ EVENTS = {
+ "devicemotion" => SENSORS, "deviceorientation" => SENSORS, "deviceorientationabsolute" => SENSORS,
+ "beforeinstallprompt" => APPS
+ }.freeze
+ ASSIGNMENTS = %w[= ||= &&= ??=].freeze
+ # Tokens after and before an interface that only test whether it exists.
+ TESTS_AFTER = %w[&& ? === !== == !=].freeze
+ TESTS_BEFORE = %w[! === !== == != typeof in instanceof].freeze
+ # A "[" or "(" after one of these keywords is no member access or call.
+ KEYWORDS = (JsTokens::BEFORE_EXPRESSION + %w[if while for with switch catch let const var]).freeze
+
+ # Every finding, also of MENTIONS, contains one of these names; a script
+ # without any of them is not read further.
+ NEEDED = Regexp.union((METHODS.keys + NAVIGATOR_METHODS.keys + INTERFACES.keys + CONSTRUCTORS.keys + EVENTS.keys +
+ %w[requestPermission sms PushManager OTPCredential]).uniq)
+ # The interface names searched for as text if the script cannot be read.
+ MENTIONS = [
+ [LOCATION, /\.geolocation\b|\bgetCurrentPosition\b|\bwatchPosition\b/],
+ [MEDIA, /\bgetUserMedia\b|\bgetDisplayMedia\b|\.mediaDevices\b/],
+ [NOTIFICATIONS, /\bNotification\s*\.\s*requestPermission\b|\bnew\s+Notification\s*\(|\.showNotification\s*\(|\b[Pp]ushManager\b/],
+ [CONTACTS, /\bnavigator\s*\.\s*contacts\b/],
+ [DEVICES, /\bnavigator\s*\.\s*(?:bluetooth|usb|serial|hid)\b|\bNDEFReader\b/],
+ [SENSORS,
+ /\b(?:on)?(?:devicemotion|deviceorientation(?:absolute)?)\b|\bDevice(?:Motion|Orientation)Event\s*\.\s*requestPermission\b|\bnew\s+(?:Accelerometer|Gyroscope|Magnetometer|AbsoluteOrientationSensor|RelativeOrientationSensor|LinearAccelerationSensor|GravitySensor|AmbientLightSensor)\s*\(/],
+ [CONTEXT, /\bnavigator\s*\.\s*connection\b|\bgetBattery\s*\(|\bIdleDetector\b/],
+ [OTP, /\bOTPCredential\b|\btransport\s*:\s*\[\s*["']sms["']/],
+ [APPS, /\bbeforeinstallprompt\b|\bgetInstalledRelatedApps\b/],
+ [BEHAVIOUR, /\bnavigator\s*\.\s*(?:wakeLock|vibrate)\b/]
+ ].freeze
+
+ # Findings in a piece of script, one per label: a use if there is one,
+ # otherwise a doubt; uses first. partial: the text is the start of a longer
+ # script.
+ def self.call(code, partial: false)
+ code = code.to_s
+ code = code.dup.force_encoding(Encoding::UTF_8).scrub unless code.encoding == Encoding::UTF_8 && code.valid_encoding?
+ return [] unless code.match?(NEEDED)
+
+ new(JsTokens.call(code, partial: partial)).findings
+ rescue JsTokens::Error
+ MENTIONS.filter_map do |label, pattern|
+ match = code.match(pattern)
+ Finding.new(:doubt, label, "#{match[0].gsub(/\s+/, ' ')} mentioned, but the script cannot be read") if match
+ end
+ end
+
+ def initialize(tokens)
+ @t = tokens
+ @found = {}
+ @open = [] # for each token: index of the innermost bracket around it
+ @opening = {} # index of a closing bracket => index of its opening one
+ @closing = {} # and the other way round
+ brackets
+ end
+
+ def findings
+ @t.each_with_index do |tok, i|
+ case tok.type
+ when :name then name_at(i, tok.value)
+ when :string then string_at(i, tok.value)
+ end
+ end
+ @found.values.sort_by { |f| [f.kind == :use ? 0 : 1, LABELS.index(f.label)] }
+ end
+
+ private
+
+ def brackets
+ stack = []
+ @t.each_with_index do |tok, k|
+ @open << stack.last
+ next unless tok.type == :punct
+
+ case tok.value
+ when "(", "[", "{" then stack.push(k)
+ when ")", "]", "}"
+ o = stack.pop
+ (@opening[k] = o) && (@closing[o] = k) if o
+ end
+ end
+ end
+
+ def name_at(i, v)
+ member(i, v, i) if member?(i)
+ construction(i) if v == "new"
+ listener(i, v) if punct?(i + 1, "(")
+ handler_property(i, v) if v.start_with?("on") && EVENTS.key?(v[2..])
+ sms(i) if v == "transport"
+ end
+
+ # A string is a member name in obj["name"], or an event name passed to a
+ # function.
+ def string_at(i, v)
+ if punct?(i - 1, "[") && punct?(i + 1, "]") && expression_end?(i - 2)
+ member(i, v, i + 1)
+ elsif EVENTS.key?(v) && (p = call_parenthesis(i)) && !(p == i - 1 && listener?(p - 1))
+ doubt(EVENTS[v], "\"#{v}\" passed to a function other than addEventListener")
+ end
+ end
+
+ # The member v of an object, read at index i and ending at index e ("]" for
+ # obj["v"]); the object is the token before "." or "[".
+ def member(i, v, e)
+ object = value(i - 2)
+ if (label = METHODS[v]) then method_use(e, label, ".#{v}")
+ elsif (label = NAVIGATOR_METHODS[v]) && object == "navigator" then method_use(e, label, "navigator.#{v}")
+ elsif v == "requestPermission" && (label = PERMISSION_CLASSES[object])
+ method_use(e, label, "#{object}.requestPermission")
+ end
+ interface(i, e, v, object)
+ end
+
+ def method_use(e, label, text)
+ if called?(e) then use(label, "#{text}()")
+ elsif !assigned?(e) then doubt(label, "#{text} read without a call")
+ end
+ end
+
+ def interface(i, e, v, object)
+ label, from, members = INTERFACES[v]
+ return unless label && (from.nil? || object == from)
+
+ text = "#{from ? "#{from}." : '.'}#{v}"
+ if (m = member_after(e))
+ interface_member(label, v, text, *m)
+ elsif !assigned?(e) && !tested?(i, e) && !hooked?(i, members)
+ doubt(label, "#{text} stored or passed on")
+ end
+ end
+
+ def interface_member(label, v, text, name, e)
+ return if v == "geolocation" || assigned?(e)
+
+ if v == "connection" then use(label, "#{text}.#{name}")
+ elsif called?(e) then use(label, "#{text}.#{name}()")
+ else doubt(label, "#{text}.#{name} read without a call")
+ end
+ end
+
+ # The member read right after index e: [name, index of its last token].
+ def member_after(e)
+ if (punct?(e + 1, ".") || punct?(e + 1, "?.")) && @t[e + 2]&.type == :name
+ [@t[e + 2].value, e + 2]
+ elsif punct?(e + 1, "[") && @t[e + 2]&.type == :string && punct?(e + 3, "]")
+ [@t[e + 2].value, e + 3]
+ elsif punct?(e + 1, "?.") && punct?(e + 2, "[") && @t[e + 3]&.type == :string && punct?(e + 4, "]")
+ [@t[e + 3].value, e + 4]
+ end
+ end
+
+ # Whether the interface read at index i (ending at e) is only tested:
+ # a && …, a ? …, a === …, !a, typeof a, "x" in a, or in the head of if or
+ # while.
+ def tested?(i, e)
+ return true if TESTS_AFTER.include?(punct_value(e + 1))
+
+ s = chain_start(i)
+ return true if TESTS_BEFORE.include?(punct_value(s - 1) || value(s - 1))
+
+ (p = @open[i]) && punct?(p, "(") && %w[if while].include?(value(p - 1))
+ end
+
+ # Whether the interface read at index i is passed to a function together
+ # with the name of one of its members, as hooks do:
+ # wrap(navigator.geolocation, ["getCurrentPosition", "watchPosition"]).
+ def hooked?(i, members)
+ p = call_parenthesis(chain_start(i))
+ return false unless p
+
+ ((p + 1)...(@closing[p] || @t.size)).any? { |k| @t[k].type == :string && members.include?(@t[k].value) }
+ end
+
+ # The first token of the member chain whose last member is at index i
+ # (a.b["c"](x).d: a).
+ def chain_start(i)
+ k = i
+ loop do
+ intro = k - 1
+ return k unless punct?(intro, ".") || punct?(intro, "?.") || (punct?(intro, "[") && @t[k].type == :string)
+
+ o = intro - 1
+ if value(o) then k = o
+ elsif punct?(o, "]") && (b = @opening[o]) && @t[b + 1]&.type == :string then k = b + 1
+ elsif punct?(o, ")") && (b = @opening[o]) && value(b - 1) then k = b - 1
+ else return [o, 0].max
+ end
+ end
+ end
+
+ # If the token at index i is an argument of a call: the index of the "(".
+ # A call may follow a function expression: function (a, b) {…}(x, y).
+ def call_parenthesis(i)
+ p = @open[i]
+ return unless (punct?(i - 1, "(") || punct?(i - 1, ",")) && p && punct?(p, "(")
+
+ p if expression_end?(p - 1) || punct?(p - 1, "}")
+ end
+
+ # new C(…), also new window.C(…).
+ def construction(i)
+ j = i + 1
+ j += 2 while value(j) && punct?(j + 1, ".")
+ c = value(j)
+ use(CONSTRUCTORS[c], "new #{c}()") if CONSTRUCTORS.key?(c)
+ end
+
+ # addEventListener("devicemotion", …) or .on("devicemotion", …).
+ def listener(i, v)
+ return unless listener?(i)
+
+ event = @t[i + 2]
+ use(EVENTS[event.value], "#{v}(\"#{event.value}\")") if event&.type == :string && EVENTS.key?(event.value)
+ end
+
+ def listener?(i) = value(i) == "addEventListener" || (value(i) == "on" && member?(i))
+
+ # window.ondevicemotion = …, but not = null.
+ def handler_property(i, v)
+ return unless assigned?(i) && value(i + 2) != "null"
+
+ use(EVENTS[v[2..]], "#{v} =")
+ end
+
+ def sms(i)
+ return unless punct?(i + 1, ":") && punct?(i + 2, "[") && @t[i + 3]&.type == :string && @t[i + 3].value == "sms"
+
+ use(OTP, "transport: [\"sms\"]")
+ end
+
+ # The member ending at index e is called: m(…), m?.(…), m.call(…),
+ # m.apply(…).
+ def called?(e)
+ return true if punct?(e + 1, "(") || (punct?(e + 1, "?.") && punct?(e + 2, "("))
+
+ (punct?(e + 1, ".") || punct?(e + 1, "?.")) && %w[call apply].include?(value(e + 2)) && punct?(e + 3, "(")
+ end
+
+ # The member ending at index e is replaced: m = …
+ def assigned?(e) = ASSIGNMENTS.include?(punct_value(e + 1))
+
+ def member?(i) = punct?(i - 1, ".") || punct?(i - 1, "?.")
+
+ # Whether the token at index i ends an expression, so that "(" or "["
+ # after it is a call or a member access.
+ def expression_end?(i)
+ return false if i.negative? || @t[i].nil?
+
+ tok = @t[i]
+ case tok.type
+ when :name then !KEYWORDS.include?(tok.value)
+ when :punct then %w[) \]].include?(tok.value)
+ else %i[string template regexp private].include?(tok.type)
+ end
+ end
+
+ def punct?(i, v) = punct_value(i) == v
+
+ def punct_value(i) = i >= 0 && @t[i]&.type == :punct ? @t[i].value : nil
+
+ def value(i) = i >= 0 && @t[i]&.type == :name ? @t[i].value : nil
+
+ def use(label, text)
+ return if @found[label]&.kind == :use
+
+ @found[label] = Finding.new(:use, label, text)
+ end
+
+ def doubt(label, text)
+ @found[label] ||= Finding.new(:doubt, label, text)
+ end
+end
diff --git a/app/services/html_check.rb b/app/services/html_check.rb
index 8c07713..d6ec8be 100644
--- a/app/services/html_check.rb
+++ b/app/services/html_check.rb
@@ -13,13 +13,15 @@
# follow are not counted; resources added by scripts at run time cannot be
# seen in the document.
#
-# device_features: forbidden looks, without running any script, for browser
-# interfaces that use functions of the reader's device (DEVICE_INTERFACES) in
-# the inline scripts, event handler attributes and javascript: URLs of the
-# document and in the scripts it loads from its own host (at most MAX_SCRIPTS,
-# the first MAX_SCRIPT_BYTES of each; a script that cannot be retrieved is not
-# checked), and in the markup for frames granted device permissions
-# (DEVICE_PERMISSIONS), file inputs with capture and permission elements.
+# device_features: forbidden looks, without running any script, for uses of
+# functions of the reader's device (DeviceUses: calls, permission requests,
+# listeners, not mere references) in the inline scripts, event handler
+# attributes and javascript: URLs of the document and in the scripts it loads
+# from its own host (at most MAX_SCRIPTS, the first MAX_SCRIPT_BYTES of each;
+# a script that cannot be retrieved is not checked), and in the markup for
+# frames granted device permissions (DEVICE_PERMISSIONS), file inputs with
+# capture and permission elements. A use is a violation; a doubt of DeviceUses
+# without any use is a warning.
#
# markup_errors: forbidden sends the document as received to the Nu Html
# Checker that dpplint runs itself (NuChecker) and reports every message of
@@ -29,19 +31,6 @@
class HtmlCheck
Outcome = Struct.new(:skipped, :code, :messages, keyword_init: true)
- DEVICE_INTERFACES = [
- ["location", /\.geolocation\b|\bgetCurrentPosition\b|\bwatchPosition\b/],
- ["camera, microphone or screen", /\bgetUserMedia\b|\bgetDisplayMedia\b|\.mediaDevices\b/],
- ["notifications", /\bNotification\s*\.\s*requestPermission\b|\bnew\s+Notification\s*\(|\.showNotification\s*\(|\b[Pp]ushManager\b/],
- ["contacts", /\bnavigator\s*\.\s*contacts\b/],
- ["Bluetooth, USB, serial, HID or NFC", /\bnavigator\s*\.\s*(?:bluetooth|usb|serial|hid)\b|\bNDEFReader\b/],
- ["motion or orientation sensors",
- /\b(?:on)?(?:devicemotion|deviceorientation(?:absolute)?)\b|\bDevice(?:Motion|Orientation)Event\s*\.\s*requestPermission\b|\bnew\s+(?:Accelerometer|Gyroscope|Magnetometer|AbsoluteOrientationSensor|RelativeOrientationSensor|LinearAccelerationSensor|GravitySensor|AmbientLightSensor)\s*\(/],
- ["context or network information", /\bnavigator\s*\.\s*connection\b|\bgetBattery\s*\(|\bIdleDetector\b/],
- ["one-time codes from SMS", /\bOTPCredential\b|\btransport\s*:\s*\[\s*["']sms["']/],
- ["app installation", /\bbeforeinstallprompt\b|\bgetInstalledRelatedApps\b/],
- ["device behaviour", /\bnavigator\s*\.\s*(?:wakeLock|vibrate)\b/]
- ].freeze
DEVICE_PERMISSIONS = %w[camera microphone geolocation display-capture bluetooth usb serial hid accelerometer gyroscope
magnetometer ambient-light-sensor idle-detection screen-wake-lock].freeze
JS_TYPES = %w[text/javascript application/javascript application/ecmascript text/ecmascript module
@@ -75,19 +64,16 @@ def outcome
messages = []
messages << external_message(res) if @check["external_resources"] == "forbidden"
- messages << device_message(res) if @check["device_features"] == "forbidden"
+ messages.concat(device_messages(res)) if @check["device_features"] == "forbidden"
messages << markup_message(res) if @check["markup_errors"] == "forbidden"
Outcome.new(messages: messages.compact)
rescue NuChecker::Unavailable => e
Outcome.new(skipped: e.message, code: "not_evaluated")
end
- # Device interfaces found in a piece of script: [[label, matched text], …].
+ # Uses of device functions in a piece of script: [[label, what], …].
def self.device_uses(code)
- DEVICE_INTERFACES.filter_map do |label, pattern|
- match = code.to_s.match(pattern)
- [label, match[0].gsub(/\s+/, " ")] if match
- end
+ DeviceUses.call(code).select { |f| f.kind == :use }.map { |f| [f.label, f.text] }
end
def self.javascript_type?(type)
@@ -176,16 +162,27 @@ def external_message(res)
message: "the HTML view loads #{external.size} resource(s) from other hosts than #{own}: #{listed.join(', ')}#{more}" }
end
- def device_message(res)
+ # A violation for the uses found, and a warning for the doubts if there is
+ # no use.
+ def device_messages(res)
doc = Nokogiri::HTML5(res.body.to_s)
- findings = markup_findings(doc) + inline_findings(doc) + script_findings(doc, res.url.to_s)
- return nil if findings.empty?
+ found = markup_findings(doc) + inline_findings(doc) + script_findings(doc, res.url.to_s)
+ uses = found.select { |kind, _| kind == :use }.map(&:last)
+ return [device_message("violation", "the HTML view uses functions of the reader's device", uses)] if uses.any?
+
+ doubts = found.map(&:last)
+ return [] if doubts.empty?
+
+ [device_message("warning", "no use of functions of the reader's device was found, but the HTML view may use them", doubts)]
+ end
+ def device_message(severity, text, findings)
listed = findings.first(MAX_FINDINGS)
more = findings.size > MAX_FINDINGS ? " and #{findings.size - MAX_FINDINGS} more" : ""
- { severity: "violation", message: "the HTML view uses functions of the reader's device: #{listed.join('; ')}#{more}" }
+ { severity: severity, message: "#{text}: #{listed.join('; ')}#{more}" }
end
+ # Findings in the markup: [[:use, text], …].
def markup_findings(doc)
found = []
doc.css("iframe[allow]").each do |e|
@@ -194,7 +191,7 @@ def markup_findings(doc)
end
found << "file input with capture" if doc.css("input[capture]").any?
%w[permission geolocation].each { |name| found << "#{name} element" if doc.css(name).any? }
- found
+ found.map { |f| [:use, f] }
end
def inline_findings(doc)
@@ -202,7 +199,7 @@ def inline_findings(doc)
doc.css("script").each_with_index do |e, i|
next if e["src"] || !self.class.javascript_type?(e["type"])
- self.class.device_uses(e.content).each { |label, use| found << "#{label} (#{use} in inline script #{i + 1})" }
+ found.concat(script_uses(e.content, "inline script #{i + 1}"))
end
doc.xpath("//@*").each do |a|
name = a.name.downcase
@@ -211,7 +208,7 @@ def inline_findings(doc)
end
next unless code
- self.class.device_uses(code).each { |label, use| found << "#{label} (#{use} in attribute #{name} of #{a.parent.name})" }
+ found.concat(script_uses(code, "attribute #{name} of #{a.parent.name}"))
end
found
end
@@ -227,10 +224,17 @@ def script_findings(doc, page_url)
res = @resolver.get(url, accept: "*/*")
next [] unless res.success?
- self.class.device_uses(res.body.to_s.byteslice(0, MAX_SCRIPT_BYTES).to_s.dup.force_encoding(Encoding::UTF_8).scrub).map { |label, use| "#{label} (#{use} in #{url})" }
+ body = res.body.to_s
+ code = body.byteslice(0, MAX_SCRIPT_BYTES).to_s.dup.force_encoding(Encoding::UTF_8).scrub
+ script_uses(code, url, partial: body.bytesize > MAX_SCRIPT_BYTES)
end
end
+ # [[:use or :doubt, "label (what in place)"], …]
+ def script_uses(code, place, partial: false)
+ DeviceUses.call(code, partial: partial).map { |f| [f.kind, "#{f.label} (#{f.text} in #{place})"] }
+ end
+
# Errors of the Nu Html Checker, with their line: "line 12: message".
def markup_message(res)
checker = @nu_checker || NuChecker.new
diff --git a/app/services/js_tokens.rb b/app/services/js_tokens.rb
new file mode 100644
index 0000000..ee397a1
--- /dev/null
+++ b/app/services/js_tokens.rb
@@ -0,0 +1,217 @@
+require "strscan"
+
+# Splits JavaScript source text into tokens, as far as needed to tell code
+# from comments, string literals, template literals and regular expression
+# literals. It does not parse: whether a "/" starts a regular expression is
+# decided from the token before it, as JavaScript tokenizers commonly do
+# (after ")" only if the parentheses close the head of if, while, for or
+# with).
+#
+# Tokens are Token(type, value) with type :name (identifiers and keywords),
+# :private (#name), :number, :string (value: the characters between the
+# quotes, escapes as written), :template (one literal part of a template
+# literal), :regexp and :punct. Comments are dropped.
+#
+# Error is raised for text that cannot be read as JavaScript this way, for
+# example a string literal that runs into a line break or brackets that do
+# not match; the caller then must not rely on the tokens. With partial: true
+# (a script cut after a number of bytes) the end of the text may cut a token
+# and leave brackets open.
+class JsTokens
+ class Error < StandardError; end
+
+ Token = Struct.new(:type, :value)
+
+ PUNCTUATORS = %w[>>>= ... === !== **= <<= >>= >>> &&= ||= ??= => == != <= >= && || ?? ?. ++ -- += -= *= %= &= |= ^=
+ << >> ** { } ( ) [ ] ; , < > + - * % & | ^ ! ~ ? : = . @].freeze
+ PUNCTUATOR = Regexp.union(PUNCTUATORS.sort_by { |p| -p.length })
+ ID_START = '(?:[\p{L}\p{Nl}$_]|\\\\u\h{4}|\\\\u\{\h+\})'.freeze
+ ID_PART = '(?:[\p{L}\p{Nl}\p{Mn}\p{Mc}\p{Nd}\p{Pc}$\u200C\u200D]|\\\\u\h{4}|\\\\u\{\h+\})'.freeze
+ NAME = /#{ID_START}#{ID_PART}*/
+ PRIVATE = /\##{ID_START}#{ID_PART}*/
+ NUMBER = /0[xX][\h_]+n?|0[oO][0-7_]+n?|0[bB][01_]+n?|(?:\d[\d_]*(?:\.[\d_]*)?|\.\d[\d_]*)(?:[eE][+-]?\d[\d_]*)?n?/
+ SPACE = /[\p{Zs}\t\v\f\uFEFF]+/
+ LINE_BREAK = /\r\n|[\n\r\u2028\u2029]/
+ LINE_COMMENT = %r{(?://|[^\n\r\u2028\u2029]*/
+ BLOCK_COMMENT = %r{/\*.*?\*/}m
+ STRING = { "'" => /'((?>[^'\\\n\r]+|\\(?:\r\n|[\s\S]))*)'/, '"' => /"((?>[^"\\\n\r]+|\\(?:\r\n|[\s\S]))*)"/ }.freeze
+ TEMPLATE_PART = /(?>[^`\\$]+|\\[\s\S]|\$(?!\{))*/
+ REGEXP = %r{/(?>[^/\\\[\n\r\u2028\u2029]+|\\[^\n\r\u2028\u2029]|\[(?>[^\]\\\n\r\u2028\u2029]+|\\[^\n\r\u2028\u2029])*\])+/#{ID_PART}*}
+ # After these keywords an expression starts, so "/" begins a regular expression.
+ BEFORE_EXPRESSION = %w[return typeof instanceof in of new delete void throw case do else yield await extends].freeze
+ CONTROL = %w[if while for with].freeze
+
+ def self.call(source, partial: false) = new(source, partial: partial).tokens
+
+ def initialize(source, partial: false)
+ @s = StringScanner.new(source.to_s)
+ @partial = partial
+ @tokens = []
+ @braces = [] # :block or :template for each open "{" or "${"
+ @parens = [] # for each open "(": whether it opens the head of if/while/for/with
+ @after_control = false
+ @line_start = true
+ end
+
+ def tokens
+ read until @s.eos?
+ raise Error, "brackets not closed at the end" unless @partial || (@braces.empty? && @parens.empty?)
+
+ @tokens
+ end
+
+ private
+
+ # One token, comment or run of white space, chosen by its first byte.
+ def read
+ case @s.string.getbyte(@s.pos)
+ when 0x0A, 0x0D then line_break
+ when 0x09, 0x0B, 0x0C, 0x20 then @s.skip(SPACE)
+ when 0x2F then slash
+ when 0x22, 0x27 then string(@s.peek(1))
+ when 0x60 then @s.getch && template
+ when 0x30..0x39 then add(:number, @s.scan(NUMBER))
+ when 0x2E then @s.check(/\.\d/) ? add(:number, @s.scan(NUMBER)) : punct(@s.scan(PUNCTUATOR))
+ when 0x3F then @s.check(/\?\.\d/) ? @s.getch && add(:punct, "?") : punct(@s.scan(PUNCTUATOR))
+ when 0x3C then @s.skip(LINE_COMMENT) || punct(@s.scan(PUNCTUATOR))
+ when 0x2D then (@line_start && @s.skip(CLOSE_COMMENT)) || punct(@s.scan(PUNCTUATOR))
+ when 0x23 then hash
+ when 0x41..0x5A, 0x61..0x7A, 0x24, 0x5F, 0x5C then name
+ when 0x80..0xFF then other
+ else (p = @s.scan(PUNCTUATOR)) ? punct(p) : unexpected
+ end
+ end
+
+ def line_break
+ @s.skip(LINE_BREAK)
+ @line_start = true
+ end
+
+ def slash
+ return if @s.skip(BLOCK_COMMENT) || @s.skip(LINE_COMMENT)
+ return unterminated("comment", anywhere: true) if @s.check(%r{/\*})
+
+ regexp_allowed? ? regexp : add(:punct, @s.scan(%r{/=?}))
+ end
+
+ def hash
+ return if @s.pos.zero? && @s.skip(/#![^\n\r\u2028\u2029]*/)
+
+ (n = @s.scan(PRIVATE)) ? add(:private, n) : unexpected
+ end
+
+ def name
+ (n = @s.scan(NAME)) ? add(:name, n) : unexpected
+ end
+
+ # A character outside ASCII: a line break, white space or part of a name.
+ def other
+ return line_break if @s.check(LINE_BREAK)
+ return if @s.skip(SPACE)
+
+ name
+ end
+
+ # In a cut text, the cut may also split an escape in a name or a character
+ # (replaced by U+FFFD).
+ def unexpected
+ return @s.terminate if @partial && @s.rest.match?(/\A(?:[\\#]|\\u\h{0,3}|\\u\{\h*|\uFFFD)\z/)
+
+ raise Error, "unexpected character #{@s.check(/./m).inspect} at #{@s.pos}"
+ end
+
+ def add(type, value)
+ @tokens << Token.new(type, value)
+ @after_control = false
+ @line_start = false
+ end
+
+ def punct(p)
+ case p
+ when "(" then @parens << CONTROL.include?(last_name)
+ when ")" then return close_paren
+ when "{" then @braces << :block
+ when "}" then return close_brace
+ end
+ add(:punct, p)
+ end
+
+ def close_paren
+ raise Error, "unmatched ) at #{@s.pos}" if @parens.empty?
+
+ control = @parens.pop
+ add(:punct, ")")
+ @after_control = control
+ end
+
+ def close_brace
+ raise Error, "unmatched } at #{@s.pos}" if @braces.empty?
+
+ if @braces.pop == :template
+ template
+ else
+ add(:punct, "}")
+ end
+ end
+
+ def last_name
+ t = @tokens.last
+ t.value if t&.type == :name
+ end
+
+ def regexp_allowed?
+ t = @tokens.last
+ return true if t.nil?
+
+ case t.type
+ when :name then BEFORE_EXPRESSION.include?(t.value)
+ when :punct
+ case t.value
+ when ")" then @after_control
+ when "]", "++", "--" then false
+ else true
+ end
+ when :template then t.equal?(@substitution)
+ else false
+ end
+ end
+
+ def regexp
+ r = @s.scan(REGEXP)
+ return add(:regexp, r) if r
+
+ unterminated("regular expression")
+ end
+
+ def string(quote)
+ return add(:string, @s[1]) if @s.scan(STRING[quote])
+
+ unterminated("string")
+ end
+
+ # The literal part of a template literal up to "`" or "${"; the part after
+ # a substitution starts after its closing "}".
+ def template
+ part = @s.scan(TEMPLATE_PART)
+ if @s.scan(/`/)
+ add(:template, part)
+ elsif @s.scan(/\$\{/)
+ @braces << :template
+ add(:template, "#{part}${")
+ @substitution = @tokens.last
+ else
+ unterminated("template literal")
+ end
+ end
+
+ # A literal or comment without its end: at the end of a cut script that is
+ # expected, otherwise the text is not read as JavaScript. A string or
+ # regular expression literal must not contain a line break, so one that
+ # stops at a line break is not cut but misread.
+ def unterminated(what, anywhere: false)
+ raise Error, "#{what} not closed at #{@s.pos}" unless @partial && (anywhere || !@s.rest.match?(LINE_BREAK))
+
+ @s.terminate
+ end
+end
diff --git a/kubernetes/deployment.yaml b/kubernetes/deployment.yaml
index d8b2675..4d05188 100644
--- a/kubernetes/deployment.yaml
+++ b/kubernetes/deployment.yaml
@@ -26,7 +26,7 @@ spec:
app: dpplint
containers:
- name: dpplint
- image: oydeu/dpplint@sha256:aa449cae9835fc0c03c29a4956c3de8d5b212eef29b521eecdcdf59b9b336e8f
+ image: oydeu/dpplint@sha256:1400a69cabdfb729eff64ca1cf0ca9139c3329ddc1843466908793b3c2581bc4
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3000
diff --git a/test/fixtures/files/html/DPP-SEC-021/invalid/polyfill-and-location-call.html b/test/fixtures/files/html/DPP-SEC-021/invalid/polyfill-and-location-call.html
new file mode 100644
index 0000000..3c42b5f
--- /dev/null
+++ b/test/fixtures/files/html/DPP-SEC-021/invalid/polyfill-and-location-call.html
@@ -0,0 +1,12 @@
+
+
+
+
+Passport
+
+
+
+
+