Skip to content

Leave sign-in and verification fields alone #848

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
senadaruc wants to merge 5 commits into
base: main
Choose a base branch
from
+473 −7
Open

Leave sign-in and verification fields alone #848

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2bf543e
Leave sign-in and verification fields alone
senadaruc Oct 2, 2026
5d040e2
Merge main into fix/skip-sign-in-fields
akramj13 Oct 4, 2026
0114633
Match sign-in ids by whole parts and catch addresses mid-typing
akramj13 Oct 4, 2026
0efd8be
Read sign-in field labels once per focus session
akramj13 Oct 4, 2026
3a9cf3b
Refresh cached sign-in labels after a second
akramj13 Oct 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,9 @@ event payloads as complete field state.

FocusSnapshotResolver finds a usable editable candidate, blocks secure/unsupported surfaces (and
Mail's compose header rows, [MailHeaderFieldDetector.swift](Cotabby/Support/Accessibility/MailHeaderFieldDetector.swift):
Tab is the way from To to Subject to body there, not an accept), bounds
Tab is the way from To to Subject to body there, not an accept; and single-line sign-in and
verification fields, [CredentialFieldDetector.swift](Cotabby/Support/Accessibility/CredentialFieldDetector.swift):
a completion there is a guess at the user's identity), bounds
text on both sides of the caret, resolves the focused process, and publishes stable domain values.
Chromium/Electron require accessibility priming, cursor hit-test recovery, and out-of-process iframe
handling. All fallbacks are revalidated and yield to a valid system-focused element.
Expand Down
10 changes: 10 additions & 0 deletions Cotabby.xcodeproj/project.pbxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@
257302D78C5AE9951C63FCEE /* SuggestionAnchorCacheTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = BE7DB9EE77511823EDA7B52E /* SuggestionAnchorCacheTests.swift */; };
25F7E6EC713F8F71DEEEAAA3 /* SystemUIFocusShadowPolicy.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2EC384A90F3D8B71584B3449 /* SystemUIFocusShadowPolicy.swift */; };
26067524E60D738791E983CD /* SOURCES.md in Resources */ = {isa = PBXBuildFile; fileRef = 054987E76CA9D1FA4F81EA8F /* SOURCES.md */; };
263CF31EDC4FAD8041831291 /* CredentialFieldDetector.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8D9E5C3F2354CC40C0B2051B /* CredentialFieldDetector.swift */; };
26EA96EB13B94A68276FA15E /* MenuBarRecoveryPolicy.swift in Sources */ = {isa = PBXBuildFile; fileRef = 1E267F3BB7FC8DEAEB5E841B /* MenuBarRecoveryPolicy.swift */; };
2740742B866BC12043B81268 /* TypefaceEvidence.swift in Sources */ = {isa = PBXBuildFile; fileRef = B616CB46A8624BC4E4FBB01A /* TypefaceEvidence.swift */; };
27A09D81E47FA601F279EF11 /* FocusCapabilityResolver.swift in Sources */ = {isa = PBXBuildFile; fileRef = 56B8D2232F271197468CBC11 /* FocusCapabilityResolver.swift */; };
Expand Down Expand Up @@ -615,6 +616,7 @@
998168DC04A6A13D7D1F3165 /* ModelDownloadManagerTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 03CA4BBA3C54F840546033E0 /* ModelDownloadManagerTests.swift */; };
9A2D50EF4911E45EEB4556D6 /* Aria2OutputParser.swift in Sources */ = {isa = PBXBuildFile; fileRef = 83457CCF1A50CE83428C363D /* Aria2OutputParser.swift */; };
9A55EDAF0F5D5127A39351C5 /* ContextBufferNavigationTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 024DDE8C1CE9BF00DE055990 /* ContextBufferNavigationTests.swift */; };
9AAD623DEBAD8AF488C37818 /* CredentialFieldDetector.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8D9E5C3F2354CC40C0B2051B /* CredentialFieldDetector.swift */; };
9AE3398FB0E4696C89550C04 /* EmojiMatcher.swift in Sources */ = {isa = PBXBuildFile; fileRef = EA8311FAC345FE431FA89855 /* EmojiMatcher.swift */; };
9B6C176547D2B6D118572E41 /* BaseCompletionPromptRenderer.swift in Sources */ = {isa = PBXBuildFile; fileRef = 1C1AE4120FA68524700C9324 /* BaseCompletionPromptRenderer.swift */; };
9B7FE4C9ED6959A6D5181EF5 /* EngineAndModelPaneView+Endpoint.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0A184538FD926947EBB88D9E /* EngineAndModelPaneView+Endpoint.swift */; };
Expand Down Expand Up @@ -1004,6 +1006,7 @@
FCD81796FE4DC55778D57686 /* ConfidenceSuppressionPolicy.swift in Sources */ = {isa = PBXBuildFile; fileRef = 122298AE151ECEEC175878BF /* ConfidenceSuppressionPolicy.swift */; };
FCEE05402A708C33F9719D7F /* OpenAICompatibleSuggestionEngineTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4CE9156494BFC0A1E12E0B6C /* OpenAICompatibleSuggestionEngineTests.swift */; };
FDA59446E91261744C6DDFDA /* TypingCadenceTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = FEA3558520A71033BBF30879 /* TypingCadenceTests.swift */; };
FDE4A159994BDD6605AFD9E9 /* CredentialFieldDetectorTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 365AE69E4E1A3DD3486D203A /* CredentialFieldDetectorTests.swift */; };
FDF71F83A24FBE17F5B63C68 /* ApplicationBundleMetadata.swift in Sources */ = {isa = PBXBuildFile; fileRef = BD1E28CF46BF59ABDC3056BF /* ApplicationBundleMetadata.swift */; };
FE0922970524121DEC4EF2D9 /* OpenAICompatibleEndpointModels.swift in Sources */ = {isa = PBXBuildFile; fileRef = 1ABCE733783332BE52E43D67 /* OpenAICompatibleEndpointModels.swift */; };
FE4F4A0778E7D2ABC9EEC155 /* EngineAndModelPaneView+Power.swift in Sources */ = {isa = PBXBuildFile; fileRef = DF5872EC7795CC1EFF6D0D04 /* EngineAndModelPaneView+Power.swift */; };
Expand Down Expand Up @@ -1160,6 +1163,7 @@
353191D1D8A1C655E1B5F562 /* CapturedInputEventTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = CapturedInputEventTests.swift; sourceTree = "<group>"; };
35AA2C8F42B510F013D86C3C /* InsertedTextAdvanceTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = InsertedTextAdvanceTests.swift; sourceTree = "<group>"; };
35C0B587D81D87ACB952C95E /* SuggestionSettingsStoreTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SuggestionSettingsStoreTests.swift; sourceTree = "<group>"; };
365AE69E4E1A3DD3486D203A /* CredentialFieldDetectorTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = CredentialFieldDetectorTests.swift; sourceTree = "<group>"; };
36652DB88C5948AA4A31524A /* ModelFileValidator.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ModelFileValidator.swift; sourceTree = "<group>"; };
3680E1B8FA712A888F509640 /* ClipboardContentDistillerTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ClipboardContentDistillerTests.swift; sourceTree = "<group>"; };
377A0BBB59988043005A138A /* FoundationModelSuggestionEngineTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = FoundationModelSuggestionEngineTests.swift; sourceTree = "<group>"; };
Expand Down Expand Up @@ -1384,6 +1388,7 @@
8BE5F414704A8264C2946A50 /* TypoGateTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TypoGateTests.swift; sourceTree = "<group>"; };
8C151BF4D39485E5CFACFECB /* ApplicationBundleMetadataTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ApplicationBundleMetadataTests.swift; sourceTree = "<group>"; };
8D881FED12A85FFC20F2C9D7 /* DisplayCoordinateConverterTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = DisplayCoordinateConverterTests.swift; sourceTree = "<group>"; };
8D9E5C3F2354CC40C0B2051B /* CredentialFieldDetector.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = CredentialFieldDetector.swift; sourceTree = "<group>"; };
8DAD5637347E83DDCF515568 /* SuggestionCoordinator+HostMarkedText.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = "SuggestionCoordinator+HostMarkedText.swift"; sourceTree = "<group>"; };
8E542E57459488F3D39A9053 /* PhrasePredictionScoringTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PhrasePredictionScoringTests.swift; sourceTree = "<group>"; };
8E89746E8CE7E9487337EE6F /* InsertionSafetyGate.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = InsertionSafetyGate.swift; sourceTree = "<group>"; };
Expand Down Expand Up @@ -2498,6 +2503,7 @@
5B5D1A7D938F633C6EE094F7 /* AXHelper.swift */,
82420F9505E69AE2ADE9F583 /* BlockBreakAlignment.swift */,
3FE7D19D22434E3E24804999 /* CalendarAccessibilityCapturePolicy.swift */,
8D9E5C3F2354CC40C0B2051B /* CredentialFieldDetector.swift */,
47F1A6BCCA20EBE7314B79D3 /* MailHeaderFieldDetector.swift */,
E286B9A912808088FDA6B4C4 /* PermissionOverlayTracker.swift */,
1EE99C84483D3A58D561C61D /* SecureFieldDetector.swift */,
Expand Down Expand Up @@ -2709,6 +2715,7 @@
6E2AA5BD865E0BCFB53DC699 /* AXHelperTests.swift */,
D681DDF8E81F868C1BC92CB4 /* BlockBreakAlignmentTests.swift */,
863B5A1DC3C4B9668F620245 /* CalendarAccessibilityCapturePolicyTests.swift */,
365AE69E4E1A3DD3486D203A /* CredentialFieldDetectorTests.swift */,
B68F9EDFE2903996F0E5524E /* MailHeaderFieldDetectorTests.swift */,
AD003D4EBE530DC0E2B87C24 /* PermissionOverlayTrackerTests.swift */,
B8EBC9F1890DD2FFC4884A5C /* SecureFieldDetectorTests.swift */,
Expand Down Expand Up @@ -4011,6 +4018,7 @@
4E7F611941736F526C3B9C1B /* CotabbyBrand.swift in Sources */,
A5D76116479357C29E2D8405 /* CotabbyDebugOptions.swift in Sources */,
6E978AD7E340B2795F120AC0 /* CotypistExportImporter.swift in Sources */,
9AAD623DEBAD8AF488C37818 /* CredentialFieldDetector.swift in Sources */,
B803D0491F5CF19735F23B65 /* CurrencyEvaluator.swift in Sources */,
81870F2D46C12CA1E6B19523 /* CurrentWordExtractor.swift in Sources */,
378EE9C111040353A6335454 /* CurrentWordSpellChecker.swift in Sources */,
Expand Down Expand Up @@ -4350,6 +4358,7 @@
085BB87581DFFA260A630E24 /* CotabbyBrand.swift in Sources */,
7A31E6395C535FF017A1EFE1 /* CotabbyDebugOptions.swift in Sources */,
5509B327A1C9E67467333B06 /* CotypistExportImporter.swift in Sources */,
263CF31EDC4FAD8041831291 /* CredentialFieldDetector.swift in Sources */,
1F39EE1D5FA0F5D32AFFB028 /* CurrencyEvaluator.swift in Sources */,
EA353CCECBFB4D297C865447 /* CurrentWordExtractor.swift in Sources */,
C56ABA04AE27A9943368035C /* CurrentWordSpellChecker.swift in Sources */,
Expand Down Expand Up @@ -4676,6 +4685,7 @@
F8D1C3FD1A1ACAE87D885D29 /* CotabbyDebugOptionsTests.swift in Sources */,
65D20F8E6309CED34A638D35 /* CotabbyTestFixtures.swift in Sources */,
DC3B4CF0634704EF2ADA7C94 /* CotypistExportImporterTests.swift in Sources */,
FDE4A159994BDD6605AFD9E9 /* CredentialFieldDetectorTests.swift in Sources */,
15BE5127E4BE29F6CBEEAA0E /* CurrencyEvaluatorTests.swift in Sources */,
99334CDC1399D03019202E85 /* CurrentWordExtractorTests.swift in Sources */,
81073963BC57B5CA9151B0EC /* CustomRulesTests.swift in Sources */,
Expand Down
103 changes: 97 additions & 6 deletions Cotabby/Services/Focus/Resolution/FocusSnapshotResolver.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,17 @@ struct FocusSnapshotResolver {
/// fields (see `FocusSessionScopedCache`).
private let secureFieldVerdictCache = FocusSessionScopedCache<Bool>()
private let terminalDetectionCache = FocusSessionScopedCache<Bool>()
/// The label and DOM-id reads behind `CredentialFieldDetector`: up to four AX round trips that
/// would otherwise repeat on every poll of every single-line field, for values that rarely
/// change while focus stays in one field. A navigation that reuses the element changes the URL,
/// title or placeholder, which starts a new focus session and so a fresh read. A field
/// relabelled in place (a reused input whose aria-label turns it into a code box) changes none
/// of those, so a reading is also refreshed after `credentialLabelRefreshInterval`.
private let credentialLabelCache = FocusSessionScopedCache<CredentialFieldLabelReading>()
/// How long a credential label reading is trusted within one focus session. One second bounds
/// how long an in-place relabel goes unnoticed, while the reads run about a dozen times less
/// often than the 80 ms active poll.
static let credentialLabelRefreshInterval: TimeInterval = 1
/// The text margin the caret's paragraph wraps to, which a field's `AXFrame` does not reveal
/// (Word's frame is the page edge, not the text margin). Up to three AX round trips, so each
/// result is cached per focus session *and* per paragraph: the margin changes between an indented
Expand Down Expand Up @@ -72,8 +83,15 @@ struct FocusSnapshotResolver {
/// answers no width query (Chromium contenteditables, Electron composers); see
/// `CaretAdvanceSampler`. One sampler follows the focused field; a new field starts a new one.
private let caretAdvanceSamples = CaretAdvanceSampleStore()
init(geometryResolver: AXTextGeometryResolver? = nil) {
/// Seconds since boot, for the credential label refresh. Injected so tests can step time.
private let uptime: () -> TimeInterval

init(
geometryResolver: AXTextGeometryResolver? = nil,
uptime: @escaping () -> TimeInterval = { ProcessInfo.processInfo.systemUptime }
) {
self.geometryResolver = geometryResolver ?? AXTextGeometryResolver()
self.uptime = uptime
}

/// Drops the cached static-text-run walk so the next capture pays a fresh one. Called through
Expand Down Expand Up @@ -336,8 +354,12 @@ struct FocusSnapshotResolver {
hostMarkedTextRange: resolvedCandidate.markedTextRange ?? chromiumCompletionRange ?? smartComposeRange
)

if let reason = Self.blockedReason(
for: resolvedCandidate, bundleIdentifier: bundleIdentifier, selection: selection, rawSelection: rawSelection
if let reason = blockedReason(
for: resolvedCandidate,
bundleIdentifier: bundleIdentifier,
selection: selection,
rawSelection: rawSelection,
focusChangeSequence: focusChangeSequence
) {
return FocusSnapshot(
applicationName: applicationName,
Expand All @@ -356,12 +378,14 @@ struct FocusSnapshotResolver {
}

/// Why a field Cotabby can read is still one it must not complete in, or nil when it may: a
/// secure field, one of Mail's header rows, or a field with text selected.
private static func blockedReason(
/// secure field, one of Mail's header rows, a sign-in or verification field, or a field with
/// text selected.
private func blockedReason(
for candidate: AXFocusCandidate,
bundleIdentifier: String,
selection: NSRange,
rawSelection: NSRange
rawSelection: NSRange,
focusChangeSequence: UInt64
) -> String? {
if candidate.isSecure {
return "Secure text input is active."
Expand All @@ -378,6 +402,21 @@ struct FocusSnapshotResolver {
return MailHeaderFieldDetector.blockedReason
}

// Email, username, phone and code boxes, single-line fields only. The labels are read at
// most once a second per field; the typed text is checked on every poll, since typing
// "alice@" is what reveals an unlabelled address box.
if CredentialFieldDetector.mightBeCredentialField(role: candidate.role) {
let labelReading = credentialLabelReading(for: candidate, focusChangeSequence: focusChangeSequence)
if CredentialFieldDetector.isCredentialField(
role: candidate.role,
labels: labelReading.labels,
domIdentifier: labelReading.domIdentifier,
text: candidate.textValue
) {
return CredentialFieldDetector.blockedReason
}
}

guard selection.length > 0 else { return nil }
if BrowserAppDetector.isChromiumBrowser(bundleIdentifier: bundleIdentifier) {
CotabbyLogger.focus.debug(
Expand Down Expand Up @@ -1554,6 +1593,58 @@ struct FocusSnapshotResolver {
descriptionLabel: AXHelper.stringValue(for: kAXDescriptionAttribute as CFString, on: element)
)
}

/// What a single-line field says about itself (title, description, placeholder, DOM id), read
/// through `credentialLabelCache`: once per focus session, refreshed after
/// `credentialLabelRefreshInterval`.
///
/// An all-empty reading from web content is returned but not cached: a web field can be read
/// before the page has filled in its name, and caching that would leave a real sign-in box
/// unrecognized until the next refresh. Such a field keeps paying the reads until it answers.
/// A native field's attributes are there as soon as it is, so its empty reading is kept.
private func credentialLabelReading(
for candidate: AXFocusCandidate,
focusChangeSequence: UInt64
) -> CredentialFieldLabelReading {
let now = uptime()
if let cached = credentialLabelCache.cachedValue(
forKey: candidate.elementIdentifier, focusChangeSequence: focusChangeSequence
), now - cached.readAt < Self.credentialLabelRefreshInterval {
return cached
}

let reading = CredentialFieldLabelReading(
readAt: now,
labels: [
AXHelper.stringValue(for: kAXTitleAttribute as CFString, on: candidate.element),
AXHelper.stringValue(for: kAXDescriptionAttribute as CFString, on: candidate.element),
AXHelper.stringValue(for: kAXPlaceholderValueAttribute as CFString, on: candidate.element)
],
// Only web content vends DOM ids; asking a native field is a wasted round trip.
domIdentifier: candidate.vendsDOMAttributes
? AXHelper.stringValue(for: "AXDOMIdentifier" as CFString, on: candidate.element)
: nil
)
if !reading.isEmpty || !candidate.vendsDOMAttributes {
credentialLabelCache.store(
reading, forKey: candidate.elementIdentifier, focusChangeSequence: focusChangeSequence
)
Comment thread
akramj13 marked this conversation as resolved.
}
return reading
}
}

/// The attributes `CredentialFieldDetector` judges a field by, cached per focus session.
private struct CredentialFieldLabelReading {
/// `uptime()` when the attributes were read, so the reading can be refreshed.
let readAt: TimeInterval
let labels: [String?]
let domIdentifier: String?

/// True when the field answered nothing usable, so a web field's reading may simply be early.
var isEmpty: Bool {
(labels + [domIdentifier]).allSatisfy { ($0 ?? "").isEmpty }
}
}

private struct FocusCandidateResolution {
Expand Down
Loading
Loading