diff --git a/CHANGELOG.md b/CHANGELOG.md index f4dff427..cb03678d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased - The family page now keeps a child's results to act on closed, as the child's own page does, and what it shows of a child no longer depends on which result is closed or where it lies. While a child has a result still closed, or the page cannot read the child's results, these close for that child, one box each: the painting's grandparents (every stretch drawn from its parent alone), the chromosome map image, all de novo and compound-het rows, the child's medicines and HLA cells in the side by side, and the carrier screen of a couple the child is part of. Each "Show the result" opens one result; the places come back once the last one is opened, and a reload closes them again. While the results cannot be read, each box opens only its own place. On a child's page, a copy-number dot and a card's fact that name a closed result's gene now close with it too, on a phone in the card's line as well. `results.json` adds `veil.spans`, `veil.facts` and `veil.widget`; nothing is renamed or removed. The veil is still a browser veil: `results.json` and `family.json` hold every row. +- A child's page no longer tells which result to act on is closed. A closed finding's card, its place in the page and the address carry a handle (`#f/closed-2`) instead of its id, which names the gene and variant; the address never holds a closable finding's id, closed (`#f/closed-`) or opened (`#f/opened-`), so no history entry keeps it, and any other `#f/` on such a page, a closed result's id or nothing, lands on the first closed finding, so a guessed id tells nothing. The home page links a child's closed result to the child's report, not to the finding's own page. A closed structural variant now closes the rows and findings that name any of its genes, not only the first. The cards whose rows are the same genes for everyone (known mobile element insertions, repeat expansions, VNtyper, Lp(a), CHIP, the copy numbers, CPIC, CYP2D6, HLA, Locityper, the traits panel, the blood groups) close whole while any result is closed, behind one box, and so do their own findings (a medicine gene, a CHIP call) and the lead's words on them, the ClinVar card's lines for the known deep intronic sites, the carrier list's two-hit line and the callability card's rows, all together; a row or line closes with a result only when it names its gene as a gene symbol (never a prose word, an ontology id or a unit such as `5 MB`), and the polygenic score card is never tagged; a card's note on its closed rows no longer counts them. `results.json` gains `veil.panels`, the tag `*panel*` in `veil.lines`, `veil.rows`, `veil.findings` and `veil.lead`, a `genes` list on each structural-variant finding, and one `veil.spans` row per gene; nothing is renamed or removed. - Step 40, the 23andMe and AncestryDNA raw-data files, now runs on the Rust `og` binary: `og consumer export`, `og consumer build-sites` and `og consumer check` replace `bin/export_consumer.py`. On a synthetic gVCF of 552 MB (32 million records) with 617,044 sites it takes about 2 s and 43 MB in a Linux container on 4 cores, where the Python took 30 s and 660 MB; the files hold the same rows. `setup.sh --consumer-sites` reads dbSNP on 4 cores without the grep pass. The zips are now the same bytes for the same day (`--date`, else `SOURCE_DATE_EPOCH`, else today). Fixes: a call of three or more alleles stops the export with the site, instead of a file the strict reader refuses; two dbSNP records of one rsID at one position have their ALT alleles joined before the strand test; a bad site row or an unreadable zip is reported with the file and line instead of a traceback; a good export removes the `.txt` files a failed one left. The musl builds of `og` use the mimalloc allocator, which keeps a long multi-threaded read at its live memory. Step 40 needs `og`: `scripts/setup.sh ` builds it, and `scripts/setup.sh --og` builds only it. - Steps 21, 32 and 9b (and the CYRIUS, PYPGX and STRANGER_RFC1 modules) now run the CYP2D6 depth check and the RFC1 motif count with `og`, and `bin/cyp2d6_depth_check.py` and `bin/rfc1_motif.py` are gone. A run that reaches these steps needs the binary: run `./scripts/setup.sh --og` once after setup and after each pull; without it the step stops with `og is not built`. The output files, their names and formats do not change; in CI, every e2e case that runs these steps in bash checks `og` against the deleted Python, byte for byte. diff --git a/assets/live/finding.js b/assets/live/finding.js index 5f6fc3a3..fbcdee3c 100644 --- a/assets/live/finding.js +++ b/assets/live/finding.js @@ -83,9 +83,18 @@ var tags = (r.veil && r.veil.findings) || {}; function tagOf(x) { var t = tags[x.id]; return t && t !== x.id ? t : null; } // Opening this finding opens it and the result it names, nothing else: another result keeps its own veil. + // A finding of a fixed-panel card (tag PANEL, bin/live_results.py FIXED_PANELS) names no result: it stays closed + // while any result to act on is, whichever, as on the person page. + var PANEL = '*panel*'; var opened = {}; - if (draw.open) { opened[f.id] = true; if (tagOf(f)) { opened[tagOf(f)] = true; } } - function shut(x) { return child && (x.level === 'act' ? !opened[x.id] : !!tagOf(x) && !opened[tagOf(x)]); } + if (draw.open) { opened[f.id] = true; if (tagOf(f) && tagOf(f) !== PANEL) { opened[tagOf(f)] = true; } } + function actShut() { return (r.findings || []).some(function (x) { return x.level === 'act' && !opened[x.id]; }); } + function shut(x) { + if (!child) { return false; } + if (x.level === 'act') { return !opened[x.id]; } + var t = tagOf(x); + return t === PANEL ? !opened[x.id] && actShut() : !!t && !opened[t]; + } // Closed: no title, gene or variant anywhere on the page (its title, crumb, cover, rail) until opened. var closed = shut(f); var title = !closed ? f.title : f.level === 'act' ? 'A result to act on' : 'A finding about a result to act on'; diff --git a/assets/live/home.js b/assets/live/home.js index 6f25e7bb..2afb54fa 100644 --- a/assets/live/home.js +++ b/assets/live/home.js @@ -192,7 +192,8 @@ }); items.sort(function (a, b) { return order[a.f.level] - order[b.f.level]; }); items.slice(0, 6).forEach(function (x) { - var a = el('a', { href: x.r.url + 'finding/' + encodeURIComponent(x.f.id) + '/' }); + // A child's closed result links to the run's page, never to its own page: its id names its gene and variant. + var a = el('a', { href: x.r.url + (closedRuns[x.r.name] && x.f.level === 'act' ? '' : 'finding/' + encodeURIComponent(x.f.id) + '/') }); var who = el('span', { 'class': 'who' }); var pc = el('span', { 'class': 'pc' }, x.r.name); if (colourOf[x.r.name]) { pc.style.setProperty('--pcol', colourOf[x.r.name]); } diff --git a/assets/live/map.js b/assets/live/map.js index f19b2d39..860382eb 100644 --- a/assets/live/map.js +++ b/assets/live/map.js @@ -106,7 +106,8 @@ // --- the figure --------------------------------------------------------------------------------------- // create({url, veil(id, gated) -> 'open'|'closed'|'opened', now(id) -> shown at this moment, open(id), // close(id), known(id) -> a finding of the page's results, closed() -> the ids of the results to act on the - // page keeps closed at this moment, stepDone(step) -> ended at this moment, fallback() -> rows}): one figure + // page keeps closed at this moment, stepDone(step) -> ended at this moment, href(id) -> the address of a finding + // the page opened (never its id on a child's page), fallback() -> rows}): one figure // per page. The page calls refresh(key) on each of its redraws; a new key // fetches the document again and keeps the camera. function create(opts) { @@ -910,7 +911,7 @@ inSection = inSection.filter(function (id) { return stateOf(id) !== 'closed' && opts.title && opts.title(id); }); inSection.forEach(function (id) { var c = el('div', { 'class': 'gm-chip opened', 'data-opened': 'section' }); - add(c, add(el('span', {}, 'Opened in its section: '), el('a', { href: '#f/' + id }, opts.title(id)))); + add(c, add(el('span', {}, 'Opened in its section: '), el('a', { href: '#f/' + (opts.href ? opts.href(id) : id) }, opts.title(id)))); add(c, button('Close it again', function () { confirming = null; opts.close(id); keyTo(); })); add(chips, c); }); diff --git a/assets/live/person.js b/assets/live/person.js index b4363b22..5955a95c 100644 --- a/assets/live/person.js +++ b/assets/live/person.js @@ -325,7 +325,14 @@ // A row outside its result's card (a ClinVar row, a card's line, a viewer link, a chart's mark, a search hit) // carries the id of the result to act on it names (results.veil, bin/live_results.py veil()): it stays closed // while that result is, and "Show the result" opens it with it. A tag naming no finding of these results stays closed. - function tagClosed(id) { var f = findingById(id); return closedPage() && (!f || veiled(f)); } + // A line of a fixed panel (PANEL, bin/live_results.py FIXED_LINES: a known deep intronic site) names no result: + // it stays closed while any result to act on is, whichever, so what is missing never tells the gene. + var PANEL = '*panel*'; + function tagClosed(id) { + if (id === PANEL) { return closedPage() && closedActs().length > 0; } + var f = findingById(id); + return closedPage() && (!f || veiled(f)); + } function veilTags(kind, key) { return ((results && results.veil && results.veil[kind]) || {})[key] || []; } // The charts (plugins/charts.js) ask for each mark. A closed page with no results.json yet cannot tell which // marks those are, so it leaves every mark out until it can. @@ -362,6 +369,32 @@ return [facts, w, shut]; } function veiledOf(c) { return (c.findings || []).map(findingById).filter(function (f) { return f && veiled(f); }); } + // The results to act on this page keeps closed, drawn yet or not. + function closedActs() { return ((results && results.findings) || []).filter(function (x) { return x.level === 'act' && veiled(x); }); } + // A card of a fixed panel (results.veil.panels, bin/live_results.py FIXED_PANELS: the same genes or loci for + // everyone, known in advance): a row or dot of it closed with a result would tell the result's gene by what is + // missing. So it closes whole while this page keeps any result to act on closed, behind one box with the same + // words whatever the result, and comes back once the last one is opened. + function panelShut(c) { + var p = (results && results.veil && results.veil.panels) || []; + return closedPage() && p.indexOf(c.id) >= 0 && closedActs().length > 0; + } + var PANEL_WORDS = 'This section lists the same genes for everyone, so it stays closed while a result to act on is closed ' + + 'on this page: what it left out would tell which gene. It opens once every such result is opened.'; + var PANEL_LINE = 'The same genes for everyone, closed while a result about a child is closed.'; + // The next closed result in the page's order (the map's Open opens the same one). + function nextClosed() { var ids = mapClosed(); return ids.length ? findingById(ids[0]) : closedActs()[0]; } + // A closed finding's handle in the page: its place in the page's list of closed findings, never its id, which + // names its gene and variant. The id comes back once the finding is opened on purpose (#f/ too). + function closedOrder() { + var out = []; + sectionsList().forEach(function (sec) { + (sec.findings || []).forEach(function (id) { var f = findingById(id); if (f && veiled(f) && out.indexOf(id) < 0) { out.push(id); } }); + }); + ((results && results.findings) || []).forEach(function (f) { if (veiled(f) && out.indexOf(f.id) < 0) { out.push(f.id); } }); + return out; + } + function handle(f) { return veiled(f) ? 'closed-' + (closedOrder().indexOf(f.id) + 1) : f.id; } // The search box (plugins/search.js) asks the same question for each hit, by the summary key and the anchor // of the section it comes from: a hit from a card this page keeps closed is left out. A closed page with // no results.json yet cannot tell which cards those are, so it leaves every hit out until it can. @@ -372,7 +405,7 @@ if (!results || key === null) { return true; } if (tag && tagClosed(tag)) { return true; } return cards().some(function (c) { - return (c.id === anchor || (c.keys || []).indexOf(key) >= 0) && veiledOf(c).length > 0; + return (c.id === anchor || (c.keys || []).indexOf(key) >= 0) && (veiledOf(c).length > 0 || panelShut(c)); }); }; @@ -414,6 +447,10 @@ 'Step ' + (c.steps || []).join(', ') + ' failed: the step timeline says why, and a rerun of the step fills this section.')); } else if (g === 'empty') { add(text, el('p', { 'class': 'notassessed' }, 'Not assessed: its step ended without a result for this section.')); + } else if (panelShut(c)) { + // A fixed-panel card says the same words whichever result is closed, its own findings' too: so it comes first. + // One button opens one result, the next closed one in the page's order, as the map's Open does. + add(text, U.veil(PANEL_WORDS, null, function () { var f = nextClosed(); if (f) { openFinding(f); } redraw(true); })); } else if (closedFs.length) { // One button opens one result: a card with two results to act on asks twice. add(text, U.veil((closedFs.length === 1 ? 'This section holds a result' : 'This section holds ' + closedFs.length + ' results') + @@ -435,12 +472,9 @@ var shut = ot[1] + (c.lines || []).length - lines.length, items = of[2]; if (tb) { add(text, tb); } else if (c.empty_text && !ot[1]) { add(text, el('p', { 'class': 'empty' }, c.empty_text)); } if (shut || items) { - // Rows of its table and lines, and items of its facts and drawing, counted apart. - var what = [shut ? (shut === 1 ? 'One row' : shut + ' rows') : '', items ? (items === 1 ? (shut ? 'one item' : 'One item') : items + ' items') + - ' of its numbers or drawing' : ''].filter(Boolean).join(' and '); - var many = shut + items > 1; - add(text, el('p', { 'class': 'dim veil-rows' }, what + (many ? ' stay' : ' stays') + - ' closed with the result to act on ' + (many ? 'they belong' : 'it belongs') + ' to, until that result is opened.')); + // The same words however many rows or items close, and whichever result they belong to: a count would + // tell one closed result from another. + add(text, el('p', { 'class': 'dim veil-rows' }, 'Some of its rows or numbers stay closed with a result to act on, until that result is opened.')); } add(text, lineList(lines)); if (c.hidden) { @@ -465,7 +499,7 @@ add(foot, el('a', { href: 'report.html#' + encodeURIComponent(c.anchor), 'class': 'report' }, 'Open in the report')); var vtags = veilTags('viewer', c.id); var reads = (c.viewer || []).filter(function (x, i) { return !(vtags[i] && tagClosed(vtags[i])); }); - if (g === 'show' && reads.length) { + if (g === 'show' && reads.length && !panelShut(c)) { add(foot, el('a', { href: 'viewer/#' + reads[0], 'class': 'viewer' }, 'See the reads at ' + reads[0])); } (g === 'show' ? c.links || [] : []).forEach(function (f) { @@ -545,7 +579,48 @@ var t = findingTag(f); return !!(t && tagClosed(t)); } - function openFinding(f) { shown[f.id] = true; var t = findingTag(f); if (t) { shown[t] = true; } } + // A finding of a fixed-panel card (tag PANEL) stays closed while any result to act on is: opening it opens the next + // closed result in the page's order, as its card's box does. + function openFinding(f) { + var t = findingTag(f), n = t === PANEL ? nextClosed() : null; + keepHash(function () { shown[f.id] = true; if (n) { shown[n.id] = true; } else if (t && t !== PANEL) { shown[t] = true; } }); + } + // The address never holds the id of a finding this page can close, closed or opened: #f/closed- while it is + // closed (its handle) and #f/opened- once opened on purpose (its place among the opened ones), so no history + // entry keeps the id. Any other #f/ on a page that keeps a result closed, a closed result's id or no id at all, + // lands the same way, on #f/closed-1: the page never picks a closed card by its id, so a guessed id tells nothing. + // Opening or closing a result moves the handles of the others: the address follows the finding it named. The + // finding behind a handle in the address is kept here only (hashId); a new address from the reader forgets it. + var hashId = null; + window.addEventListener('hashchange', function () { hashId = null; }); + function closable(f) { return closedPage() && (f.level === 'act' || !!findingTag(f)); } + function openedOrder() { + var out = []; + sectionsList().forEach(function (sec) { + (sec.findings || []).forEach(function (id) { var f = findingById(id); if (f && closable(f) && !veiled(f) && out.indexOf(id) < 0) { out.push(id); } }); + }); + ((results && results.findings) || []).forEach(function (f) { if (closable(f) && !veiled(f) && out.indexOf(f.id) < 0) { out.push(f.id); } }); + return out; + } + function addr(f) { return !closable(f) ? f.id : veiled(f) ? handle(f) : 'opened-' + (openedOrder().indexOf(f.id) + 1); } + function hashFinding() { + var h; + try { h = decodeURIComponent((window.location.hash || '').slice(1)); } catch (e) { return null; } + if (h.indexOf('f/') !== 0) { return null; } + var x = h.slice(2), m = /^(closed|opened)-(\d+)$/.exec(x); + var f = (m && hashId && findingById(hashId)) || + (m ? findingById((m[1] === 'closed' ? closedOrder() : openedOrder())[Number(m[2]) - 1]) : findingById(x)); + if (f && !(veiled(f) && !m)) { return f; } + return closedOrder().length ? findingById(closedOrder()[0]) : null; + } + function syncHash() { + var f = hashFinding(); + if (!f || !window.history || !window.history.replaceState) { return; } + hashId = f.id; + var to = '#f/' + addr(f); + if (window.location.hash !== to) { window.history.replaceState(null, '', to); } + } + function keepHash(change) { var at = hashFinding(); if (at) { hashId = at.id; } change(); syncHash(); } // The genome map's veil (map.js): 'closed' while this page keeps the result closed, 'opened' once it was // opened on purpose, 'open' when this page never closes it. A gated result (APOE, behind its warning) is // closed on every page, an adult's too, until opened; a tag naming no finding of these results stays closed. @@ -561,8 +636,10 @@ function setShown(id, on) { var f = findingById(id); if (on) { if (f) { openFinding(f); } else { shown[id] = true; } } else { - delete shown[id]; - ((results && results.findings) || []).forEach(function (x) { if (findingTag(x) === id) { delete shown[x.id]; } }); + keepHash(function () { + delete shown[id]; + ((results && results.findings) || []).forEach(function (x) { if (findingTag(x) === id) { delete shown[x.id]; } }); + }); } redraw(true); } @@ -582,6 +659,8 @@ return findingsNow().filter(function (f) { return f.level === 'act' && veiled(f); }).map(function (f) { return f.id; }); } // The title of a result once the page shows it, for the map's "Opened in its section" chip; never a closed one's. + // The address of a result the map opened, for its "Opened in its section" link: never its id (addr()). + function mapHref(id) { var f = findingById(id); return f ? addr(f) : id; } function mapTitle(id) { var f = findingById(id); return f && !veiled(f) ? f.title : ''; } // Whether a row of the map is a finding of these results: then the page's own veil alone decides it (map.js). function mapKnown(id) { return !!findingById(id); } @@ -596,7 +675,7 @@ } function findingArticle(f, big) { var row = el('div', { 'class': 'row' }); - var art = el('div', { 'class': 'finding', id: 'f-' + f.id, 'data-finding': f.id, 'data-level': f.level }); + var art = el('div', { 'class': 'finding', id: 'f-' + handle(f), 'data-finding': handle(f), 'data-level': f.level }); var at = newAt(f); if (at) { art.classList.add('new'); add(art, el('span', { 'class': 'newband' }, 'New at ' + U.hm(at))); add(art, el('br', {})); } add(art, U.findingLevel(f.level, f.level_words)); @@ -625,14 +704,14 @@ var rel = (f.related || []).map(findingById).filter(function (r) { return r && !veiled(r); }); if (rel.length) { var box = el('div', {}); - rel.slice(0, 4).forEach(function (r, i) { if (i) { add(box, el('br', {})); } add(box, el('a', { href: '#f/' + r.id }, r.title)); }); + rel.slice(0, 4).forEach(function (r, i) { if (i) { add(box, el('br', {})); } add(box, el('a', { href: '#f/' + addr(r) }, r.title)); }); add(side, note('Linked', box, 'linked-note')); } add(row, art, side); return row; } function findingLine(f) { - var li = el('li', { id: 'f-' + f.id, 'data-finding': f.id, 'data-level': f.level }); + var li = el('li', { id: 'f-' + handle(f), 'data-finding': handle(f), 'data-level': f.level }); if (veiled(f)) { // A finding that names a closed result (a symptom ranking of its gene): closed with it, opened with it. var b = el('button', { type: 'button', 'class': 'btn' }, 'Show the result'); @@ -766,6 +845,7 @@ function cardLine(c) { var g = gate(c); return g === 'show' && veiledOf(c).length ? 'A result about a child, closed until opened.' + : g === 'show' && panelShut(c) ? PANEL_LINE : g === 'show' ? (c.headline || (openFacts(c)[0][0] ? openFacts(c)[0][0][0] + ': ' + openFacts(c)[0][0][1] : NOT_ALL_CLEAR)) : pending(g) ? 'Waiting for ' + stepWords(c) + '.' : g === 'failed' ? 'Not assessed: its step failed.' : 'Not assessed in this run.'; } @@ -897,7 +977,7 @@ add(trust, el('span', { title: f[0] + ': ' + f[1] }, String(f[0]).replace(/\s*\([^)]*\)/g, '') + ': ' + String(f[1]).split(/[.;(]\s/)[0].replace(/[.;]$/, '').trim())); }); } else { add(trust, el('span', {}, 'Waiting for the quality steps.')); } - openProblems.forEach(function (f) { add(trust, el('span', { 'class': 'sep' }, '·'), el('a', { href: '#f/' + f.id }, f.title)); }); + openProblems.forEach(function (f) { add(trust, el('span', { 'class': 'sep' }, '·'), el('a', { href: '#f/' + addr(f) }, f.title)); }); add(trust, el('span', { 'class': 'sep' }, '·'), el('a', { href: '#s5' }, 'Can I trust this?')); add(main, trust); var side = el('aside', {}); @@ -909,7 +989,7 @@ // reader left it. It asks this page's veil for each closed row, and opens or closes a result through it. if (window.PGPMap) { if (!genomeMap) { - genomeMap = window.PGPMap.create({ url: 'map.json', veil: mapVeil, now: mapNow, known: mapKnown, closed: mapClosed, title: mapTitle, stepDone: stepDone, open: function (id) { setShown(id, true); }, + genomeMap = window.PGPMap.create({ url: 'map.json', veil: mapVeil, now: mapNow, known: mapKnown, closed: mapClosed, title: mapTitle, href: mapHref, stepDone: stepDone, open: function (id) { setShown(id, true); }, close: function (id) { setShown(id, false); }, fallback: mapFallback }); } add(slot, genomeMap.node); @@ -1104,7 +1184,7 @@ if (h.indexOf('s/') === 0 || (h && !/^(s\d+|f\/.*|q\/.*|progress)$/.test(h) && cardById(h))) { target = document.getElementById('card-' + (h.indexOf('s/') === 0 ? h.slice(2) : h)); isCard = true; - } else if (h.indexOf('f/') === 0) { target = document.getElementById('f-' + h.slice(2)); } + } else if (h.indexOf('f/') === 0) { var hf = hashFinding(); target = document.getElementById('f-' + (hf ? handle(hf) : h.slice(2))); } else if (h.indexOf('q/') === 0) { target = document.getElementById('s' + (CHAPTER_SECTION[h.slice(2)] || 1)); } else { target = document.getElementById(h); } return target ? { target: target, isCard: isCard } : null; @@ -1124,12 +1204,18 @@ } } function route() { + syncHash(); var t = routeTarget(); if (!t) { return; } var target = t.target; openAround(t); target.scrollIntoView({ block: 'start' }); if (target.focus) { target.focus({ preventScroll: true }); } + markToc(target); + } + // The contents mark the section the address names; a redraw marks it again, so the mark never depends on + // whether the page drew again after the address was read. + function markToc(target) { Array.prototype.forEach.call(rail.querySelectorAll('[data-toc]'), function (a) { var sec = target.closest ? target.closest('section.sec') : null; a.setAttribute('aria-current', sec && sec.id === 's' + a.getAttribute('data-toc') ? 'true' : 'false'); @@ -1205,6 +1291,9 @@ // A gated result's fold follows its switch, whatever fold state the reader left (setShown). Array.prototype.forEach.call(panel.querySelectorAll('details[data-gate]'), function (d) { d.open = !!shown[d.getAttribute('data-gate')]; }); window.scrollTo(0, y); + syncHash(); + var rt = routeTarget(); + if (rt) { markToc(rt.target); } document.getElementById('progress-q').textContent = running() ? 'Is it moving, when will it end, is anything wrong? One line per step.' : 'One line per step: when it ran, and what it found.'; // The step timeline closes the report, folded once the page has a report to show, while the run goes diff --git a/bin/live_results.py b/bin/live_results.py index 3f0a3a60..65c542fe 100644 --- a/bin/live_results.py +++ b/bin/live_results.py @@ -179,7 +179,7 @@ def a_clinvar(S, s): lines = [x for x in (rr.cdna_scan_lines(d) or [])] for x in d.get("deep_intronic") or []: # Always reported by the report, hit or not: each site's call stays visible on the card. - lines.append(f"Known deep intronic site (always reported): {x.get('gene')} {x.get('hgvs')} " + lines.append(f"{FIXED_LINES['clinvar'][0]} {x.get('gene')} {x.get('hgvs')} " f"{x.get('chrom')}:{x.get('pos')} {x.get('ref')}>{x.get('alt')}: {rr.deep_intronic_call(x)}") freq = rr.clinvar_freq_line(d) if freq: @@ -888,6 +888,9 @@ def gene(x): gene=(x.get("genes") or [None])[0], detail=x.get("sv"), evidence=[["Class", x.get("acmg_class")], ["Callers", rr.annotsv_support_text(x)], ["Genotype", x.get("gt")]], viewer=str(x.get("sv") or "").replace(" ", "") or None)) + # Every gene it covers, not only the first: a closed one closes the rows that name any of them (closable()). + gl = x.get("genes") + out[-1]["genes"] = [str(g) for g in gl if g] if isinstance(gl, list) else [] # 6. Repeat expansions outside Stranger's "normal", or in an intermediate zone. for l in _items(_data(S, "expansions").get("loci")): with _Row(out, "expansions", "act"): @@ -1144,8 +1147,76 @@ def lead(found, levels, summary): # belongs to a closed result when it names the result's gene, or its position falls inside the result's # position or span; a row that is the result's own position or span belongs to that result first, else # the first such result in finding order. The veil lives in the browser: the files still hold every row. +# The cards whose rows are a fixed panel: the same genes or loci for everyone, known before the run. A row or +# dot of one that closes with a result would tell which gene the result is in by what is missing, so a child's +# page closes each of these cards whole while any result to act on is closed, behind one box with the same words +# whatever the result (results.json veil.panels; assets/live/person.js panelShut()). The one table: a card of a +# fixed panel is listed here and nowhere else. +FIXED_PANELS = { + "mei": "the known disease-causing insertions of assets/mei_sites.tsv", + "expansions": "the repeat loci of the catalog", + "vntyper": "MUC1", + "lpa": "LPA", + "chip": "the CHIP genes and their detection limits", + "paralogs": "SMN1, SMN2, HBA1, HBA2, the RCCX module and the other paralog genes", + "cpic": "the CPIC genes", + "cyp2d6": "CYP2D6", + "hla": "the HLA genes", + "locityper": "the Locityper loci", + "traits-panel": "the trait panel's genes", + "bloodtype": "the blood group systems", +} +# The lines and table rows of a card that are a fixed panel inside it (card -> the words such a line, or a cell of +# such a row, starts with): the ClinVar card prints one line per known deep intronic site of +# assets/deep_intronic_sites.tsv, hit or not; the carrier list prints its two-hit line, which names SMN1, HBA1 and +# HBA2 as the callers it read, whatever was found; the callability card lists the catalog genes this reference does +# not hold, the same for every genome on it, and the hard genes the reads did not cover, which are nearly the same +# for every genome (segmental duplications, collapsed regions). veil() tags each of them PANEL, never by a gene, so they close all +# together while any result to act on is closed and open all together once the last one is opened +# (assets/live/person.js tagClosed()). Every other line or row says something about this genome alone. +FIXED_LINES = {"clinvar": ("Known deep intronic site (always reported):",), + "carrier-list": ("Two hits across callers",), + "callability": ("not on this reference", "not read")} +PANEL = "*panel*" +# The cards whose rows, lines and facts name no gene, only things known before the run: veil() never tags them. The +# score card names its scores and their traits (assets/pgs_scores.tsv) and its words "OR 1.57", "HR 1.4", "(TG)" +# are odds and hazard ratios and a trait, though HR and TG are gene symbols too. +NAMES_NO_GENE = {"prs": "the polygenic scores of assets/pgs_scores.tsv"} + + +def panel_finding(f): + """Whether finding F is one of a fixed-panel card's own (a medicine gene, a CHIP call, a repeat), not a result + to act on: veil() tags it PANEL whatever it names.""" + return f.get("level") != "act" and f.get("card") in FIXED_PANELS + + +def fixed_line(card_id, text): + """Whether TEXT, a line of the card CARD_ID or a cell of one of its rows, is one of its fixed lines (FIXED_LINES).""" + return any(str(text).startswith(w) for w in FIXED_LINES.get(card_id) or ()) LOCUS_RE = re.compile(r"(? one id or None per row; the ClinVar card's by each hit's own position), each card's viewer - links (viewer: card -> one per link), each card's lines (lines: card -> one per line; a known deep - intronic site is printed whether hit or not), the marks of the report's charts (marks: chart -> {mark id: - id}), every other finding that names a closed result (findings: finding id -> id; a symptom ranking can - rank the gene) and the lead's segments that name one (lead: one per segment of LEAD_PARTS, the "act" - names left out: the page closes those itself), each card's facts (facts: card -> one per fact) and the items - of its drawing (widget: card -> one per CHIP call or copy-number dot). spans is closable() itself: the family + links (viewer: card -> one per link), each card's lines (lines: card -> one per line; a line of a fixed + panel, such as a known deep intronic site, printed whether hit or not, is tagged PANEL: FIXED_LINES), the + marks of the report's charts (marks: chart -> {mark id: id}), every other finding that names a closed result (findings: finding id -> id; a symptom ranking can + rank the gene; PANEL for every finding of a FIXED_PANELS card) and the lead's segments that name one or name a + fixed-panel card's finding (lead: one per segment of LEAD_PARTS, the "act" names left out: the page closes those + itself), each card's facts (facts: card -> one per fact) and the items + of its drawing (widget: card -> one per CHIP call or copy-number dot), and the cards of a fixed panel + (panels: FIXED_PANELS, which close whole while any result is closed). spans is closable() itself: the family page (assets/live/family-page.js) reads which results are closed from it, and closes every place of the child that could carry one while one is. Empty when nothing can be closed.""" spans = closable(found) if not spans: return {} out = {"rows": {}, "viewer": {}, "lines": {}, "facts": {}, "widget": {}, "marks": {}, "findings": {}, "lead": [], - "spans": [list(s) for s in spans]} + "spans": [list(s) for s in spans], + "panels": [c["id"] for ch in chapters for c in ch["sections"] if c["id"] in FIXED_PANELS and c.get("state") != "missing"]} S = summary.get("sections") or {} hits = [h for h in _items(_data(S, "clinvar").get("hits"))[:50]] tags = [veil_id(spans, str(h.get("gene") or ""), h.get("chrom"), h.get("pos")) if isinstance(h, dict) else None @@ -1227,9 +1308,12 @@ def veil(summary, found, chapters, lead_parts=()): out["rows"]["clinvar"] = tags for ch in chapters: for c in ch["sections"]: + if c["id"] in NAMES_NO_GENE: + continue rows = (c.get("table") or {}).get("rows") or [] if c["id"] != "clinvar" and rows: - tags = [veil_id(spans, " ".join(str(x) for x in r) if isinstance(r, (list, tuple)) else str(r)) + tags = [PANEL if any(fixed_line(c["id"], x) for x in (r if isinstance(r, (list, tuple)) else [r])) + else veil_id(spans, " ".join(str(x) for x in r) if isinstance(r, (list, tuple)) else str(r)) for r in rows] if any(tags): out["rows"][c["id"]] = tags @@ -1239,7 +1323,7 @@ def veil(summary, found, chapters, lead_parts=()): tags = [(rt[i] if i < len(rt) else None) or t for i, t in enumerate(tags)] if any(tags): out["viewer"][c["id"]] = tags - tags = [veil_id(spans, ln) for ln in c.get("lines") or []] + tags = [PANEL if fixed_line(c["id"], ln) else veil_id(spans, ln) for ln in c.get("lines") or []] if any(tags): out["lines"][c["id"]] = tags tags = [veil_id(spans, " ".join(str(x) for x in f)) for f in c.get("facts") or []] @@ -1264,13 +1348,23 @@ def veil(summary, found, chapters, lead_parts=()): m.get("end")) if fid: out["marks"].setdefault(sp["id"], {})[m["id"]] = fid + # A finding of a fixed-panel card (a medicine gene, a CHIP call, a repeat) is tagged PANEL whatever it names: it + # closes with its card while any result to act on is closed, so the findings left open never tell which gene the + # closed one is in. So is each segment of the lead that names one (its gene or its title). + panel_found = [f for f in found if panel_finding(f)] + panel_spans = [(PANEL, str(g).upper(), None, None, None) for f in panel_found for g in [f.get("gene")] if g] for f in found: if f.get("level") != "act": - fid = veil_id(spans, f"{f.get('gene') or ''} {f.get('detail') or ''} {f.get('title') or ''} " - f"{f.get('viewer') or ''}") + fid = PANEL if f in panel_found else veil_id(spans, f"{f.get('gene') or ''} {f.get('detail') or ''} " + f"{f.get('title') or ''} {f.get('viewer') or ''}") if fid: out["findings"][f["id"]] = fid - tags = [None if len(x) > 2 else veil_id(spans, x[0]) for x in lead_parts or ()] + + def lead_tag(t): + if veil_id(panel_spans, t) or any(f.get("title") and f["title"] in t for f in panel_found): + return PANEL + return veil_id(spans, t) + tags = [None if len(x) > 2 else lead_tag(x[0]) for x in lead_parts or ()] out["lead"] = tags if any(tags) else [] return {k: v for k, v in out.items() if v} @@ -1315,7 +1409,8 @@ def overview(summary): order["trust"] = -1 # a problem with the data comes before any finding read from it # A finding that names a result to act on (veil: a symptom ranking of its gene) is not listed: a child's tile # closes only the results to act on themselves, and the result it names already comes first. - named = (r.get("veil") or {}).get("findings") or {} + # A finding tagged PANEL (a fixed-panel card's own, veil()) names no result: it stays, the same for everyone. + named = {k: v for k, v in ((r.get("veil") or {}).get("findings") or {}).items() if v != PANEL} top = sorted((f for f in r["findings"] if f["level"] != "info" and f["id"] not in named), key=lambda f: order[f["level"]])[:4] tags = (r.get("veil") or {}).get("lead") or [] diff --git a/docs/live-site.md b/docs/live-site.md index b2b16746..7d9ea19a 100644 --- a/docs/live-site.md +++ b/docs/live-site.md @@ -56,7 +56,12 @@ Every result carries one of six levels, with a word, a shape and a colour, so th 9. **What it cannot see**: what the run does not assess and the steps that did not run. 10. **Sources and files**: the database versions, the database re-check, and the report and its files. -A card has its rows (the first ones, then a button for all), its own sentences and its charts at reading width, and in a margin beside them its numbers, its caveat (the first sentence, then the rest behind "Full caveat") and its links; below 1100 px the margin goes under the text. A card with more than six lines keeps the ones that say something in view and folds the rest (an allele absent, a call not determined) behind one line that counts them. A section waiting for its step says "being written" or "waiting", one whose step failed says so, and a newer database release shows on the cards it would change. Some cards draw their numbers as well: the repeat expansions as one gauge per locus (the count against Stranger's normal and pathologic thresholds, with its 95% interval; an allele not sized from short reads is drawn hollow and never called normal), Lp(a) as the KIV-2 count with its 95% interval, CHIP as each call's VAF against the detection limits at this depth (hatched below every gene's limit, shaded where some genes see a clone and others do not, and each call named against its own gene's limit), the copy numbers of SMN1, SMN2, HBA1, HBA2 and the RCCX module as dots (Parascopy's SMN1 and SMN2 at exon 7 too, from short reads, hollow with its reason when the count is not resolved), and the Y haplogroup as its path from the root, with the marker counts of each level. Each drawing has the same numbers as text beside it. APOE stays behind its warning in the traits card; the polygenic scores show only once they are placed against the ancestry panel. On a child's page in a family, a result to act on stays closed behind "Show the result" until someone opens it on purpose: its title, gene and variant stay out of the lead, the ledger, the genome map, the home page and its own page, and the card it comes from stays closed with it (its rows, numbers and drawing), in its section and in section 8. Anything else on the child's pages that names a closed result's gene or position stays closed with it: another card's rows, lines and facts, a copy-number dot (a CHIP call is a finding of its own card, so the CHIP card closes whole), a viewer link, a chart's marks and their titles, another finding that names it (a symptom ranking of the gene) on the report, on its own page, in a finding page's next link and in its related list, and the part of the lead that names it. A count says how many rows stay closed, and opening the result opens them with it. Each result opens on its own: opening one leaves every other result to act on closed, on the report and on the finding's page. A person in a family whose family report has not answered yet, or answered without listing them, is treated the same way, since the page cannot tell whether they are a child. So is a page whose run state is unavailable or comes without its plugins, and a person no readable family lists while some family's pedigree cannot be read (the family card says so): an unknown role is never read as a standalone adult. The header search follows the same veil: a hit from a card the page keeps closed, or one that names a closed result, is left out, and so is a suggested gene that only such hits name. On the genome map a closed result has no mark, no tooltip, no line in the margin and no place in the address; a dashed chip above the map counts the closed results without naming a gene or a place, and its Open button asks once before it opens the next one. The chip counts every result to act on the page keeps closed, with a place on the map or without, so the count never tells whether a closed result has a place. Open opens one result per click, the next closed result in the page's order (the order of its closed cards), so the order does not tell either whether a closed result has a place. Opening a result there or with its card's "Show the result" is one switch: it opens on the map and in its card together, the chip then reads "Opened:" with the result's title and a button to close it again, and a reload closes it. A result with no place on the map opens in its section only: the map draws nothing new, and a chip reads "Opened in its section:" with the result's title, a link to its card and a button to close it again. That chip names only a result that is open, and it goes when the result closes. After Open the keyboard moves to "Show the result", and after "Show the result" or "Keep it closed" to the chips' first button, or to the map when no chip is left. A finding whose card the page shows stays on the map even when a closed result lies inside its gene's span: hiding it would show that the closed result sits in that gene. The page's own veil decides each of its findings, and a result opened with its card shows on the map even when it overlaps another result that is still closed. APOE is closed on the map on every page, an adult's too, until it is opened with its own chip or with the fold of its warning in the traits card, which is the same switch; the chip and the switch appear only once the traits card shows, so a replay before its step has none. The veil lives in the browser: the map document, like `results.json`, reaches the browser whole, with each row a closed result could reveal tagged with that result's id. +A card has its rows (the first ones, then a button for all), its own sentences and its charts at reading width, and in a margin beside them its numbers, its caveat (the first sentence, then the rest behind "Full caveat") and its links; below 1100 px the margin goes under the text. A card with more than six lines keeps the ones that say something in view and folds the rest (an allele absent, a call not determined) behind one line that counts them. A section waiting for its step says "being written" or "waiting", one whose step failed says so, and a newer database release shows on the cards it would change. Some cards draw their numbers as well: the repeat expansions as one gauge per locus (the count against Stranger's normal and pathologic thresholds, with its 95% interval; an allele not sized from short reads is drawn hollow and never called normal), Lp(a) as the KIV-2 count with its 95% interval, CHIP as each call's VAF against the detection limits at this depth (hatched below every gene's limit, shaded where some genes see a clone and others do not, and each call named against its own gene's limit), the copy numbers of SMN1, SMN2, HBA1, HBA2 and the RCCX module as dots (Parascopy's SMN1 and SMN2 at exon 7 too, from short reads, hollow with its reason when the count is not resolved), and the Y haplogroup as its path from the root, with the marker counts of each level. Each drawing has the same numbers as text beside it. APOE stays behind its warning in the traits card; the polygenic scores show only once they are placed against the ancestry panel. On a child's page in a family, a result to act on stays closed behind "Show the result" until someone opens it on purpose: its title, gene and variant stay out of the lead, the ledger, the genome map, the home page and its own page, and the card it comes from stays closed with it (its rows, numbers and drawing), in its section and in section 8. Anything else on the child's pages that names a closed result's gene or position stays closed with it: another card's rows, lines and facts, a viewer link, a chart's marks and their titles, another finding that names it (a symptom ranking of the gene) on the report, on its own page, in a finding page's next link and in its related list, and the part of the lead that names it. A structural variant names every gene it covers, so a row that names any of them closes with it. A card with such rows says that some of its rows stay closed, in the same words however many there are, and opening the result opens them with it. Two pages whose open rows are the same differ only in how many results are closed, not in which; the section that holds a closed result still says it holds one: + +- A card whose rows are a fixed panel, the same genes or loci for everyone, would tell the closed result's gene by the row it leaves out. So these cards close whole while any result to act on is closed, behind one box with the same words whatever the result: the known mobile element insertions, the repeat expansions, VNtyper, Lp(a), CHIP, the paralog copy numbers (SMN1, SMN2, HBA1, HBA2, the RCCX module), CPIC, CYP2D6, HLA, Locityper, the traits panel and the blood groups. The one list is `FIXED_PANELS` in `bin/live_results.py`, and `results.json` gives it as `veil.panels`. Their "Show the result" opens the next closed result, and the cards come back once every result is opened. The cost: a child's page with a closed result shows none of these cards until the results are opened. Their own findings (a medicine gene, a CHIP call, a repeat) close with them, whatever gene they name, and so does a segment of the lead that names one (`veil.findings` and `veil.lead` tag them `*panel*`): a medicine finding that closed only when the closed result is in its gene would tell the gene. A card can hold a fixed panel among its lines or rows: the ClinVar card prints a line for each known deep intronic site, hit or not; the carrier list prints its two-hit line, which names the copy-number callers it read; and the callability card lists the genes this reference does not hold and the hard genes the reads did not cover, nearly the same for everyone. Those lines and rows close all together while any result to act on is closed and come back once every result is opened (`FIXED_LINES` in `bin/live_results.py`; `veil.lines` and `veil.rows` tag each of them `*panel*`). A row or line closes with a result only when it names the result's gene written as a gene symbol: a prose word ("set"), an ontology id (`HP:0001250`) or a unit after a number (`5 MB`, `48-52% GC`) names no gene, and the polygenic score card, which names scores and traits only, is never tagged (`NAMES_NO_GENE`). +- A closed finding carries no id in the page or the address, since its id names its gene and variant. It is known by its place among the page's closed findings (`#f/closed-2`). On a page that keeps a result closed, the address never holds the id of a finding the page can close, open or closed: once opened on purpose it is `#f/opened-`, its place among the opened ones, so no history entry keeps the id. Any other `#f/` on such a page, the id of a closed result or no id at all, lands the same way, on the first closed finding (`#f/closed-1`): the page never picks a closed card by its id, so a guessed id tells nothing. On the home page, a child's closed result links to the child's report, never to the finding's own page, whose address names it. + +Each result opens on its own: opening one leaves every other result to act on closed, on the report and on the finding's page. A person in a family whose family report has not answered yet, or answered without listing them, is treated the same way, since the page cannot tell whether they are a child. So is a page whose run state is unavailable or comes without its plugins, and a person no readable family lists while some family's pedigree cannot be read (the family card says so): an unknown role is never read as a standalone adult. The header search follows the same veil: a hit from a card the page keeps closed, or one that names a closed result, is left out, and so is a suggested gene that only such hits name. On the genome map a closed result has no mark, no tooltip, no line in the margin and no place in the address; a dashed chip above the map counts the closed results without naming a gene or a place, and its Open button asks once before it opens the next one. The chip counts every result to act on the page keeps closed, with a place on the map or without, so the count never tells whether a closed result has a place. Open opens one result per click, the next closed result in the page's order (the order of its closed cards), so the order does not tell either whether a closed result has a place. Opening a result there or with its card's "Show the result" is one switch: it opens on the map and in its card together, the chip then reads "Opened:" with the result's title and a button to close it again, and a reload closes it. A result with no place on the map opens in its section only: the map draws nothing new, and a chip reads "Opened in its section:" with the result's title, a link to its card and a button to close it again. That chip names only a result that is open, and it goes when the result closes. After Open the keyboard moves to "Show the result", and after "Show the result" or "Keep it closed" to the chips' first button, or to the map when no chip is left. A finding whose card the page shows stays on the map even when a closed result lies inside its gene's span: hiding it would show that the closed result sits in that gene. The page's own veil decides each of its findings, and a result opened with its card shows on the map even when it overlaps another result that is still closed. APOE is closed on the map on every page, an adult's too, until it is opened with its own chip or with the fold of its warning in the traits card, which is the same switch; the chip and the switch appear only once the traits card shows, so a replay before its step has none. While the traits card is closed whole on a child's page, the chip is the switch. The veil lives in the browser: the map document, like `results.json`, reaches the browser whole, with each row a closed result could reveal tagged with that result's id. A stopped run's page says so under the name and in the summary line, with the day it was last heard from, and warns that the report may be incomplete; a section still waiting for its step says the run stopped before that step ended. @@ -90,7 +95,7 @@ The reads cross the network to the browser that logged in: only the byte ranges `GET /` lists the runs: each folder under `GENOME_DIR` holding `logs/run_status.tsv`, `nextflow/` or `run_manifest.tsv`, and each run the site has messages for. The home page reads `/runs.json`: each run's progress and attention count, and each family with its members. A run with results also has `levels` (its findings counted by level), `top` (up to four findings to show first, data-quality problems first: id, level, title; a finding that names a result to act on, such as a symptom ranking of its gene, is never among them), `lead` (its lead paragraph as plain text), `lead_closed` (the same with the names of the findings to act on left out, and every other part that names one, for a child's tile) and `screened` (whether ClinVar, the clinical filter or CPSR has a result; without one the home page never says that no finding needs a doctor). While a run goes these come from the summary its page last read, as long as no step has finished and `summary.json` has not changed since that read, so the home page never shows less than the person page. While a family's members cannot be read, the home page keeps the findings to act on closed for every run that no readable family lists. -The person page reads `/runs//results.json` from `bin/owngenome_live_results.py`: `bin/live_results.py` turns the run's summary (its `summary.json` when step 24 wrote it after every finished step, else the step files read again when a step ends; a rewritten `summary.json` is read again too, and the page fetches the cards again) into the report's chapters, one card per section. Each card also has its `level`; the file adds `findings` (each with its id, level, section, title, plain text, gene, evidence, caveat, plan and related findings), `levels` (the counts), `journal` (the ten sections and the cards in each), `lead` (the lead paragraph as text pieces, the bold ones marked, and the names of the findings to act on marked `act`), `level_words`, `databases` and `veil` (on a child's page, the rows outside a result's own card that close with it, and `spans`, the closed results with their gene and position, from which the family page knows whether a child has a result still closed). It computes nothing about genetics and holds only what the report prints. The family page reads `/runs//family.json` from the family plugin, and asks again every 30 seconds, so a member's progress and a new couple screen show without a reload. +The person page reads `/runs//results.json` from `bin/owngenome_live_results.py`: `bin/live_results.py` turns the run's summary (its `summary.json` when step 24 wrote it after every finished step, else the step files read again when a step ends; a rewritten `summary.json` is read again too, and the page fetches the cards again) into the report's chapters, one card per section. Each card also has its `level`; the file adds `findings` (each with its id, level, section, title, plain text, gene, evidence, caveat, plan and related findings), `levels` (the counts), `journal` (the ten sections and the cards in each), `lead` (the lead paragraph as text pieces, the bold ones marked, and the names of the findings to act on marked `act`), `level_words`, `databases` and `veil` (on a child's page, the rows outside a result's own card that close with it; `spans`, the closed results with their position, one row per gene they name, from which the family page knows whether a child has a result still closed; and `panels`, the cards of a fixed panel). A structural variant's finding also lists every gene it covers in `genes`. It computes nothing about genetics and holds only what the report prints. The family page reads `/runs//family.json` from the family plugin, and asks again every 30 seconds, so a member's progress and a new couple screen show without a reload. ## Privacy and what is stored diff --git a/tests/test_live_demo.py b/tests/test_live_demo.py index 02fbf1b4..d0f75cf8 100644 --- a/tests/test_live_demo.py +++ b/tests/test_live_demo.py @@ -77,9 +77,14 @@ opened from its box, the same page whichever result is closed, and with his results unreadable "Show the rows" leaves it closed and "Show the map" opens it and no table; on - his page a CHIP call, a copy-number dot and a card's fact in the gene of a - closed result stay out of sight (on a phone the card's line too) until it - is opened; DEMO-001's page + his page the copy-number and CHIP cards, a fixed panel, close whole while + a result of his is closed (on a phone their line too), their box opens one + result per click and every dot, call and fact is back once all are open; + two of his pages that differ only in which result is closed (two SNV + genes, two deletions of two genes each, SMN1 against HBA1, a result with a + place on the map against one without, one ClinVar row against two, a link + to each) are the same before any click, HTML, attributes, address and map + canvas, and a closed deletion's second gene is named nowhere; DEMO-001's page mid-replay and at the end leads with the report (a short run band above the lead while it runs, none once finished), ends with the step timeline folded, keeps the wire to at most eight one-sentence lines with the plain @@ -99,8 +104,9 @@ request read as nothing closed, malformed veil.spans read as given, results without veil.spans read as nothing closed or as closed with nothing to act on, an opened result kept per run only or per child only, a couple screen closed only for a result in its genes or opened by 'Show the cells', a map image closed only for a result with a -position, and de novo rows that 'Show the map' opens; for the son's page, person.js that draws -every item, shows every fact or builds a card's line from every fact), the phone page (person.js that never folds, and a route that opens only the nearest fold), the +position, and de novo rows that 'Show the map' opens; for the son's page, person.js that draws a fixed-panel +card's open rows or builds its line from its facts, that puts a closed card's id back or counts the closed rows, +and a veil that tags only a deletion's first gene), the phone page (person.js that never folds, and a route that opens only the nearest fold), the folds (a flagged section 5 card folded, a redraw that forgets the folds, a Ctrl+click that opens one) and the person page's layout (a wire showing every line or every plain finish, a card showing every line, a step timeline left open) also run against a planted mistake and must @@ -110,6 +116,7 @@ """ import base64 import copy +import difflib import gzip import hashlib import json @@ -358,6 +365,7 @@ def browser_checks(out): map_checks(b, base, out) family_veil_checks(b, base, out) widget_veil_checks(b, base, out) + infer_checks(b, base, out) finally: b.close() srv.shutdown() @@ -402,8 +410,9 @@ def person_checks(b, base): WIDGETS = [("DEMO-002", "expansions", "str", "[data-widget=str] .w-row", 8), ("DEMO-002", "lpa", "ci", "[data-widget=ci] .w-tick", 1), ("DEMO-002", "chip", "vaf", "[data-widget=vaf] .w-tick.w-call", 1), - # five copy-number genes from Paraphase, then Parascopy's SMN1 and SMN2 - ("DEMO-003", "paralogs", "copies", "[data-widget=copies] li", 7), + # five copy-number genes from Paraphase, then Parascopy's SMN1 and SMN2, on the partner's page: the son's keeps + # the copy-number card closed while a result of his is closed (a fixed panel) + ("DEMO-002", "paralogs", "copies", "[data-widget=copies] li", 7), ("DEMO-003", "y-chromosome", "ypath", "[data-widget=ypath] li", 5), ("DEMO-002", "clinvar", "MoChA bar", "figure[data-chart=genome] rect.mca", 1), ("DEMO-003", "clinvar", "AnnotSV span", "figure[data-chart=genome] rect.sv", 1)] @@ -636,19 +645,61 @@ def planted_closed(b, base): return out -def planted_problems(b, base): +# The handle of a closed finding, as the page counts it (person.js closedOrder()): the closed findings in the order +# of the sections, then any other; a result to act on is closed until opened, a finding tagged with one while that +# result is, a finding tagged PANEL while any result to act on is, a finding tagged with no finding of the page always. +HANDLE_JS = r"""fetch('results.json', {cache: 'no-store'}).then(function (r) { return r.json(); }).then(function (r) { + var fid = %s, byId = {}, tags = (r.veil && r.veil.findings) || {}; + r.findings.forEach(function (f) { byId[f.id] = f; }); + function open(id) { return !!document.getElementById('f-' + id); } + var acts = r.findings.filter(function (f) { return f.level === 'act'; }); + var anyShut = acts.some(function (f) { return !open(f.id); }); + function shut(f) { + if (f.level === 'act') { return !open(f.id); } + var t = tags[f.id]; + if (!t || t === f.id) { return false; } + if (t === '*panel*') { return anyShut; } + return !byId[t] || !open(t); + } + var order = []; + (r.journal || []).forEach(function (s) { (s.findings || []).forEach(function (id) { + if (byId[id] && shut(byId[id]) && order.indexOf(id) < 0) { order.push(id); } }); }); + r.findings.forEach(function (f) { if (shut(f) && order.indexOf(f.id) < 0) { order.push(f.id); } }); + return order.indexOf(fid) + 1; +})""" + + +def open_closed(b, fid): + """Opens the closed result FID as a reader does who knows its place among the page's closed findings: its handle + (#f/closed-), never its id, which the page does not use to find a closed card; then its "Show the result". + False when the handle names no closed card.""" + r = b.call("Runtime.evaluate", {"expression": HANDLE_JS % json.dumps(fid), "awaitPromise": True, + "returnByValue": True}, b.session) + n = (r.get("result") or {}).get("value") or 0 + if not n: + return False + b.eval("location.hash = %s; true" % json.dumps(f"#f/closed-{n}")) + if not b.wait("location.hash === %s && document.getElementById('f-closed-%d') && location.hash" + % (json.dumps(f"#f/closed-{n}"), n), 5): + return False + b.eval("document.getElementById('f-closed-%d').querySelector('.veil button').click(); true" % n) + return True + + +def planted_problems(b, base, out_dir): """[problems] of the planted results on the son's page: closed, then opened one at a time, then reloaded.""" gene, hit, sv = planted_words() out = planted_closed(b, base) - ids = b.eval("Array.prototype.map.call(document.querySelectorAll('[data-finding][data-level=act]'), " - "function (e) { return e.getAttribute('data-finding'); }).join(' ')") or "" + with open(os.path.join(out_dir, "runs", OTHERS[1], "results.json")) as f: + ids = " ".join(x["id"] for x in json.load(f)["findings"] if x["level"] == "act") sf = next((i for i in ids.split() if i.startswith("sf-")), None) import make_demo_sample start = str(make_demo_sample.PLANTED["sv"][1]) del_ = next((i for i in ids.split() if i.startswith("sv-") and start in i), None) if not sf or not del_: return out + [f"the planted results are not among the son's results to act on: {ids}"] - b.eval("document.querySelector('#f-%s .veil button').click(); true" % sf) + if not open_closed(b, sf): + return out + [f"a link to {sf} lands on no closed card"] time.sleep(0.8) shown = b.eval(SHOWN_JS) or "" rows = b.eval("Array.prototype.filter.call(document.querySelectorAll('#card-clinvar tbody tr'), function (r) { " @@ -665,7 +716,8 @@ def planted_problems(b, base): hits, _ = son_search(b, gene) if hits in (None, "0"): out.append(f"the search for {gene} finds nothing once its result is opened") - b.eval("document.querySelector('#f-%s .veil button').click(); true" % del_) + if not open_closed(b, del_): + return out + [f"a link to {del_} lands on no closed card"] time.sleep(0.8) shown = b.eval(SHOWN_JS) or "" spans = b.eval("document.querySelectorAll('figure[data-chart=genome] a[data-mark^=sv-]').length") @@ -730,7 +782,7 @@ def write(p, text): with open(p, "w") as f: f.write(text) try: - bad = planted_problems(b, base) + bad = planted_problems(b, base, out) check("browser: the son's planted secondary finding and class 5 deletion name their gene and positions nowhere " "(cards, charts, titles, links, search) until each is opened, each opens only its own rows and marks, " "and a reload closes them", not bad, bad) @@ -754,11 +806,11 @@ def write(p, text): check("browser: the home page and every finding page of the son name the planted gene and positions nowhere " "(the rail's next link and the related list included) until opened, and the symptom ranking's page " "opened on purpose shows it", not bad, bad) - a = "!!tagOf(x) && !opened[tagOf(x)]" + a = "return t === PANEL ? !opened[x.id] && actShut() : !!t && !opened[t];" if a not in keep[fj]: check("control: finding.js whose findings ignore their tags (the planted line is in finding.js)", False, a) else: - write(fj, keep[fj].replace(a, "false")) + write(fj, keep[fj].replace(a, "return false;")) control("finding.js whose findings ignore their tags", not planted_pages(b, base, out)) write(fj, keep[fj]) finally: @@ -1399,10 +1451,379 @@ def planted_control(desc, a, z, problems): write(p, text) +# What a reader of the son's page gets before any click: the whole document (every text and every attribute, the +# map's timing attributes aside), the address, where the page is scrolled to and what has the focus, and a hash of +# the map's canvas. +PAGE_JS = r"""JSON.stringify((function () { + var c = document.documentElement.cloneNode(true); + Array.prototype.forEach.call(c.querySelectorAll('[data-pgp-map-parse-ms],[data-pgp-map-paint-ms]'), function (e) { + e.removeAttribute('data-pgp-map-parse-ms'); e.removeAttribute('data-pgp-map-paint-ms'); }); + // The root's data attributes come from scripts that answer in any order: sort them by name. + var at = Array.prototype.map.call(c.attributes, function (x) { return [x.name, x.value]; }).sort(); + at.forEach(function (x) { c.removeAttribute(x[0]); }); + at.forEach(function (x) { c.setAttribute(x[0], x[1]); }); + var cv = document.querySelector('.gm-canvas'), a = document.activeElement; + return { html: c.outerHTML, href: location.href + ' scroll ' + Math.round(window.scrollY) + ' focus ' + + (a ? a.tagName + '#' + a.id : ''), canvas: cv ? cv.toDataURL() : '' }; +}()))""" + + +def page_now(b, base, hash_="", typed=False): + """The son's page before any click, once its results, its charts and its map have drawn: (lines of its HTML, + its address with its scroll position and focus, its canvas hash). TYPED: the page loads without HASH_, which is + then set as a reader types it into the address bar (a hashchange); its canvas is left out then (a hashchange + can move the map's camera).""" + b.go("about:blank") + b.wait("location.href === 'about:blank'") + b.go(f"{base}/runs/{OTHERS[1]}/{'' if typed else hash_}") + # The son's family answers on its own request: wait for its name in the kicker too, or a pair can differ by it. + b.wait("var d = document.documentElement.dataset; d.pgpResults && d.pgpCharts && d.pgpStatus === 'ready' && (" + MAP_READY + + ") && document.getElementById('kicker').textContent.indexOf('Family') >= 0 && 'ready'", 15) + if typed: + b.eval(f"location.hash = {json.dumps(hash_)}; true") + time.sleep(0.8) # the margin follows the camera 140 ms after it settles + got = json.loads(b.eval(PAGE_JS) or "{}") + return (got.get("html") or "").replace("><", ">\n<").split("\n"), got.get("href") or "", \ + "" if typed else hashlib.sha256((got.get("canvas") or "").encode()).hexdigest()[:12] + + +def plant_summary(out, og, change): + """The son's results.json built again by live_results from his summary.json with CHANGE(sections) applied, and + his map.json built again from it by og map build. Returns what to write back afterwards.""" + son = OTHERS[1] + rj, mj = (os.path.join(out, "runs", son, n) for n in ("results.json", "map.json")) + keep = {} + for p in (rj, mj): + with open(p, "rb") as fh: + keep[p] = fh.read() + root = os.path.join(os.path.dirname(out), "root") + with open(os.path.join(root, son, "summary.json")) as fh: + summary = json.load(fh) + change(summary.setdefault("sections", {})) + res = live_results.build(son, summary) + with open(rj, "w") as fh: + json.dump(res, fh) + doc = build_live_demo.og_map(og, ["build", "--root", root, "--run", son, "--assets", os.path.join(REPO, "assets"), + "--cache-dir", os.path.join(root, ".owngenome-live", "map")], + json.dumps({"results": res}).encode()) + with open(mj, "wb") as fh: + fh.write(doc) + return keep + + +def cpsr(gene): + """A CPSR result to act on in GENE, the same variant whatever the gene: a closed result known by its gene.""" + def change(S): + S["cpsr"] = {"state": "ok", "data": {"call_floor_dp": 10, "low_confidence": 0, "plp": [ + {"gene": gene, "variant": "c.68_69del", "final": "Pathogenic", "clinvar": "Pathogenic"}]}} + return change + + +def deep_sites(S): + """The ClinVar card's line for each known deep intronic site of assets/deep_intronic_sites.tsv, reference calls: + a fixed panel the son's demo summary leaves out.""" + with open(os.path.join(REPO, "assets", "deep_intronic_sites.tsv")) as fh: + rows = [ln.rstrip("\n").split("\t") for ln in fh if ln.strip() and not ln.startswith("#")] + S["clinvar"]["data"]["deep_intronic"] = [dict(zip(rows[0], r), call="ref", detail="GQ 50") for r in rows[1:]] + return len(rows) - 1 + + +def with_sites(gene): + """cpsr(GENE) with the known deep intronic sites' lines (deep_sites), the same on both pages.""" + def change(S): + cpsr(gene)(S) + deep_sites(S) + return change + + +def sv_over(locus, genes): + """A class 5 deletion over GENES, and a carrier row that names only its second gene.""" + def change(S): + rows = S["annotsv"]["data"]["rows"] + S["annotsv"]["data"]["rows"] = rows + [{"sv": locus, "type": "DEL", "gt": "0/1", "length": "-100000", + "acmg_class": "5", "genes": genes, "caller": "Manta", "support": 1, + "support_callers": ["Manta"]}] + S["carrier"]["data"]["carriers"] = [{"gene": genes[1], "variant": "c.100C>T", "genotype": "het", + "significance": "Pathogenic", "stars": 2, "inheritance": "AR"}] + return change + + +def second_hit(S): + """A second ClinVar row in the gene of the son's planted secondary finding, the card's counts kept.""" + S["clinvar"]["data"]["hits"] = S["clinvar"]["data"]["hits"] + [dict( + [h for h in S["clinvar"]["data"]["hits"] if h.get("gene") == "KCNH2"][0], pos=150970000)] + + +# Pairs of son's pages that differ only in which result is closed (or in what only a closed result names): the +# page before any click must be the same, byte for byte. The two sides of each pair differ in the thing it guards: +# a gene that a fixed-panel card's own finding names (the medicine card's CYP2D6 and NAT2) against one no card names, +# a gene of the carrier list's two-hit line (SMN1) against one it does not name, a gene of the callability card's +# list of genes not on this reference (GSTT1) or of hard genes not covered (NBPF1) against one not on it, a gene +# spelled like a word or unit the report prints (HR, MB) against one that is not, and one guessed address on two +# pages where it is a closed result's id on one side only, loaded with the page and typed after it. +GUESS = "#f/cpsr-brca1-c-68-69del" +INFER = (("two SNV results in different genes (BRCA1, BRCA2)", cpsr("BRCA1"), cpsr("BRCA2")), + ("a deletion over BRCA1 and NBR2 against one over CFTR and CTTNBP2", + sv_over("chr17:43044295-43144295", ["BRCA1", "NBR2"]), sv_over("chr7:117480025-117580025", ["CFTR", "CTTNBP2"])), + ("SMN1 against HBA1, the same copy numbers", cpsr("SMN1"), cpsr("HBA1")), + ("a result with a place on the map (BRCA2) against one without (MLH1)", cpsr("BRCA2"), cpsr("MLH1")), + ("a closed result with one ClinVar row against two", lambda S: None, second_hit), + ("a closed result in a gene of a known deep intronic site (ATM) against one in none (BRCA2)", with_sites("ATM"), + with_sites("BRCA2")), + ("a link to each of two closed results (#f/)", cpsr("BRCA1"), cpsr("BRCA2"), + ("#f/cpsr-brca1-c-68-69del", "#f/cpsr-brca2-c-68-69del")), + ("a gene of the medicine card's own finding (CYP2D6) against the other one (NAT2)", cpsr("CYP2D6"), cpsr("NAT2")), + ("a gene of the medicine card's own finding (CYP2D6) against one no card names (BRCA2)", cpsr("CYP2D6"), + cpsr("BRCA2")), + ("a gene the carrier list's two-hit line names (SMN1) against one it does not (BRCA2)", cpsr("SMN1"), + cpsr("BRCA2")), + ("a gene not on this reference (GSTT1) against one on it (BRCA2)", cpsr("GSTT1"), cpsr("BRCA2")), + ("a hard gene the reads did not cover (NBPF1) against one they did (BRCA2)", cpsr("NBPF1"), cpsr("BRCA2")), + ("a gene spelled like a word of the report (HR) against one that is not (BRCA2)", cpsr("HR"), + cpsr("BRCA2")), + ("a gene spelled like a unit of the report (MB) against one that is not (BRCA2)", cpsr("MB"), cpsr("BRCA2")), + ("one guessed address, a closed result's id on one page only (BRCA1 against BRCA2 closed)", cpsr("BRCA1"), + cpsr("BRCA2"), (GUESS, GUESS)), + ("the same guessed address typed after the page drew (BRCA1 against BRCA2 closed)", cpsr("BRCA1"), + cpsr("BRCA2"), (GUESS, GUESS), True)) + + +def infer_problems(b, base, out, og, cases=INFER): + """[problems]: for each pair of CASES, the son's page before any click differs (its HTML, an attribute, the + address or the map's canvas), with the number of differing lines and the first ones.""" + bad = [] + for what, one, two, *more in cases: + got = [] + for change, h in zip((one, two), (more or [("", "")])[0]): + keep = plant_summary(out, og, change) + try: + got.append(page_now(b, base, h, typed=more[1:] == [True])) + finally: + unplant(keep) + (h1, a1, c1), (h2, a2, c2) = got + diff = [x for x in difflib.unified_diff(h1, h2, lineterm="", n=0) if x[:1] in "+-" and x[:3] not in ("+++", "---")] + if diff or a1 != a2 or c1 != c2: + bad.append(f"{what}: {len(diff)} lines differ, address {a1 == a2}, canvas {c1 == c2}: {diff[:6]}") + return bad + + +def sv_gene_problems(b, base, out, og): + """[problems]: with a closed deletion over BRCA1 and NBR2, the page names NBR2 nowhere (a carrier row and its + finding name only NBR2) before any click.""" + keep = plant_summary(out, og, sv_over("chr17:43044295-43144295", ["BRCA1", "NBR2"])) + try: + html = "\n".join(page_now(b, base)[0]) + finally: + unplant(keep) + return [f"NBR2 on the page: ...{html[max(0, html.find('NBR2') - 120):html.find('NBR2') + 40]}..."] if "NBR2" in html else [] + + +def panel_words_problems(b, base, out, og): + """[problems]: with a closed result in CYP2D6, a gene the medicine card's own finding names, the medicine card shows + the fixed-panel box's words, never 'This section holds a result'.""" + keep = plant_summary(out, og, cpsr("CYP2D6")) + try: + page_now(b, base) + text = b.eval("(document.querySelector('#card-cpic .card-text') || {}).textContent || ''") + finally: + unplant(keep) + ok = "This section lists the same genes for everyone" in text and "This section holds" not in text + return [] if ok else [f"the medicine card says {text[:160]!r}"] + + +ADDRESS_JS = ("window.__seen = window.__seen || []; if (!window.__heard) { window.__heard = 1; ['hashchange', 'popstate']" + ".forEach(function (e) { window.addEventListener(e, function () { window.__seen.push(e + ' ' + location.hash); }); " + "}); } true") + + +def address_problems(b, base, out): + """[problems]: on the son's page, a result opened from its card and the reader's moves after it never put the + id of a result his page can close into the address or a history entry: #f/closed-1, Show the result, a link to + section 3, Back.""" + with open(os.path.join(out, "runs", OTHERS[1], "results.json")) as fh: + res = json.load(fh) + tags = (res.get("veil") or {}).get("findings") or {} + ids = [f["id"] for f in res["findings"] if f["level"] == "act" or f["id"] in tags] + seen = [] + b.go("about:blank") + b.wait("location.href === 'about:blank'") + b.go(f"{base}/runs/{OTHERS[1]}/#f/closed-1") + b.wait("document.documentElement.dataset.pgpResults && document.documentElement.dataset.pgpStatus === 'ready' && " + "document.querySelector('.finding[data-level=act] .veil button') && 'ready'", 15) + b.eval(ADDRESS_JS) + seen.append(b.eval("location.hash")) + b.eval("document.querySelector('.finding[data-level=act] .veil button').click(); true") + time.sleep(0.3) + seen.append(b.eval("location.hash")) + b.eval("location.hash = '#s3'; true") + time.sleep(0.3) + b.eval("history.back(); true") + time.sleep(0.5) + seen += [b.eval("location.hash")] + json.loads(b.eval("JSON.stringify(window.__seen)") or "[]") + bad = [h for h in seen if any(i in (h or "") for i in ids)] + if not str(seen[1]).startswith("#f/opened-"): + bad.append(f"the opened result's address: {seen[1]!r}, want #f/opened-") + return [f"addresses {seen}: {bad}"] if bad else [] + + +def home_now(b, base, out, change): + """The home page with the son's view (runs.json, live_results.overview) built from his summary with CHANGE applied: + the whole document.""" + p = os.path.join(out, "runs.json") + with open(p, "rb") as fh: + keep = fh.read() + with open(os.path.join(os.path.dirname(out), "root", OTHERS[1], "summary.json")) as fh: + summary = json.load(fh) + change(summary.setdefault("sections", {})) + idx = json.loads(keep) + idx["runs"] = [dict(r, **live_results.overview(summary)) if r["name"] == OTHERS[1] else r for r in idx["runs"]] + try: + with open(p, "w") as fh: + json.dump(idx, fh) + b.go("about:blank") + b.wait("location.href === 'about:blank'") + b.go(f"{base}/") + b.wait("document.documentElement.dataset.pgpStatus === 'ready' && document.documentElement.dataset.pgpPeople " + "&& document.querySelector('#attention li') && 'ready'", 15) + time.sleep(0.3) + return b.eval("document.documentElement.outerHTML").replace("><", ">\n<").split("\n") + finally: + with open(p, "wb") as fh: + fh.write(keep) + + +def renamed(gene): + """The son's summary with his planted gene (KCNH2) named GENE everywhere: his first result to act on, the one the + home page lists, is then in GENE.""" + def change(S): + S.update(json.loads(json.dumps(S).replace("KCNH2", gene))) + return change + + +HOME = (("his first result in BRCA1 against BRCA2", renamed("BRCA1"), renamed("BRCA2")), + ("a gene of the medicine card's own finding (CYP2D6) against BRCA2", cpsr("CYP2D6"), cpsr("BRCA2"))) + + +def home_problems(b, base, out, cases=HOME): + """[problems]: the home page is the same whichever of the son's results is closed (his tile and the list of what + needs attention: no link to a closed result's own page, whose address names its gene).""" + bad = [] + for what, one, two in cases: + h1, h2 = home_now(b, base, out, one), home_now(b, base, out, two) + diff = [x for x in difflib.unified_diff(h1, h2, lineterm="", n=0) if x[:1] in "+-" and x[:3] not in ("+++", "---")] + if diff: + bad.append(f"home, {what}: {len(diff)} lines differ: {diff[:4]}") + return bad + + +def infer_checks(b, base, out): + """The son's page is a function of his open results and the count of his closed ones: two pages that differ only + in which result is closed are the same before any click (INFER), and a closed deletion's second gene is named + nowhere. Controls: person.js that puts a closed card's id back, that draws a fixed-panel card's open rows, or + that counts the closed rows; a veil that tags only a deletion's first gene.""" + og = og_binary() or build_live_demo.OG + if not os.path.isfile(os.path.join(out, "runs", OTHERS[1], "map.json")): + print("[SKIP] infer checks: the demo has no map.json") + return + t0 = time.time() + bad = infer_problems(b, base, out, og) + check("browser: two pages of the son that differ only in which result is closed are the same before any click, " + "HTML, attributes, address and map canvas (" + "; ".join(x[0] for x in INFER) + ")", not bad, bad) + bad = sv_gene_problems(b, base, out, og) + check("browser: a closed deletion over two genes closes the rows that name only its second gene", not bad, bad) + bad = panel_words_problems(b, base, out, og) + check("browser: a fixed-panel card whose own finding names the closed result's gene shows the panel's words", not bad, + bad) + bad = address_problems(b, base, out) + check("browser: opening a closed result and moving on never puts its id in the address or a history entry", not bad, + bad) + bad = home_problems(b, base, out) + check("browser: the home page is the same whichever of the son's results is closed (" + "; ".join(x[0] for x in HOME) + + ")", not bad, bad) + home = os.path.join(out, "assets", "home.js") + with open(home) as fh: + keep_home = fh.read() + a = "x.r.url + (closedRuns[x.r.name] && x.f.level === 'act' ? '' : 'finding/' + encodeURIComponent(x.f.id) + '/')" + try: + if a not in keep_home: + check("control: home.js that links a closed result's own page (the planted line is in home.js)", False, a) + else: + with open(home, "w") as fh: + fh.write(keep_home.replace(a, "x.r.url + 'finding/' + encodeURIComponent(x.f.id) + '/'")) + control("home.js that links a closed result's own page", not home_problems(b, base, out, HOME[:1])) + finally: + with open(home, "w") as fh: + fh.write(keep_home) + first = live_results.panel_finding + try: + live_results.panel_finding = lambda f: False + control("a veil that tags a fixed-panel card's own finding by the closed result's gene", + not infer_problems(b, base, out, og, [x for x in INFER if "(NAT2)" in x[0]])) + finally: + live_results.panel_finding = first + fixed = live_results.FIXED_LINES + try: + live_results.FIXED_LINES = {k: v for k, v in fixed.items() if k != "carrier-list"} + control("a veil that tags the carrier list's two-hit line by its genes", + not infer_problems(b, base, out, og, [x for x in INFER if "two-hit" in x[0]])) + finally: + live_results.FIXED_LINES = fixed + js = os.path.join(out, "assets", "person.js") + with open(js) as fh: + keep_js = fh.read() + try: + for desc, a, z, case in ( + # The guard is the pair of lines: either alone lands every guess the same way. + ("person.js that picks a closed card by a guessed id", + " if (f && !(veiled(f) && !m)) { return f; }\n return closedOrder().length ? findingById(closedOrder()[0]) : null;", + " return f || null;", [x for x in INFER if x[0].startswith("the same guessed")][0]), + ("person.js that picks a closed card by a guessed id, at load", + " if (f && !(veiled(f) && !m)) { return f; }\n return closedOrder().length ? findingById(closedOrder()[0]) : null;", + " return f || null;", [x for x in INFER if x[0].startswith("one guessed")][0]), + ("person.js that puts a closed card's id back", + "function handle(f) { return veiled(f) ? 'closed-' + (closedOrder().indexOf(f.id) + 1) : f.id; }", + "function handle(f) { return f.id; }", INFER[0]), + ("person.js that draws a fixed-panel card's open rows", "} else if (panelShut(c)) {", "} else if (false) {", + INFER[2]), + ("person.js that counts the closed rows", + "'Some of its rows or numbers stay closed with a result to act on, until that result is opened.'", + "(ot[1] + (c.lines || []).length - lines.length) + ' rows stay closed with a result to act on.'", INFER[4])): + if a not in keep_js: + check(f"control: {desc} (the planted line is in person.js)", False, a) + continue + with open(js, "w") as fh: + fh.write(keep_js.replace(a, z)) + control(desc, not infer_problems(b, base, out, og, [case])) + for desc, a, z, still in ( + ("person.js whose fixed-panel card says 'This section holds a result' first", "} else if (panelShut(c)) {", + "} else if (panelShut(c) && !closedFs.length) {", lambda: not panel_words_problems(b, base, out, og)), + ("person.js that writes an opened result's id into the address", + "function addr(f) { return !closable(f) ? f.id : veiled(f) ? handle(f) : 'opened-' + (openedOrder().indexOf(f.id) + 1); }", + "function addr(f) { return veiled(f) ? handle(f) : f.id; }", lambda: not address_problems(b, base, out))): + if a not in keep_js: + check(f"control: {desc} (the planted line is in person.js)", False, a) + continue + with open(js, "w") as fh: + fh.write(keep_js.replace(a, z)) + control(desc, still()) + finally: + with open(js, "w") as fh: + fh.write(keep_js) + first = live_results.closable + try: + live_results.closable = lambda found: first([{k: v for k, v in f.items() if k != "genes"} for f in found]) + control("a veil that tags only a deletion's first gene", not sv_gene_problems(b, base, out, og)) + finally: + live_results.closable = first + print(f" infer checks: {time.time() - t0:.0f} s") + + def widget_veil_checks(b, base, out): - """On the son's page a CHIP call and a copy-number dot in the gene of a closed result stay out of sight until that - result is opened; the other dot stays drawn. A CHIP call is also a finding of its card that names its gene, so - the CHIP card closes whole with the result; SMN1 at two copies is no finding, so only its dot's tag closes it.""" + """On the son's page the copy-number card and the CHIP card, whose rows are a fixed panel (the same genes for + everyone), close whole while any result of his to act on is closed: no dot, no call, no fact of theirs shows, + and their line on a phone is the same words whatever the result; so do the ClinVar card's lines for the known + deep intronic sites, a fixed panel inside it. Each "Show the result" of their box opens one result; once every + result is opened every dot, call, fact and deep intronic line is back.""" no_cache(b) rj = os.path.join(out, "runs", OTHERS[1], "results.json") js = os.path.join(out, "assets", "person.js") @@ -1416,12 +1837,11 @@ def drawn(): b.wait("location.href === 'about:blank'") b.go(f"{base}/runs/{OTHERS[1]}/") b.wait("document.documentElement.dataset.pgpResults && document.documentElement.dataset.pgpStatus === 'ready' " - "&& document.querySelector('[data-widget=copies]') && document.querySelector('.veil') && 'drawn'", 15) + "&& document.querySelector('#card-paralogs') && document.querySelector('.veil') && 'drawn'", 15) return shown() def phone_line(): - """The copy-number card's one line on a phone (its fold's summary, after the card's own title, the same for - everyone), drawn from its first open fact.""" + """The copy-number card's one line on a phone (its fold's summary, after the card's own title).""" b.call("Emulation.setDeviceMetricsOverride", {"width": 390, "height": 844, "deviceScaleFactor": 2, "mobile": False}, b.session) try: @@ -1431,26 +1851,29 @@ def phone_line(): return b.wait("document.documentElement.dataset.pgpStatus === 'ready' && document.documentElement.dataset.pgpResults " "&& (document.querySelector('[data-card=fold-paralogs] summary .what') || {}).textContent", 15) or "" finally: - b.call("Emulation.clearDeviceMetricsOverride", {}, b.session) + b.call("Emulation.setDeviceMetricsOverride", {"width": 1440, "height": 900, "deviceScaleFactor": 1, "mobile": False}, + b.session) def phone_problems(): line = phone_line() - return [] if line and "SMN1" not in line else [f"the copy-number card's line on a phone: {line!r}"] + want = "The same genes for everyone, closed while a result about a child is closed." + return [] if line == want else [f"the copy-number card's line on a phone: {line!r}, want {want!r}"] def shown(): - return b.eval("JSON.stringify([" + SHOWN_JS + ", (document.querySelector('[data-widget=copies]') || {}).textContent || ''])") + return b.eval("JSON.stringify([" + SHOWN_JS + ", (document.querySelector('[data-widget=copies]') || {}).textContent || '', " + "document.querySelectorAll('.finding[data-level=act] .veil').length])") def problems(text, opened): - page, cops = json.loads(text) if text else ["", ""] + page, cops, _ = json.loads(text) if text else ["", "", 0] out_ = [] - if "SMN2" not in cops: - out_.append(f"the open dot is not drawn: {cops[:120]}") - if ("p.R175H" in page) != opened or ("SMN1" in cops) != opened: - out_.append(f"the CHIP call or the SMN1 dot of the closed results {'not ' if opened else ''}shown: " - f"{'p.R175H' in page} | {cops[:200]}") - # The copy-number card's facts name SMN1: closed in its margin and in its line of section 8. - if ("SMN1 copies" in page) != opened: - out_.append(f"the card's fact on SMN1 {'not ' if opened else ''}shown") + deep = page.count(live_results.FIXED_LINES["clinvar"][0]) + if opened and not ("SMN1" in cops and "SMN2" in cops and "p.R175H" in page and "SMN1 copies" in page + and deep == sites[0]): + out_.append(f"every result opened, yet a dot, the call, the fact or a deep intronic line ({deep} of {sites[0]}) is " + f"missing: {cops[:200]}") + if not opened and (cops or "p.R175H" in page or "SMN1 copies" in page or deep): + out_.append(f"a closed result, yet the copy-number or CHIP card shows its rows or the ClinVar card " + f"{deep} deep intronic lines: {cops[:200]}") return out_ try: with open(os.path.join(os.path.dirname(out), "root", OTHERS[1], "summary.json")) as f: @@ -1463,6 +1886,7 @@ def problems(text, opened): {"gene": g, "protein_change": ch, "vaf": v, "alt_reads": 6, "depth": 75, "status": "pass", "note": "a clone"} for g, ch, v in (("DNMT3A", "p.R882H", 0.12), ("TP53", "p.R175H", 0.08))], "genes": [{"gene": g, "detection_limit_vaf": 0.05, "mean_depth": 75, "status": "ok"} for g in ("DNMT3A", "TP53")]}} + sites = [deep_sites(S)] S["paralogs"] = {"state": "ok", "data": {"genes": [{"gene": g, "copy_number": n, "status": "called", "detail": "."} for g, n in (("SMN1", 2), ("SMN2", 2))]}} res = live_results.build(OTHERS[1], summary) @@ -1470,19 +1894,25 @@ def problems(text, opened): json.dump(res, f) bad = problems(drawn(), False) + phone_problems() if not bad: - for _ in range(2): # the CPSR card holds both results: one click opens one - b.eval("document.querySelector('#card-cpsr .veil button').click(); true") - time.sleep(0.5) - bad = problems(shown(), True) - check("browser: on the son's page a CHIP call, a copy-number dot and a card's fact in the gene of a closed " - "result stay out of sight (the fact in the card's line on a phone too) until it is opened", not bad, bad) + # The copy-number card's box opens one result per click, until none is left closed. + n = json.loads(shown())[2] + for k in range(n): + b.eval("var v = document.querySelector('#card-paralogs .veil button'); v && v.click(); true") + time.sleep(0.3) + left = json.loads(shown())[2] + if left != n - k - 1: + bad.append(f"click {k + 1} on the copy-number card's box left {left} of {n} results closed") + break + bad += problems(shown(), True) + check("browser: on the son's page the copy-number and CHIP cards, a fixed panel, close whole while a result is " + "closed (their line on a phone too), and so do the ClinVar card's deep intronic lines; the box opens one " + "result per click, and every dot, call, fact and line is back once every result is opened", not bad, bad) for desc, a, z in ( - ("person.js that draws every item", "add(text, widget(of[1]));", "add(text, widget(c.widget));"), - ("person.js that shows every fact", "facts = (c.facts || []).filter(function (f, i) { return !(ft[i] && tagClosed(ft[i])); });", - "facts = c.facts || [];"), - ("person.js whose card line reads every fact", - "(openFacts(c)[0][0] ? openFacts(c)[0][0][0] + ': ' + openFacts(c)[0][0][1] : NOT_ALL_CLEAR)", - "(c.facts && c.facts[0] ? c.facts[0][0] + ': ' + c.facts[0][1] : NOT_ALL_CLEAR)")): + ("person.js that draws a fixed-panel card's open rows", "} else if (panelShut(c)) {", "} else if (false) {"), + ("person.js whose card line reads a fixed-panel card's facts", ": g === 'show' && panelShut(c) ? PANEL_LINE", + ": false ? PANEL_LINE"), + ("person.js that draws a fixed panel's lines", "if (id === PANEL) { return closedPage() && closedActs().length > 0; }", + "if (id === PANEL) { return false; }")): if a not in keep_js: check(f"control: {desc} (the planted line is in person.js)", False, a) continue @@ -2053,19 +2483,37 @@ def map_now(b): # The page's closed results to act on and the map's chips, read at once (a chip's button rebuilds the chips at -# once: nothing to wait for): the closed act cards in page order, the act chip's count, whether its confirm step +# once: nothing to wait for): the act cards in page order (ids: each one's id, its handle while it is closed) and +# the places of the closed ones among them (closed: a closed card names no id, so a result is known by its place), the act chip's count, whether its confirm step # shows, where the keyboard is (first: the chips' first button; canvas: the map), and the opened chips. CHIP_JS = r"""JSON.stringify((function () { - var ids = []; - Array.prototype.forEach.call(document.querySelectorAll('.finding[data-level=act]'), function (a) { - if (a.querySelector('.veil') && ids.indexOf(a.dataset.finding) < 0) { ids.push(a.dataset.finding); } }); + var ids = [], closed = []; + Array.prototype.forEach.call(document.querySelectorAll('.finding[data-level=act]'), function (a, i) { + ids.push(a.dataset.finding); + if (a.querySelector('.veil')) { closed.push(i); } }); var c = document.querySelector('.gm-chip[data-closed=act]'), f = document.activeElement; var first = document.querySelector('.gm-chips button'); - return { closed: ids, buttons: !!first, chip: c ? c.firstChild.textContent : '', confirm: !!(c && c.querySelector('.gm-confirm')), + return { closed: closed, ids: ids, buttons: !!first, chip: c ? c.firstChild.textContent : '', confirm: !!(c && c.querySelector('.gm-confirm')), focus: f && f.classList.contains('gm-yes') ? 'gm-yes' : f === first ? 'first' : f === document.querySelector('.gm-canvas') ? 'canvas' : (f ? f.tagName + '.' + f.className : ''), opened: Array.prototype.map.call(document.querySelectorAll('.gm-chip.opened'), function (o) { var a = o.querySelector('a'); - return { kind: o.dataset.opened, text: o.textContent, href: a ? a.getAttribute('href') : '' }; }) }; + var raw = a ? a.getAttribute('href') : ''; + return { kind: o.dataset.opened, text: o.textContent, raw: raw, href: opened(raw) }; }) }; + // The finding an "Opened in its section" link names: #f/opened- is the n-th finding the page can close and + // shows opened, in the order of the sections (person.js openedOrder()). + function opened(raw) { + var m = /^#f\/opened-(\d+)$/.exec(raw); + if (!m) { return raw; } + if (!window.__pgpResults) { + var x = new XMLHttpRequest(); x.open('GET', 'results.json', false); x.send(); window.__pgpResults = JSON.parse(x.responseText); + } + var r = window.__pgpResults, tags = (r.veil && r.veil.findings) || {}, byId = {}, order = []; + r.findings.forEach(function (f) { byId[f.id] = f; }); + function ok(f) { return f && (f.level === 'act' || (tags[f.id] && tags[f.id] !== f.id)) && !!document.getElementById('f-' + f.id); } + (r.journal || []).forEach(function (s) { (s.findings || []).forEach(function (id) { if (ok(byId[id]) && order.indexOf(id) < 0) { order.push(id); } }); }); + r.findings.forEach(function (f) { if (ok(f) && order.indexOf(f.id) < 0) { order.push(f.id); } }); + return '#f/' + (order[Number(m[1]) - 1] || raw); + } }()))""" @@ -2079,8 +2527,8 @@ def closed_words(n): def chip_open(b): - """The act chip's Open, then its Show the result, as a keyboard reader does it. Returns (the id it opened, - problems): one Open opens exactly one of the page's closed results, the count falls by one, the keyboard + """The act chip's Open, then its Show the result, as a keyboard reader does it. Returns (the id it opened, its + place among the act cards, problems): one Open opens exactly one of the page's closed results, the count falls by one, the keyboard goes to Show the result and then to the chips' first button (the map when no chip is left), and an "Opened in its section" chip never names a result that is still closed.""" before = chips_now(b) @@ -2101,10 +2549,18 @@ def chip_open(b): bad.append(f"Show the result: the chip reads {after['chip']!r}, want {closed_words(n)!r}") if after["focus"] != ("first" if after["buttons"] else "canvas"): bad.append(f"Show the result: the keyboard is on {after['focus']!r}") - named = [o["href"] for o in after["opened"] if o["kind"] == "section" and o["href"][3:] in after["closed"]] + named = [o["href"] for o in after["opened"] if o["kind"] == "section" and o["href"][3:] not in after["ids"]] if named: - bad.append(f"an Opened in its section chip names a closed result: {named}") - return (gone[0] if len(gone) == 1 else None), bad + bad.append(f"an Opened in its section chip names a result that is not open: {named}") + raw = [o["raw"] for o in after["opened"] if o["kind"] == "section" and not re.match(r"^#f/opened-\d+$", o["raw"])] + if raw: + bad.append(f"an Opened in its section chip links a result by its id: {raw}") + leaked = [after["ids"][i] for i in after["closed"] if not re.match(r"^closed-\d+$", after["ids"][i] or "")] + if leaked: + bad.append(f"a closed card carries an id, not a handle: {leaked}") + if len(gone) != 1: + return None, None, bad + return after["ids"][gone[0]], gone[0], bad def parity_problems(b, base, rows): @@ -2219,12 +2675,15 @@ def leaks(c, but=""): # Open opens the page's closed results one per click, in the page's order, so the son's own result with no # place (his first closed card) opens before the deletion: the map then names it only in its section chip. order = chips_now(b)["closed"] + if not want: + return bad + ["the son's page shows no closed result to open"] + k, at = 0, None for k in range(want): - got, more = chip_open(b) + got, at, more = chip_open(b) if more: return bad + [f"Open {k + 1}: {x}" for x in more] - if got != order[k]: - return bad + [f"Open {k + 1} opened {got}, not the page's next closed result {order[k]} ({order})"] + if at != order[k]: + return bad + [f"Open {k + 1} opened {got}, the act card at {at}, not the page's next closed one {order[k]} ({order})"] if got == fid: break section = [o for o in chips_now(b)["opened"] if o["kind"] == "section"] @@ -2252,10 +2711,10 @@ def leaks(c, but=""): bad.append(f"Close it again on the section chips: chips {chips_now(b)['opened']}, closed {chips_now(b)['closed']}") b.eval("document.querySelector('.gm-chip.opened[data-opened=\\'1\\'] button').click(); true") c = map_now(b) - card = b.eval("!!document.querySelector('#f-%s .veil')" % fid) + card = at in chips_now(b)["closed"] if leaks(c) or not card: bad.append(f"Close it again: map shows {leaks(c)}, card closed {card}") - b.eval("document.querySelector('#f-%s .veil button').click(); true" % fid) + b.eval("document.querySelectorAll('.finding[data-level=act]')[%d].querySelector('.veil button').click(); true" % at) c = map_now(b) if [m["id"] for m in c["marks"] if m["id"] == fid] != [fid]: bad.append("the card's Show the result does not open it on the map") @@ -2314,7 +2773,8 @@ def overlap_problems(b, base, out): c = map_page(b, base, son, "#" + f["chrom"][i]) if c["state"] != "drawn": return [f"the planted map: {c['state']}"] - b.eval("document.querySelector('#f-%s .veil button').click(); true" % fid) + if not open_closed(b, fid): + return [f"a link to {fid} lands on no closed card"] c = map_now(b) bad = [] if fid not in [m["id"] for m in c["marks"]]: @@ -2410,8 +2870,13 @@ def act(fid, gene, viewer, title): [act("pl-iota", "PLIOTA", "chr7:117400000-117700000", "Planted iota deletion")], [{"id": "pl-kappa", "level": "review", "card": "exomiser", "gene": "CFTR", "title": "Planted kappa review", "detail": "CFTR", "text": "A planted review finding that names CFTR only."}], "pl-kappa", "chr7"), - ("the page's own CYP2D6 medicine finding over a closed variant naming another gene", - [act("pl-theta", "PLTHETA", "chr22:42125000", "Planted theta result")], [], "pgx-cyp2d6", "chr22")) + # The page's own CYP2D6 medicine finding closes with the medicine card while any result is closed (a fixed + # panel), so a carrier finding stands in for a gene-only finding of a card that is no panel. + ("a gene-only carrier finding (CYP2D6) over a closed variant naming another gene", + [act("pl-theta", "PLTHETA", "chr22:42125000", "Planted theta result")], + [{"id": "pl-lambda", "level": "carrier", "card": "carrier-list", "gene": "CYP2D6", + "title": "Planted lambda carrier", "detail": "CYP2D6", "text": "A planted carrier finding that names CYP2D6 only."}], + "pl-lambda", "chr22")) def under_problems(b, base, out, og): @@ -2457,7 +2922,8 @@ def second_open_problems(b, base, out, og): ids = [m["id"] for m in c["marks"]] if "pl-alpha" in ids or "pl-beta" in ids: return [f"closed: marks {ids}"] - b.eval("document.querySelector('#f-pl-beta .veil button').click(); true") + if not open_closed(b, "pl-beta"): + return ["a link to pl-beta lands on no closed card"] c = map_now(b) ids = [m["id"] for m in c["marks"]] bad = [] @@ -2519,7 +2985,8 @@ def named_problems(b, base, out, og): if fig != before: bad.append(f"closed: the figure differs from the page without them " f"({[k for k in ('html', 'canvas') if fig.get(k) != before.get(k)]})") - b.eval("document.querySelector('#f-pl-x .veil button').click(); true") + if not open_closed(b, "pl-x"): + return bad + ["a link to pl-x lands on no closed card"] c = map_now(b) ids = [m["id"] for m in c["marks"]] if not {"pl-x", "pl-q", "pl-r"} <= set(ids): @@ -2581,7 +3048,7 @@ def open_order_problems(b, base, out, og): f"keyboard on {kept['focus']!r}"] seq = [] for _ in first["closed"]: - got, more = chip_open(b) + got, _, more = chip_open(b) if more: return bad + [f"{what}, Open {len(seq) + 1}: {x}" for x in more] seq.append(got) @@ -2592,10 +3059,14 @@ def open_order_problems(b, base, out, og): seqs[what] = seq if k == 0: gone = [x["id"] for x in extra if not x["viewer"]][0] - b.eval("var o = document.querySelector('.gm-chip.opened[data-opened=section] a[href=\"#f/%s\"]'); " - "o.parentNode.parentNode.querySelector('button').click(); true" % gone) + at = chips_now(b)["ids"].index(gone) if gone in chips_now(b)["ids"] else -1 + was = [o for o in chips_now(b)["opened"] if o["kind"] == "section"] + raw = [o["raw"] for o in was if o["href"] == f"#f/{gone}"] + b.eval("var o = document.querySelector('.gm-chip.opened[data-opened=section] a[href=\"%s\"]'); " + "o.parentNode.parentNode.querySelector('button').click(); true" % (raw[0] if raw else "")) after = chips_now(b) - if after["closed"] != [gone] or any(o["kind"] == "section" and gone in o["href"] for o in after["opened"]): + left = [o for o in after["opened"] if o["kind"] == "section"] + if after["closed"] != [at] or len(left) != len(was) - 1: bad.append(f"{what}: Close it again on {gone}'s section chip: closed {after['closed']}, " f"opened chips {after['opened']}") finally: diff --git a/tests/test_live_results.py b/tests/test_live_results.py index f5bec452..2562d3ab 100644 --- a/tests/test_live_results.py +++ b/tests/test_live_results.py @@ -36,9 +36,10 @@ link of the planted hit, its genome-chart tick and its review-star bar carry the secondary finding's id, the deletion's span carries its own, the search index's items that name the gene carry one of them, and nothing else is - tagged; DEMO-001, with no result to act on, has no tag; and a known deep - intronic site's line in the gene of a closed secondary finding carries - that finding's id (the card prints each site whether hit or not), a gene + tagged; DEMO-001, with no result to act on, has no tag; and every known + deep intronic site's line carries the fixed-panel tag, never the id of a + closed result in its gene (the card prints each site whether hit or not, + so one missing line would name the gene), a gene named before a full stop is that gene, and a ClinVar row closed by its gene alone closes its "See the reads at" link with it; every other card's row that names the gene (the son's symptom ranking ranks it) carries one of @@ -48,12 +49,16 @@ first ones and its closed lead names none; a CHIP call, a copy-number dot and a fact that name a closed result's gene carry its id (the CHIP chart's panel of genes never does), and results.json carries the closed results as - veil.spans (the family page closes a child's places while one is closed). + veil.spans (the family page closes a child's places while one is closed); + a closed structural variant over two genes closes the rows and the finding + that name only its second gene, and veil.panels lists the cards of a fixed + panel (live_results.FIXED_PANELS). Each check runs again on a planted mistake and must fail there: 1 with VNTYPER's entry dropped, 2 with one card's caveat blanked, 3 with "ok" planted in a card and "All clear" in a script, 4 with the hits emptied, 5 with the results route registered without the login, 6 with a row printed as its Python dict, 7 with the Slivar adapter returning nothing, -8 with a veil rule that tags nothing (rows, lines, facts and drawings) and one that tags every row. A check +8 with a veil rule that tags nothing (rows, lines, facts and drawings), one that tags every row and one that tags +only a structural variant's first gene. A check that cannot fail is no check. Run: python3 tests/test_live_results.py @@ -389,7 +394,9 @@ def veil_problems(res, idx): out.append(f"{cid} rows tagged wrong: {bad or got}") exo = (v.get("rows") or {}).get("exomiser") or [] ranked = [f["id"] for f in res["findings"] if f["level"] != "act" and gene in f["title"]] - if sf not in exo or not ranked or v.get("findings") != {i: sf for i in ranked}: + # A fixed-panel card's own findings (a medicine gene) carry the panel tag, whatever the closed result. + named = {k: t for k, t in (v.get("findings") or {}).items() if t != live_results.PANEL} + if sf not in exo or not ranked or named != {i: sf for i in ranked}: out.append(f"the symptom ranking of {gene}: rows tagged {exo}, findings tagged {v.get('findings')}, want {ranked} -> {sf}") tagged = {(t, it["section"]): it.get("veil") for t, items in idx["terms"].items() for it in items if it.get("veil")} named = {(gene, it["section"]) for it in idx["terms"].get(gene) or []} @@ -400,7 +407,8 @@ def veil_problems(res, idx): def veil_lines_problems(): """What is wrong with the veil's tags of the ClinVar card's lines: a known deep intronic site is printed - whether hit or not, so its line names the gene of a closed secondary finding in that gene.""" + whether hit or not, a fixed panel, so every such line carries the panel tag (live_results.PANEL), the one in + a closed secondary finding's gene and the others alike: a line closed alone would name the gene.""" S = {"clinvar": {"state": "ok", "data": {"hits": [], "deep_intronic": [ {"gene": "GLA", "hgvs": "NM_000169.3:c.640-801G>A", "chrom": "chrX", "pos": 101399747, "ref": "C", "alt": "T", "call": "hom", "detail": "GQ 40"}, @@ -409,9 +417,9 @@ def veil_lines_problems(): lines = live_results.a_clinvar(S, {"sections": S}).get("lines") or [] found = [{"id": "sf-gla-test", "level": "act", "gene": "GLA", "detail": "GLA"}] v = live_results.veil({"sections": S}, found, [{"sections": [{"id": "clinvar", "lines": lines}]}]) - want = ["sf-gla-test" if "GLA " in ln else None for ln in lines] + want = [live_results.PANEL if ln.startswith(live_results.FIXED_LINES["clinvar"]) else None for ln in lines] got = (v.get("lines") or {}).get("clinvar") - out = [] if "sf-gla-test" in want and None in want and got == want else [f"lines {lines} tagged {got}, want {want}"] + out = [] if want.count(live_results.PANEL) == 2 and got == want else [f"lines {lines} tagged {got}, want {want}"] # A gene named at the end of a sentence ("in GLA.") is that gene. if live_results.veil_id(live_results.closable(found), "A change in GLA.") != "sf-gla-test": out.append("'A change in GLA.' carries no tag") @@ -464,6 +472,120 @@ def widget_problems(): return out +def sv_genes_problems(): + """What is wrong with the veil's tags of a closed structural variant over two genes: a ClinVar row and a carrier + row that name only its second gene (the hit sits elsewhere, so only the gene can tie it) carry its id, and so + does the carrier finding; veil.spans lists it once per gene; veil.panels lists the cards of a fixed panel.""" + S = {"annotsv": {"state": "ok", "data": {"rows": [ + {"sv": "chr17:43044295-43170245", "type": "deletion", "length": -125950, "acmg_class": "5", + "genes": ["BRCA1", "NBR2"], "gt": "0/1", "support": 2, "support_callers": ["Manta", "Delly"]}]}}, + "clinvar": {"state": "ok", "data": {"hits": [ + {"gene": "NBR2", "chrom": "chr1", "pos": 5, "ref": "A", "alt": "G", "significance": "Pathogenic"}, + {"gene": "GENE5", "chrom": "chr1", "pos": 9, "ref": "A", "alt": "G", "significance": "Pathogenic"}]}}, + "carrier": {"state": "ok", "data": {"genes_on_list": 2, "carriers": [ + {"gene": "NBR2", "variant": "c.1A>G", "genotype": "het", "significance": "Pathogenic"}]}}, + "paralogs": {"state": "ok", "data": {"genes": [{"gene": "SMN1", "copy_number": 2, "status": "called"}]}}} + res = live_results.build("T", {"sections": S}) + v = res.get("veil") or {} + sv = [f["id"] for f in res["findings"] if f["card"] == "structural-variants"] + car = [f["id"] for f in res["findings"] if f["card"] == "carrier-list"] + out = [] + if len(sv) != 1 or len(car) != 1: + return [f"the planted findings: {[f['id'] for f in res['findings']]}"] + if (v.get("rows") or {}).get("clinvar") != [sv[0], None]: + out.append(f"ClinVar rows tagged {(v.get('rows') or {}).get('clinvar')}, want the deletion's id on the NBR2 row") + if (v.get("rows") or {}).get("carrier-list") != [sv[0]]: + out.append(f"carrier rows tagged {(v.get('rows') or {}).get('carrier-list')}") + if (v.get("findings") or {}).get(car[0]) != sv[0]: + out.append(f"the NBR2 carrier finding tagged {(v.get('findings') or {}).get(car[0])}") + if sorted(s[1] for s in v.get("spans") or [] if s[0] == sv[0]) != ["BRCA1", "NBR2"]: + out.append(f"spans {v.get('spans')}") + if v.get("panels") != ["paralogs"]: + out.append(f"panels {v.get('panels')}, want the copy-number card, the only fixed panel here") + return out + + +# The veil's tables, pinned here so that an entry dropped from one turns this file red (each is checked by what +# veil() does with it, not by reading the table back): the fixed-panel cards, the fixed lines of a card (a sample +# line or row naming a gene, its card, and whether it is a row), the cards that name no gene, and the words that +# name no gene though a gene has their name. +WANT_PANELS = ("mei", "expansions", "vntyper", "lpa", "chip", "paralogs", "cpic", "cyp2d6", "hla", "locityper", + "traits-panel", "bloodtype") +WANT_LINES = (("clinvar", "Known deep intronic site (always reported): GLA c.640-801G>A chrX:1 C>T: hom", False), + ("carrier-list", "Two hits across callers (a deletion plus a variant on the other copy): none found; " + "deletions read from Parascopy's SMN1 count (step 35)", False), + ("callability", ["GSTT1", "not on this reference", ".", "ALT-only"], True), + ("callability", ["NBPF1", "not read", "69%", "segdup, collapsed"], True)) +WANT_NO_GENE = (("prs", ["Asthma", "PGS002727", "68.3 (EUR)", "2605/2872", "HR 1.57 (EUR, PPM014751)"], "HR"),) +WANT_NOT_A_GENE = (("SET", "the same set of genes"), ("MB", "0.0 MB in runs of 5 MB or more"), + ("HP", "Seizure (HP:0001250)"), ("GC", "reads per million with 48-52% GC")) +WANT_A_GENE = (("KCNH2", "A change in KCNH2."), ("C9ORF72", "C9orf72 repeat"), ("HLA-A", "HLA-A*02:01")) + + +def gene_of(text): + """The first upper-case word of TEXT (a list is a table row) that looks like a gene symbol.""" + t = " ".join(map(str, text)) if isinstance(text, list) else text + return next(w for w in re.findall(r"[A-Z][A-Z0-9]{2,}", t.split(":", 1)[-1] if not isinstance(text, list) else t) + if w not in ("ALT", "SMN")) + + +def table_problems(): + """What is wrong with veil()'s tables (WANT_*): a fixed-panel card missing from veil.panels, a fixed line or row + tagged by the gene it names instead of the panel tag, a card that names no gene tagged, a word that names no gene + read as one, or a gene symbol not read as one.""" + out = [] + found = lambda g: [{"id": "sf-" + g.lower(), "level": "act", "gene": g, "detail": g}] + chapters = [{"sections": [{"id": c, "state": "ok"} for c in WANT_PANELS]}] + got = live_results.veil({"sections": {}}, found("BRCA2"), chapters).get("panels") or [] + out += [f"card {c} is not in veil.panels" for c in WANT_PANELS if c not in got] + for card, item, is_row in WANT_LINES: + g = gene_of(item) + sec = {"id": card, "table": {"head": ["a", "b", "c", "d"], "rows": [item]}} if is_row else {"id": card, "lines": [item]} + v = live_results.veil({"sections": {}}, found(g), [{"sections": [sec]}]) + tag = ((v.get("rows" if is_row else "lines") or {}).get(card) or [None])[0] + if tag != live_results.PANEL: + out.append(f"{card}: {str(item)[:60]} tagged {tag} with a closed result in {g}, want the panel tag") + for card, row, g in WANT_NO_GENE: + sec = {"id": card, "table": {"head": ["a"] * len(row), "rows": [row]}, "lines": [" ".join(row)], "facts": [row[:2]]} + v = live_results.veil({"sections": {}}, found(g), [{"sections": [sec]}]) + if any((v.get(k) or {}).get(card) for k in ("rows", "lines", "facts")): + out.append(f"{card} names no gene, yet a closed result in {g} tags it: {v}") + for g, text in WANT_NOT_A_GENE: + if live_results.veil_id(live_results.closable(found(g)), text): + out.append(f"{text!r} read as naming {g}") + for g, text in WANT_A_GENE: + if not live_results.veil_id(live_results.closable(found(g)), text): + out.append(f"{text!r} not read as naming {g}") + return out + + +def table_controls(): + """Each entry of the veil's tables dropped in turn: table_problems() must fail every time.""" + keep = (live_results.FIXED_PANELS, live_results.FIXED_LINES, live_results.NAMES_NO_GENE, live_results.UNIT_RE) + first_words = live_results.gene_words + try: + for c in WANT_PANELS: + live_results.FIXED_PANELS = {k: v for k, v in keep[0].items() if k != c} + control(f"FIXED_PANELS without {c}", not table_problems()) + live_results.FIXED_PANELS = keep[0] + for card, words in keep[1].items(): + for w in words: + live_results.FIXED_LINES = dict(keep[1], **{card: tuple(x for x in words if x != w)}) + control(f"FIXED_LINES[{card}] without {w!r}", not table_problems()) + live_results.FIXED_LINES = keep[1] + live_results.NAMES_NO_GENE = {} + control("NAMES_NO_GENE empty", not table_problems()) + live_results.NAMES_NO_GENE = keep[2] + live_results.UNIT_RE = re.compile(r"(?!)") + control("gene_words with no units", not table_problems()) + live_results.UNIT_RE = keep[3] + live_results.gene_words = lambda t: {w.upper().rstrip("._-") for w in re.findall(r"[A-Za-z0-9][A-Za-z0-9._-]*", str(t))} + control("gene_words that reads every word, in any case, as a gene", not table_problems()) + finally: + (live_results.FIXED_PANELS, live_results.FIXED_LINES, live_results.NAMES_NO_GENE, live_results.UNIT_RE) = keep + live_results.gene_words = first_words + + def overview_problems(): """What is wrong with the home page's view of a child's run (overview): a finding that names a closed result (a symptom ranking of its gene) is listed among the first findings, or the closed lead still names it.""" @@ -575,15 +697,34 @@ def tagged(sample_dir): r3["_summary"] = s3 return r3, results_index.build(s3, sample_dir, rr.SECTION_OF) bad = veil_lines_problems() - check("8. a known deep intronic site's line in a closed secondary finding's gene carries its id, a gene " + check("8. every known deep intronic site's line carries the fixed-panel tag, not a closed result's id, a gene " "before a full stop is that gene, a ClinVar row closed by its gene closes its reads link, and a lead " "segment naming the gene carries its id", not bad, bad) + fixed = live_results.FIXED_LINES + try: # the card still prints its lines; veil() no longer knows they are a panel + live_results.FIXED_LINES = type("Unknown", (dict,), {"get": lambda self, k, d=None: None})(fixed) + control("a veil that tags a known deep intronic site's line by its gene", not veil_lines_problems()) + finally: + live_results.FIXED_LINES = fixed + bad = table_problems() + check("8. every fixed-panel card, fixed line and gene-free card of the veil's tables closes as a panel or never, " + "and only a gene symbol written as one names a gene", not bad, bad) + table_controls() bad = overview_problems() check("8. a child's home tile lists no finding that names a closed result, and its closed lead names none", not bad, bad) bad = widget_problems() check("8. a CHIP call, a copy-number dot and a fact that name a closed result's gene carry its id, the CHIP " "chart's gene panel none, and results.json carries the closed results as veil.spans", not bad, bad) + bad = sv_genes_problems() + check("8. a closed structural variant over two genes closes the rows and the finding that name only its second " + "gene, veil.spans lists it per gene, and veil.panels lists the fixed-panel cards", not bad, bad) + first = live_results.closable + try: + live_results.closable = lambda found: first([{k: v for k, v in f.items() if k != "genes"} for f in found]) + control("a veil that tags only a structural variant's first gene", not sv_genes_problems()) + finally: + live_results.closable = first bad = veil_problems(*tagged(d3)) check("8. the planted ClinVar hit's row, viewer link, tick and star bar carry its secondary finding's id, the " "deletion's span its own, every other card's row of the gene and the symptom ranking's finding one of them, "