Repository navigation
perf(analysis): one walk of the syntax tree per script, indexed by node type and command name #17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,79 @@ | ||
| # AST helpers shared by the rules. Type names are compared as strings so the module loads on | ||
| # Windows PowerShell 5.1, where the PowerShell 7 node types (ternary, pipeline chain) don't exist. | ||
|
|
||
| # One walk per tree, shared by every rule that asks about it: the nodes grouped by type name and | ||
| # the commands grouped by name. Fifteen rules each walking the tree, some of them once per command | ||
| # name they look for, was most of the time an analysis took. The table is keyed on the AST object | ||
| # itself, so a sub-tree a caller passes gets an index of its own and a tree that goes out of scope | ||
| # takes its index with it. | ||
| $script:IslAstIndex = [System.Runtime.CompilerServices.ConditionalWeakTable[object, object]]::new() | ||
|
|
||
| # The walk and the grouping stay in .NET: FindAll calls its predicate once per node, and a script | ||
| # block there costs more than the walk itself; BlockingCollection.TryAdd is a Func<Ast, bool> that | ||
| # is always true and keeps the nodes in the order they were visited. Object.GetType as an open | ||
| # delegate is the key selector for Enumerable.ToLookup, which keeps each group in source order. | ||
| $script:IslAstNodeType = [System.Management.Automation.Language.Ast] | ||
| $script:IslAstCollectorType = | ||
| [System.Collections.Concurrent.BlockingCollection[System.Management.Automation.Language.Ast]] | ||
| $script:IslAstGetType = [System.Delegate]::CreateDelegate( | ||
| [System.Func[System.Management.Automation.Language.Ast, type]], [object].GetMethod('GetType')) | ||
| $script:IslAstToLookup = ([System.Linq.Enumerable].GetMethods() | | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Reflection because Windows PowerShell 5.1 has no syntax for calling a generic method with explicit type arguments; the MethodInfo is resolved once at load and reused. |
||
| Where-Object { $_.Name -eq 'ToLookup' -and $_.GetParameters().Count -eq 2 } | | ||
| Select-Object -First 1).MakeGenericMethod($script:IslAstNodeType, [type]) | ||
|
|
||
| function Get-IslAstIndex { | ||
| <# | ||
| .SYNOPSIS | ||
| The node and command index of an AST, built on first use and kept for the tree's lifetime. | ||
| #> | ||
| [CmdletBinding()] | ||
| [OutputType('IntuneScriptLab.AstIndex')] | ||
| param( | ||
| [Parameter(Mandatory)] | ||
| [System.Management.Automation.Language.Ast]$Ast | ||
| ) | ||
| $index = $null | ||
| if ($script:IslAstIndex.TryGetValue($Ast, [ref]$index)) { return $index } | ||
|
|
||
| # FindAll visits the tree in document order, the root first, and that order is what the groups keep | ||
| $collector = $script:IslAstCollectorType::new() | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. BlockingCollection is the first .NET collection whose add method returns bool and keeps insertion order (ConcurrentQueue underneath). HashSet.Add also returns true but its enumeration order is not guaranteed. The ordering was checked against FindAll on both hosts for a 767-node tree. |
||
| $collect = [System.Delegate]::CreateDelegate([System.Func[System.Management.Automation.Language.Ast, bool]], | ||
| $collector, $script:IslAstCollectorType.GetMethod('TryAdd', [type[]]@($script:IslAstNodeType))) | ||
| $null = $Ast.FindAll($collect, $true) | ||
| $lookup = $script:IslAstToLookup.Invoke($null, @([object]$collector.ToArray(), $script:IslAstGetType)) | ||
|
|
||
| # Keyed by the type's short name, the way the rules ask: a 7-only node type is a key that is | ||
| # never there on 5.1, not a type that fails to resolve | ||
| $byType = [System.Collections.Generic.Dictionary[string, object]]::new([System.StringComparer]::Ordinal) | ||
| foreach ($group in $lookup) { $byType[$group.Key.Name] = $group } | ||
| $byCommand = [System.Collections.Generic.Dictionary[string, System.Collections.Generic.List[object]]]::new( | ||
| [System.StringComparer]::OrdinalIgnoreCase) | ||
| if ($byType.ContainsKey('CommandAst')) { | ||
| foreach ($command in $byType['CommandAst']) { | ||
| # A command whose name is not a constant (& $tool, "$prefix-Item") has no name to index | ||
| $commandName = $command.GetCommandName() | ||
| if (-not $commandName) { continue } | ||
| if (-not $byCommand.ContainsKey($commandName)) { | ||
| $byCommand[$commandName] = [System.Collections.Generic.List[object]]::new() | ||
| } | ||
| $byCommand[$commandName].Add($command) | ||
| } | ||
| } | ||
| $index = [pscustomobject]@{ | ||
| PSTypeName = 'IntuneScriptLab.AstIndex' | ||
| ByType = $byType | ||
| ByCommand = $byCommand | ||
| } | ||
| $script:IslAstIndex.Add($Ast, $index) | ||
| $index | ||
| } | ||
|
|
||
| function Find-IslAstNode { | ||
| <# | ||
| .SYNOPSIS | ||
| Finds AST nodes by type name, optionally filtered by a predicate. | ||
| #> | ||
| [CmdletBinding()] | ||
| # The parameters are used inside the FindAll predicate; the analyzer can't see through it | ||
| [Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSReviewUnusedParameter', '')] | ||
| param( | ||
| [Parameter(Mandatory)] | ||
| [System.Management.Automation.Language.Ast]$Ast, | ||
|
|
@@ -19,12 +84,15 @@ function Find-IslAstNode { | |
|
|
||
| [scriptblock]$Where | ||
| ) | ||
| $Ast.FindAll({ | ||
| param($node) | ||
| if ($node.GetType().Name -notin $TypeName) { return $false } | ||
| if ($Where) { return [bool](& $Where $node) } | ||
| $true | ||
| }, $true) | ||
| $index = Get-IslAstIndex -Ast $Ast | ||
| $nodes = foreach ($name in $TypeName) { | ||
| if ($index.ByType.ContainsKey($name)) { $index.ByType[$name] } | ||
| } | ||
| # Each type's list is in document order; several types are merged back into it | ||
| if ($TypeName.Count -gt 1) { $nodes = $nodes | Sort-Object -Property { $_.Extent.StartOffset } } | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Only the three callers that ask for several types pay for the sort; a single type comes straight from its group in document order, which is the common case. |
||
| foreach ($node in $nodes) { | ||
| if (-not $Where -or [bool](& $Where $node)) { $node } | ||
| } | ||
| } | ||
|
|
||
| function Find-IslCommand { | ||
|
|
@@ -33,19 +101,19 @@ function Find-IslCommand { | |
| Finds command invocations by name (case-insensitive), including aliases the caller lists. | ||
| #> | ||
| [CmdletBinding()] | ||
| [Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSReviewUnusedParameter', '')] | ||
| param( | ||
| [Parameter(Mandatory)] | ||
| [System.Management.Automation.Language.Ast]$Ast, | ||
|
|
||
| [Parameter(Mandatory)] | ||
| [string[]]$Name | ||
| ) | ||
| Find-IslAstNode -Ast $Ast -TypeName CommandAst -Where { | ||
| param($node) | ||
| $commandName = $node.GetCommandName() | ||
| $commandName -and $commandName -in $Name | ||
| $index = Get-IslAstIndex -Ast $Ast | ||
| $commands = foreach ($commandName in $Name) { | ||
| if ($index.ByCommand.ContainsKey($commandName)) { $index.ByCommand[$commandName] } | ||
| } | ||
| if ($Name.Count -gt 1) { $commands = $commands | Sort-Object -Property { $_.Extent.StartOffset } } | ||
| $commands | ||
| } | ||
|
|
||
| function Test-IslCommandParameter { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A ConditionalWeakTable rather than a hashtable so the index does not keep a tree alive: Test-IntuneScript parses thousands of scripts in a CI run and each tree would otherwise stay in memory with its index for the life of the module.