Skip to content

#289 Implementation of Search-IshDocumentObj parameter -SemanticSearch #290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ddemeyer wants to merge 1 commit into
base: master
Choose a base branch
from
Open

#289 Implementation of Search-IshDocumentObj parameter -SemanticSearch #290

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/instructions/source-cmdlets--pester.instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,23 @@ test that needs a new piece of tenant data, add a defaulted variable to `ISHRemo
(so CI/other tenants can override it) rather than hard-coding a value. **Never commit
`ISHRemote.PesterSetup.Debug.ps1`** — it holds credentials.

## Gating tests on the server version — inside the `It`, never in `-Skip`
Pester evaluates `-Skip:` expressions during the **Discovery** phase, before any `BeforeAll` has
run, so `$ishSession` is not initialized yet and a `-Skip:($ishSession...)` expression is silently
wrong. Check the server version **inside** the `It` block, where `$ishSession` exists, and wrap the
version-specific body in an `if`, as `AddIshBackgroundTask.Tests.ps1` does:

```powershell
It "Parameter Count" {
if (([Version]$ishSession.ServerVersion).Major -gt 15 -or (([Version]$ishSession.ServerVersion).Major -eq 15 -and ([Version]$ishSession.ServerVersion).Minor -ge 4)) { # 15.4.0 and up
# version-specific assertions
}
}
```

`-Skip:` remains fine for conditions known at Discovery time, such as
`-Skip:($PSVersionTable.PSVersion.Major -lt 7)`.

## Build test data per test, clean it up per test
Each file is responsible for **creating its own data and removing all of it**, leaving the tenant in
its original state:
Expand Down
3 changes: 3 additions & 0 deletions .github/instructions/source-codereview-pester.instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,9 @@ Authoring detail lives in
`ISHRemote.PesterSetup.ps1` — they must be runnable without a live server.
- [ ] **PS 7+ only tests** guard with `-Skip:($PSVersionTable.PSVersion.Major -lt 7)` on the
`It` or `Context` block.
- [ ] **Server-version gates** are an `if` on `[Version]$ishSession.ServerVersion` **inside** the
`It`, never `-Skip:($ishSession...)` — `-Skip:` is evaluated at Discovery, before `$ishSession`
exists.
- [ ] **Script / MCP tests** that mock collaborators use `Mock -ModuleName ISHRemote <FunctionName>
{ ... }` scoped to the module, not a global mock.

Expand Down
1 change: 1 addition & 0 deletions Doc/ReleaseNotes-ISHRemote-8.3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ The below text describes the delta compared to fielded release ISHRemote v8.2.
* Fixed `New-IshSession` (protocol `WcfSoapWithOpenIdConnect`, PowerShell 7.2+/.NET 6.0+) throwing `FileLoadException: Could not load file or assembly 'Microsoft.IdentityModel.Tokens, Version=8.14.0.0, ...'. The located assembly's manifest definition does not match the assembly reference.` on machines where a different build of `Microsoft.IdentityModel.Tokens` (and related `Duende.IdentityModel.OidcClient`) is registered in the Global Assembly Cache (GAC) — observed on machines with Microsoft Intune Management Extension installed. `AppDomainModuleAssemblyInitializer` now force-loads ISHRemote's own bundled copies of `Duende.IdentityModel`, `Duende.IdentityModel.OidcClient`, `Microsoft.IdentityModel.Abstractions/.Logging/.Tokens/.Tokens.Saml/.Xml` as early as possible during module import, and `SessionCmdlet.BeginProcessing` now reports the full forced list over `-Verbose`. Root cause for the `Duende.IdentityModel.OidcClient` variant: `InfoShareOpenIdConnectSystemBrowser` was `public` and implemented `Duende.IdentityModel.OidcClient.Browser.IBrowser`, which put it in `Trisoft.ISHRemote.dll`'s exported types, forcing PowerShell's own binary-module cmdlet discovery (`Assembly.GetExportedTypes()`) to resolve `Duende.IdentityModel.OidcClient` before `IModuleAssemblyInitializer.OnImport()` ever ran — see Breaking Changes - Code. A new `TestPrerequisite.Tests.ps1` check asserts no assembly is ever loaded from the GAC on PowerShell Core. See #272. Thanks @ddemeyer
* Fixed cmdlets over protocol `WcfSoapWithOpenIdConnect` occasionally throwing `An unsecured or incorrectly secured fault was received from the other party` on the first SOAP call after a channel fault, requiring the user to re-run the same cmdlet for it to succeed (the existing #201/#219 rebuild-on-next-call logic only kicked in on a second, separate call). Each `Get*25Channel()` method in `InfoShareWcfSoapWithOpenIdConnectConnection` now returns the channel wrapped in a new `RetryOnFaultProxy<T>` (`System.Reflection.DispatchProxy`) that catches `CommunicationException`/`FaultException` on the actual SOAP call, rebuilds the channel via the existing rebuild logic, and retries exactly once within the same cmdlet invocation before propagating any further failure — with the original exception type/stack trace preserved. Requires a new `net48`-only NuGet dependency, `System.Reflection.DispatchProxy` (built into the BCL on `net6.0`/`net10.0`). See #273. Thanks @ddemeyer
* Package bumps, deliberate adding of `System.Security.Cryptography.Xml` to fix vulnerability coming from transitive `System.ServiceModel.Primitives`. See #278 and #279. Thanks @ddemeyer
* Augmented `Search-IshDocumentObj` with parameter `-SemanticSearch` as shorthand for a semantic vector search, next to the existing `-SimpleQuery` keyword search. It generates the same default `<ishquery>` but targets the `ISHSEMANTICINCONTENT` field instead of `ISHANYWHERE`, so `Search-IshDocumentObj -SemanticSearch "replace oil in vehicle" -MaxHitsToReturn 50` is all you need. Requires Tridion Docs 15.4.0 and up. See #289. Thanks @ddemeyer



Expand Down
35 changes: 35 additions & 0 deletions Source/ISHRemote/Trisoft.ISHRemote/Cmdlets/DocumentObj/SearchIshDocumentObj.Tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,41 @@ Describe "Search-IshDocumentObjData" -Tags "Read" {
$ishObject[0].fishstatustype -ge 0 | Should-Be $false # Basic does not return FISHSTATUSTYPE but extra RequestedMetadata parameter does
}
}
Context "Search-IshDocumentObj SemanticSearchGroup" {
It "Parameter IshSession invalid" {
{ Search-IshDocumentObj -IShSession "INVALIDISHSESSION" -SemanticSearch "cordless or wireless" } | Should-Throw
}
It "Parameter MaxHitsToReturn invalid" {
{ Search-IshDocumentObj -MaxHitsToReturn "INVALIDMAXHITSTORETURN" -SemanticSearch "cordless or wireless" } | Should-Throw
}
It "Parameter SemanticSearch throws before 15.4.0" {
if (([Version]$ishSession.ServerVersion).Major -lt 15 -or (([Version]$ishSession.ServerVersion).Major -eq 15 -and ([Version]$ishSession.ServerVersion).Minor -lt 4)) {
{ Search-IshDocumentObj -IshSession $ishSession -SemanticSearch "cordless or wireless" } | Should-Throw
}
}
It "Parameter MaxHitsToReturn" {
if (([Version]$ishSession.ServerVersion).Major -gt 15 -or (([Version]$ishSession.ServerVersion).Major -eq 15 -and ([Version]$ishSession.ServerVersion).Minor -ge 4)) {
$ishObject = Search-IshDocumentObj -IshSession $ishSession -MaxHitsToReturn 3 -SemanticSearch "cordless or wireless"
($ishObject.Count -le 3) | Should-Be $true
}
}
It "Parameter RequestedMetadata" {
if (([Version]$ishSession.ServerVersion).Major -gt 15 -or (([Version]$ishSession.ServerVersion).Major -eq 15 -and ([Version]$ishSession.ServerVersion).Minor -ge 4)) {
$requestedMetadata = Set-IshRequestedMetadataField -Level Lng -Name FISHSTATUSTYPE
$ishObject = Search-IshDocumentObj -IshSession $ishSession -MaxHitsToReturn 2 -SemanticSearch "cordless or wireless" -RequestedMetadata $requestedMetadata
if ($ishObject) {
$ishObject[0].IshRef | Should-NotBeNull
$ishObject[0].fishstatustype -ge 0 | Should-Be $true
}
}
}
It "Parameter Count" {
if (([Version]$ishSession.ServerVersion).Major -gt 15 -or (([Version]$ishSession.ServerVersion).Major -eq 15 -and ([Version]$ishSession.ServerVersion).Minor -ge 4)) {
$count = Search-IshDocumentObj -IshSession $ishSession -SemanticSearch "cordless or wireless" -Count
$count -ge 0 | Should-Be $true
}
}
}
Context "Search-IshDocumentObj XmlQueryQueryGroup" {
BeforeAll {
$xmlQueryLatestVersion = @"
Expand Down
66 changes: 58 additions & 8 deletions Source/ISHRemote/Trisoft.ISHRemote/Cmdlets/DocumentObj/SearchIshDocumentObj.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,14 @@ namespace Trisoft.ISHRemote.Cmdlets.DocumentObj
/// <example>
/// <code>
/// $ishSession = New-IshSession -WsBaseUrl "https://example.com/ISHWS/"
/// Search-IshDocumentObj -SemanticSearch "cordless or wireless" -MaxHitsToReturn 50
/// </code>
/// <para>Executes a Full-Text-Index semantic vector search for 'cordless or wireless' in the SEMANTICINCONTENT field of the LatestVersion collection (compared to AllVersion), with a filter on the default object types and only in the user's language. Requires Tridion Docs 15.4.0 and up, otherwise a PlatformNotSupportedException is thrown.</para>
/// <para>Results are returned sorted by the Full-Text-Index engine, first by score then by title.</para>
/// </example>
/// <example>
/// <code>
/// $ishSession = New-IshSession -WsBaseUrl "https://example.com/ISHWS/"
/// Search-IshDocumentObj -SimpleQuery "bluetooth" -Count
/// </code>
/// <para>Executes a Full-Text-Index search for 'bluetooth' in the ANY field of the LatestVersion collection (compared to AllVersion), with no filter on object types and only in the user's language.</para>
Expand Down Expand Up @@ -99,7 +107,7 @@ namespace Trisoft.ISHRemote.Cmdlets.DocumentObj
/// $ishSession = New-IshSession -WsBaseUrl "https://example.com/ISHWS/"
/// $xmlQuery = @"
/// &lt;ishquery&gt;
/// &lt;and&gt;&lt;ishfield name='ISHSEMANTICINCONTENT' level='none' ishoperator='contains'&gt;replace oil in vehicle&lt;/ishfield&gt;&lt;/and&gt;
/// &lt;and&gt;&lt;ishfield name='ISHSEMANTICINCONTENT' level='none' ishoperator='contains'&gt;cordless or wireless&lt;/ishfield&gt;&lt;/and&gt;
/// &lt;ishsort&gt;
/// &lt;ishsortfield name='ISHSCORE' level='none' ishorder='d'/&gt;
/// &lt;ishsortfield name='FTITLE' level='logical' ishorder='d'/&gt;
Expand All @@ -118,7 +126,7 @@ namespace Trisoft.ISHRemote.Cmdlets.DocumentObj
/// $requestedMetadata = Set-IshRequestedMetadataField -Level Lng -Name FISHSTATUSTYPE
/// Search-IshDocumentObj -XmlQuery $xmlQuery -MaxHitsToReturn 100 -RequestedMetadata $requestedMetadata
/// </code>
/// <para>Executes a Full-Text-Index semantic search for 'replace oil in vehicle' in the SEMANTICINCONTENT field of the LatestVersion collection (compared to AllVersion), filtered on these object types and language; results are returned sorted by the Full-Text-Index engine, and enriched with FISHSTATUSTYPE field on top of the $ishSession.DefaultRequestedMetadata</para>
/// <para>Executes a Full-Text-Index semantic search for 'cordless or wireless' in the SEMANTICINCONTENT field of the LatestVersion collection (compared to AllVersion), filtered on these object types and language; results are returned sorted by the Full-Text-Index engine, and enriched with FISHSTATUSTYPE field on top of the $ishSession.DefaultRequestedMetadata</para>
/// <para>The provided $xmlQuery is the default query of parameter set lead by -SimpleQuery; where the ISHANYWHERE is overwritten with the given -SimpleQuery value, and the ishlanguagefilter is overwritten with the user's language.</para>
/// </example>
/// <example>
Expand Down Expand Up @@ -155,15 +163,17 @@ public sealed class SearchIshDocumentObj : DocumentObjCmdlet
/// <summary>
/// <para type="description">The IshSession variable holds the authentication and contract information. This object can be initialized using the New-IshSession cmdlet.</para>
/// </summary>
[Parameter(Mandatory =false, ValueFromPipelineByPropertyName = false, ParameterSetName = "XmlQueryGroup")]
[Parameter(Mandatory =false, ValueFromPipelineByPropertyName = false, ParameterSetName = "SimpleQueryGroup")]
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "XmlQueryGroup")]
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "XmlQueryCountGroup")]
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "SimpleQueryGroup")]
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "SimpleQueryCountGroup")]
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "SemanticSearchGroup")]
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "SemanticSearchCountGroup")]
[ValidateNotNullOrEmpty]
public IshSession IshSession { get; set; }

/// <summary>
/// <para type="description">The RAW xml ishquery used by Search25 API</para>
/// <para type="description">The keyword search text, for example 'bluetooth' or 'red AND green AND blue', that is placed in the ISHANYWHERE field (term search in content and metadata) of a default ishquery that is passed to the Search25 API.</para>
/// </summary>
[Parameter(Mandatory = true, ValueFromPipelineByPropertyName = false, ParameterSetName = "SimpleQueryGroup")]
[Parameter(Mandatory = true, ValueFromPipelineByPropertyName = false, ParameterSetName = "SimpleQueryCountGroup")]
Expand All @@ -174,6 +184,18 @@ public string SimpleQuery
set { _simpleQuery = value; }
}

/// <summary>
/// <para type="description">The natural language search text, for example 'cordless or wireless', that is placed in the ISHSEMANTICINCONTENT field (semantic vector search in content) of a default ishquery that is passed to the Search25 API. Requires server-side semantic search, so Tridion Docs 15.4.0 and up.</para>
/// </summary>
[Parameter(Mandatory = true, ValueFromPipelineByPropertyName = false, ParameterSetName = "SemanticSearchGroup")]
[Parameter(Mandatory = true, ValueFromPipelineByPropertyName = false, ParameterSetName = "SemanticSearchCountGroup")]
[ValidateNotNullOrEmpty]
public string SemanticSearch
{
get { return _semanticSearch; }
set { _semanticSearch = value; }
}

/// <summary>
/// <para type="description">The RAW xml ishquery used by Search25 API</para>
/// </summary>
Expand All @@ -191,6 +213,7 @@ public string XmlQuery
/// </summary>
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "XmlQueryGroup")]
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "SimpleQueryGroup")]
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "SemanticSearchGroup")]
[ValidateNotNullOrEmpty]
public long MaxHitsToReturn
{
Expand All @@ -203,6 +226,7 @@ public long MaxHitsToReturn
/// </summary>
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "XmlQueryGroup")]
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "SimpleQueryGroup")]
[Parameter(Mandatory = false, ValueFromPipelineByPropertyName = false, ParameterSetName = "SemanticSearchGroup")]
[ValidateNotNullOrEmpty]
public IshField[] RequestedMetadata { get; set; }

Expand All @@ -211,6 +235,7 @@ public long MaxHitsToReturn
/// </summary>
[Parameter(Mandatory = true, ValueFromPipelineByPropertyName = false, ParameterSetName = "XmlQueryCountGroup")]
[Parameter(Mandatory = true, ValueFromPipelineByPropertyName = false, ParameterSetName = "SimpleQueryCountGroup")]
[Parameter(Mandatory = true, ValueFromPipelineByPropertyName = false, ParameterSetName = "SemanticSearchCountGroup")]
[ValidateNotNullOrEmpty]
public SwitchParameter Count
{
Expand All @@ -225,6 +250,11 @@ public SwitchParameter Count
/// </summary>
private string _simpleQuery = "";

/// <summary>
/// Private field to store the semantic ISHSEMANTICINCONTENT query to later construct the XmlQuery
/// </summary>
private string _semanticSearch = "";

/// <summary>
/// Private field to store the incoming &gt;ishquery&lt; that is litteraly passed into Search25 API.
/// </summary>
Expand All @@ -247,6 +277,12 @@ protected override void BeginProcessing()
if (IshSession == null) { IshSession = (IshSession)SessionState.PSVariable.GetValue(ISHRemoteSessionStateGlobalIshSession); }
if (IshSession == null) { throw new ArgumentException(ISHRemoteSessionStateIshSessionException); }
WriteDebug($"Using IshSession[{IshSession.Name}] from SessionState.{ISHRemoteSessionStateIshSession} or in turn SessionState.{ISHRemoteSessionStateGlobalIshSession}");

if ((ParameterSetName == "SemanticSearchGroup" || ParameterSetName == "SemanticSearchCountGroup")
&& ((IshSession.ServerIshVersion.MajorVersion < 15) || ((IshSession.ServerIshVersion.MajorVersion == 15) && (IshSession.ServerIshVersion.MinorVersion < 4))))
{
throw new PlatformNotSupportedException($"Search-IshDocumentObj with -SemanticSearch requires server-side semantic search which is only available starting from 15.4.0 and up. ServerIshVersion[{IshSession.ServerVersion}]");
}
base.BeginProcessing();
}

Expand All @@ -262,9 +298,11 @@ protected override void ProcessRecord()
{
case "SimpleQueryGroup":
case "SimpleQueryCountGroup":
case "SemanticSearchGroup":
case "SemanticSearchCountGroup":
WriteDebug("Loading default query");
_xmlQuery = @"<ishquery>
<and><ishfield name='ISHANYWHERE' level='none' ishoperator='contains'>change oil filter</ishfield></and>
<and><ishfield name='ISHANYWHERE' level='none' ishoperator='contains'></ishfield></and>
<ishsort>
<ishsortfield name='ISHSCORE' level='none' ishorder='d'/>
<ishsortfield name='FTITLE' level='logical' ishorder='d'/>
Expand All @@ -282,8 +320,18 @@ protected override void ProcessRecord()
XmlDocument xmlDocument = new XmlDocument();
xmlDocument.LoadXml(_xmlQuery);
// Converting type filter, could be something like FolderTypeFilter in Get-IshFolder, not for now
WriteDebug("Setting query string based on SimpleQuery");
xmlDocument.SelectSingleNode("ishquery/and/ishfield").InnerText = SimpleQuery;
XmlNode ishFieldNode = xmlDocument.SelectSingleNode("ishquery/and/ishfield");
if (ParameterSetName.StartsWith("SemanticSearch"))
{
WriteDebug("Setting query string based on SemanticSearch");
ishFieldNode.Attributes["name"].Value = "ISHSEMANTICINCONTENT";
ishFieldNode.InnerText = SemanticSearch;
}
else
{
WriteDebug("Setting query string based on SimpleQuery");
ishFieldNode.InnerText = SimpleQuery;
}
WriteDebug("Setting language based on IshSession");
xmlDocument.SelectSingleNode("ishquery/ishobjectfilters/ishlanguagefilter").InnerText = IshSession.UserLanguage;
WriteDebug("Validating transformed XmlQuery");
Expand Down Expand Up @@ -313,6 +361,7 @@ protected override void ProcessRecord()
switch (ParameterSetName)
{
case "SimpleQueryCountGroup":
case "SemanticSearchCountGroup":
case "XmlQueryCountGroup":
{
// 3A. Executing the count query
Expand All @@ -325,6 +374,7 @@ protected override void ProcessRecord()

case "XmlQueryGroup":
case "SimpleQueryGroup":
case "SemanticSearchGroup":
{
// 3B. Executing the result query
WriteVerbose("Executing a result query");
Expand Down
Loading