diff --git a/README.md b/README.md index 87cf44a90..c7a869910 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ Project Mobius is an Unreal Engine 5.5 application for loading pedestrian trajec ## Why HDF5 For larger trajectory files, convert JSON to HDF5 with the bundled -[`json_to_hdf5_converter.py`](UnrealFolder/ProjectMobius/Plugins/Hdf5DataPlugin/Scripts/json_to_hdf5_converter.py) +[`json_to_hdf5_converter.py`](UnrealFolder/ProjectMobius/Plugins/MobiusDataImporter/Scripts/json_to_hdf5_converter.py) script. In one 5,000-agent sample, HDF5 cut load time from `21.29 s` to `3.18 s` and reduced memory use from `317,555 KB` to `55,283 KB`. @@ -88,6 +88,11 @@ and reduced memory use from `317,555 KB` to `55,283 KB`. More on supported formats, the conversion command, and the benchmark: [docs/DATA-PIPELINE.md](docs/DATA-PIPELINE.md) +To export trajectories from your own evacuation or crowd model, see +[Pedestrian Data Format](docs/PEDESTRIAN-DATA-FORMAT.md) for the exact file layout and the +[HDF5 Export Guide](docs/HDF5-EXPORT-GUIDE.md) for writing it from Python or with the bundled +HDF5 C library ([buildingEXODUS users](docs/EXODUS-EXPORT.md)). + ## Quick Start ### Prerequisites @@ -169,6 +174,9 @@ Full build, packaging, and automation notes live in - [Architecture](docs/ARCHITECTURE.md) - [Data Pipeline](docs/DATA-PIPELINE.md) - [Agent Sample Data](docs/AGENT-SAMPLE-DATA.md) +- [Pedestrian Data Format](docs/PEDESTRIAN-DATA-FORMAT.md) +- [HDF5 Export Guide](docs/HDF5-EXPORT-GUIDE.md) +- [Exporting from buildingEXODUS](docs/EXODUS-EXPORT.md) - [Research and Publications](docs/RESEARCH.md) - [UE Automation Guide](UnrealFolder/ProjectMobius/BuildDocs/UE-Automation-Guide.md) - [Contributing](CONTRIBUTING.md) diff --git a/UnrealFolder/ProjectMobius/Plugins/MobiusDataImporter/Source/ThirdParty/README.md b/UnrealFolder/ProjectMobius/Plugins/MobiusDataImporter/Source/ThirdParty/README.md index 93a661f5a..2c93cba7f 100644 --- a/UnrealFolder/ProjectMobius/Plugins/MobiusDataImporter/Source/ThirdParty/README.md +++ b/UnrealFolder/ProjectMobius/Plugins/MobiusDataImporter/Source/ThirdParty/README.md @@ -15,15 +15,17 @@ ## HDF5 +> **Writing pedestrian data from another simulator?** You can build this HDF5 source on its own, +> without Unreal or the superbuild: see `docs/HDF5-EXPORT-GUIDE.md` at the repository root. + ## Layout - `hdf5-2.0.0/` — full upstream HDF5 2.0.0 source (CMake-only since 2.0; autotools was dropped - upstream). Kept in-tree so the prebuilt libraries below can be rebuilt without hunting down the - exact upstream snapshot. -- `hdf5-2.0.0/install/` — the build output the UE module actually consumes: - - `include/` — public headers, **including the generated `H5pubconf.h`** (deliberately tracked so - a fresh clone compiles without running CMake; it carries harmless machine stamps from whoever - generated it last). + upstream). This source is what git tracks; the libraries below are built from it. +- `hdf5-2.0.0/install/` — the build output the UE module actually consumes. **None of it is in + git** (see `.gitignore`): a fresh clone gets it by running the superbuild + (`python superbuild.py` from `UnrealFolder/ProjectMobius`). + - `include/` — public headers, including the generated `H5pubconf.h`. - `lib/libhdf5.lib`, `lib/libhdf5_hl.lib`, `lib/zlib-static.lib` — static libs, linked by `UeHdf5Library.Build.cs`. - `bin/` — MSVC runtime redistributables (`UeHdf5Library.Build.cs` checks the dir exists; nothing @@ -33,7 +35,7 @@ ## What is intentionally NOT tracked (see repo `.gitignore`) -`install/cmake/`, `install/lib/pkgconfig/`, `install/lib/libhdf5.settings`, `hdf5-2.0.0/build*` — +The whole `install/` tree and `hdf5-2.0.0/build*` — machine-generated at configure/install time and re-stamped with absolute local paths on every `cmake --install`. If they show up as modified/untracked after a rebuild, that is expected. @@ -66,8 +68,7 @@ the Windows one for the Mac libs. Notes: - `ZLIB_USE_LOCALCONTENT=OFF` makes the superbuild download the zlib TGZ at configure time (URLs in `config/CacheURLs.cmake`) — network access required. -- After an install, only `install/include/H5pubconf.h` may show a real tracked diff (path/date - stamps). Revert it unless the rebuild was intentional; commit it together with the new libs when - it was. +- `install/` is regenerated as a whole (headers, including the platform-generated `H5pubconf.h`, + and libs together), so nothing under it needs reverting or committing after a rebuild. - Keep the record layout expectations of the importer in mind: the libs must stay ABI-compatible - with `H5pubconf.h` as committed. + with the `H5pubconf.h` installed beside them. diff --git a/UnrealFolder/ProjectMobius/Scripts/MacBeta/READ ME FIRST - Installing the beta.txt b/UnrealFolder/ProjectMobius/Scripts/MacBeta/READ ME FIRST - Installing the beta.txt index 8fb8009e5..5cac8bc96 100644 --- a/UnrealFolder/ProjectMobius/Scripts/MacBeta/READ ME FIRST - Installing the beta.txt +++ b/UnrealFolder/ProjectMobius/Scripts/MacBeta/READ ME FIRST - Installing the beta.txt @@ -71,10 +71,12 @@ Reporting a problem Please send these two files along with what you were doing: ~/Library/Logs/ProjectMobius/ProjectMobius.log - ~/Library/Logs/ProjectMobius/MobiusCustomLog.txt + ~/Library/Logs/ProjectMobius/MobiusCustomLog_.txt (the newest one) ProjectMobius.log is replaced each time the app starts, so copy it before reopening the app after a problem. Older runs are kept beside it as ProjectMobius-backup-*.log. +MobiusCustomLog_.txt is a new file for each run, named after when that run +started; the last 20 are kept. Licences diff --git a/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Subsystems/MobiusCustomLoggerSubsystem.cpp b/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Subsystems/MobiusCustomLoggerSubsystem.cpp index 497404b78..9c44fd19f 100644 --- a/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Subsystems/MobiusCustomLoggerSubsystem.cpp +++ b/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Subsystems/MobiusCustomLoggerSubsystem.cpp @@ -21,6 +21,48 @@ namespace { return GEngine ? GEngine->GetEngineSubsystem() : nullptr; } + + /** Per-session files are MobiusCustomLog_[_n].txt; the stamp makes name order chronological. */ + const TCHAR* const SessionLogPrefix = TEXT("MobiusCustomLog_"); + + /** Session files kept, the current one included. The pre-rotation MobiusCustomLog.txt never matches. */ + constexpr int32 MaxSessionLogFiles = 20; + + FString MakeSessionLogFilePath(const FString& LogDir, const FDateTime& SessionStart) + { + const FString Stamp = SessionStart.ToString(TEXT("%Y.%m.%d-%H.%M.%S")); + FString Path = FPaths::Combine(LogDir, FString::Printf(TEXT("%s%s.txt"), SessionLogPrefix, *Stamp)); + + // Two processes started in the same second (an editor and a scripted headless run, say) get a file + // each rather than interleaving into one. + for (int32 Suffix = 2; IFileManager::Get().FileExists(*Path); ++Suffix) + { + Path = FPaths::Combine(LogDir, FString::Printf(TEXT("%s%s_%d.txt"), SessionLogPrefix, *Stamp, Suffix)); + } + return Path; + } + + /** Delete the oldest session files beyond MaxSessionLogFiles. Returns how many went. */ + int32 PruneOldSessionLogs(const FString& LogDir, const FString& CurrentLogPath) + { + TArray SessionFiles; + IFileManager::Get().FindFiles(SessionFiles, *FPaths::Combine(LogDir, FString(SessionLogPrefix) + TEXT("*.txt")), + /*Files=*/true, /*Directories=*/false); + SessionFiles.Remove(FPaths::GetCleanFilename(CurrentLogPath)); + SessionFiles.Sort(); + + const int32 ToDelete = SessionFiles.Num() - (MaxSessionLogFiles - 1); + int32 Deleted = 0; + for (int32 Index = 0; Index < ToDelete; ++Index) + { + if (IFileManager::Get().Delete(*FPaths::Combine(LogDir, SessionFiles[Index]), /*RequireExists=*/false, + /*EvenReadOnly=*/false, /*Quiet=*/true)) + { + ++Deleted; + } + } + return Deleted; + } } UMobiusCustomLoggerSubsystem::UMobiusCustomLoggerSubsystem() @@ -45,17 +87,36 @@ void UMobiusCustomLoggerSubsystem::Initialize(FSubsystemCollectionBase& Collecti // Packaged builds expect the log next to the launched executable. const FString LogDir = FPaths::ConvertRelativePathToFull(FPaths::LaunchDir()); #endif - LogFilePath = FPaths::Combine(LogDir, TEXT("MobiusCustomLog.txt")); - + // One file per session (process). A single shared MobiusCustomLog.txt grew without bound (115 MB on the + // dev machine) and interleaved processes running side by side. The newest MaxSessionLogFiles are kept; + // the old shared file is left alone. IPlatformFile& PlatformFile = FPlatformFileManager::Get().GetPlatformFile(); - PlatformFile.CreateDirectoryTree(*FPaths::GetPath(LogFilePath)); + PlatformFile.CreateDirectoryTree(*LogDir); + + const FDateTime SessionStart = FDateTime::Now(); + LogFilePath = MakeSessionLogFilePath(LogDir, SessionStart); + + // Claim the name now, so a process starting a moment later sees it taken. + TUniquePtr Claim(PlatformFile.OpenWrite(*LogFilePath, /*bAppend=*/true)); + Claim.Reset(); + + const int32 PrunedSessionLogs = PruneOldSessionLogs(LogDir, LogFilePath); FlushTickerHandle = FTSTicker::GetCoreTicker().AddTicker( FTickerDelegate::CreateUObject(this, &UMobiusCustomLoggerSubsystem::PumpLogs), 0.25f); // flush 4x per second to keep overhead tiny UE_LOG(LogTemp, Display, TEXT("MobiusCustomLogger initialised. Writing to %s"), *LogFilePath); + + // First line of every file: when this session started, matching the stamp in the file name. + EnqueueLogMessage(FString::Printf(TEXT("=== Mobius session started %s (pid %u) ==="), + *SessionStart.ToString(TEXT("%Y-%m-%d %H:%M:%S")), FPlatformProcess::GetCurrentProcessId())); EnqueueLogMessage(FString::Printf(TEXT("Custom logger initialised. Writing to %s"), *LogFilePath)); + if (PrunedSessionLogs > 0) + { + EnqueueLogMessage(FString::Printf(TEXT("Removed %d old session log(s); keeping the newest %d"), + PrunedSessionLogs, MaxSessionLogFiles)); + } } void UMobiusCustomLoggerSubsystem::Deinitialize() diff --git a/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Subsystems/MobiusUserFeedbackSubsystem.cpp b/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Subsystems/MobiusUserFeedbackSubsystem.cpp index 9a2658a08..19eaeb455 100644 --- a/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Subsystems/MobiusUserFeedbackSubsystem.cpp +++ b/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Subsystems/MobiusUserFeedbackSubsystem.cpp @@ -35,6 +35,7 @@ void UMobiusUserFeedbackSubsystem::Deinitialize() PendingErrors.Reset(); CachedLogLines.Reset(); + CachedLogHead = 0; RecentPromptTimes.Reset(); Super::Deinitialize(); @@ -226,7 +227,24 @@ TArray UMobiusUserFeedbackSubsystem::DrainPendingErrors() TArray UMobiusUserFeedbackSubsystem::GetCachedLogLines() const { - return CachedLogLines; + // Oldest first, which is what the log window's SetLines expects. The head only moves once the ring + // is full, so a zero head means the buffer is still in plain append order. + if (CachedLogHead == 0) + { + return CachedLogLines; + } + + TArray Ordered; + Ordered.Reserve(CachedLogLines.Num()); + for (int32 Index = CachedLogHead; Index < CachedLogLines.Num(); ++Index) + { + Ordered.Add(CachedLogLines[Index]); + } + for (int32 Index = 0; Index < CachedLogHead; ++Index) + { + Ordered.Add(CachedLogLines[Index]); + } + return Ordered; } FOnMobiusErrorReported& UMobiusUserFeedbackSubsystem::OnErrorReported() @@ -310,11 +328,18 @@ void UMobiusUserFeedbackSubsystem::ReportErrorInternal(const FMobiusErrorMessage void UMobiusUserFeedbackSubsystem::HandleLogLine(const FString& Line) { - CachedLogLines.Add(Line); - if (CachedLogLines.Num() > MaxCachedLogLines) + if (MaxCachedLogLines > 0) { - const int32 Overflow = CachedLogLines.Num() - MaxCachedLogLines; - CachedLogLines.RemoveAt(0, Overflow, EAllowShrinking::No); + if (CachedLogLines.Num() < MaxCachedLogLines) + { + CachedLogLines.Add(Line); + } + else + { + // Full: overwrite the oldest slot in place (reuses its allocation) and advance the head. + CachedLogLines[CachedLogHead] = Line; + CachedLogHead = (CachedLogHead + 1) % MaxCachedLogLines; + } } LogLineDelegate.Broadcast(Line); diff --git a/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Util/FrameGrabberHelper.cpp b/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Util/FrameGrabberHelper.cpp index 258aa37fc..0c7cbff89 100644 --- a/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Util/FrameGrabberHelper.cpp +++ b/UnrealFolder/ProjectMobius/Source/MobiusCore/Private/Util/FrameGrabberHelper.cpp @@ -172,20 +172,20 @@ bool UFrameGrabberHelper::TryInitialize() if (!GEngine) { - MobiusLog(TEXT("[FrameGrabber] TryInitialize failed: GEngine is null")); + NoteInitFailure(TEXT("GEngine is null")); return false; } if (!GEngine->GameViewport) { - MobiusLog(TEXT("[FrameGrabber] TryInitialize failed: GameViewport is null")); + NoteInitFailure(TEXT("GameViewport is null")); return false; } TSharedPtr GameViewportWidget = GEngine->GameViewport->GetGameViewportWidget(); if (!GameViewportWidget.IsValid()) { - MobiusLog(TEXT("[FrameGrabber] TryInitialize failed: GameViewportWidget not constructed yet")); + NoteInitFailure(TEXT("GameViewportWidget not constructed yet")); return false; } @@ -193,7 +193,7 @@ bool UFrameGrabberHelper::TryInitialize() TSharedPtr SlateViewportInterface = GameViewportWidget->GetViewportInterface().Pin(); if (!SlateViewportInterface.IsValid()) { - MobiusLog(TEXT("[FrameGrabber] TryInitialize failed: SlateViewportInterface not wired up yet")); + NoteInitFailure(TEXT("SlateViewportInterface not wired up yet")); return false; } @@ -201,13 +201,16 @@ bool UFrameGrabberHelper::TryInitialize() TSharedPtr SceneViewport = StaticCastSharedPtr(SlateViewportInterface); if (!SceneViewport.IsValid()) { - MobiusLog(TEXT("[FrameGrabber] TryInitialize failed: Unexpected viewport type (not FSceneViewport)")); - MobiusReportError( - FText::FromString(TEXT("Viewport Error")), - FText::FromString(TEXT("Could not access scene viewport for screenshot capture.")), - FText::FromString(TEXT("FrameGrabberHelper::TryInitialize")), - EMobiusErrorSeverity::Warning, - false); // Don't show popup for this, it may resolve on retry + // Escalate once per reason, not per retry: ReportError UE_LOGs and re-enqueues into the custom log too. + if (NoteInitFailure(TEXT("Unexpected viewport type (not FSceneViewport)"))) + { + MobiusReportError( + FText::FromString(TEXT("Viewport Error")), + FText::FromString(TEXT("Could not access scene viewport for screenshot capture.")), + FText::FromString(TEXT("FrameGrabberHelper::TryInitialize")), + EMobiusErrorSeverity::Warning, + false); // Don't show popup for this, it may resolve on retry + } return false; } @@ -217,8 +220,7 @@ bool UFrameGrabberHelper::TryInitialize() // If either dimension is 0, we can't capture yet if (ViewportSize.X <= 0 || ViewportSize.Y <= 0) { - MobiusLog(FString::Printf(TEXT("[FrameGrabber] TryInitialize failed: Invalid viewport size %dx%d"), - ViewportSize.X, ViewportSize.Y)); + NoteInitFailure(FString::Printf(TEXT("Invalid viewport size %dx%d"), ViewportSize.X, ViewportSize.Y)); return false; } @@ -227,23 +229,51 @@ bool UFrameGrabberHelper::TryInitialize() // Clamp/validate target too (avoid passing nonsense to frame grabber) if (TargetSize.X <= 0 || TargetSize.Y <= 0) { - MobiusLog(FString::Printf(TEXT("[FrameGrabber] TryInitialize failed: Invalid target size %dx%d"), - TargetSize.X, TargetSize.Y)); - MobiusReportError( - FText::FromString(TEXT("Configuration Error")), - FText::FromString(FString::Printf(TEXT("Invalid screenshot target size: %dx%d"), TargetSize.X, TargetSize.Y)), - FText::FromString(TEXT("FrameGrabberHelper::TryInitialize")), - EMobiusErrorSeverity::Error, - true); + if (NoteInitFailure(FString::Printf(TEXT("Invalid target size %dx%d"), TargetSize.X, TargetSize.Y))) + { + MobiusReportError( + FText::FromString(TEXT("Configuration Error")), + FText::FromString(FString::Printf(TEXT("Invalid screenshot target size: %dx%d"), TargetSize.X, TargetSize.Y)), + FText::FromString(TEXT("FrameGrabberHelper::TryInitialize")), + EMobiusErrorSeverity::Error, + true); + } return false; } FrameGrabber = MakeUnique(SceneViewport.ToSharedRef(), ViewportSize); FrameGrabber->StartCapturingFrames(); - MobiusLog(FString::Printf(TEXT("[FrameGrabber] Initialized successfully. ViewportSize: %dx%d, TargetSize: %dx%d"), - ViewportSize.X, ViewportSize.Y, TargetSize.X, TargetSize.Y)); + const FString QuietRetries = SuppressedInitFailures > 0 + ? FString::Printf(TEXT(" (after %d unlogged repeats of: %s)"), SuppressedInitFailures, *LastInitFailureReason) + : FString(); + MobiusLog(FString::Printf(TEXT("[FrameGrabber] Initialized successfully. ViewportSize: %dx%d, TargetSize: %dx%d%s"), + ViewportSize.X, ViewportSize.Y, TargetSize.X, TargetSize.Y, *QuietRetries)); + LastInitFailureReason.Reset(); + SuppressedInitFailures = 0; + + return true; +} + +bool UFrameGrabberHelper::NoteInitFailure(const FString& Reason) +{ + // Tick retries until the viewport is usable, and in editor / -game -nullrhi / headless automation runs + // it stays 0x0 for the whole session. Logging every attempt put one identical line per frame into + // MobiusCustomLog.txt - up to ~3,000 a second, 1.07 M of its 1.26 M lines - and each one went through + // the log window too. A reason is logged once, when it changes; repeats are only counted. + if (Reason == LastInitFailureReason) + { + ++SuppressedInitFailures; + return false; + } + const FString PreviousRepeats = SuppressedInitFailures > 0 + ? FString::Printf(TEXT(" - previous reason '%s' repeated %d more times"), *LastInitFailureReason, SuppressedInitFailures) + : FString(); + MobiusLog(FString::Printf(TEXT("[FrameGrabber] TryInitialize failed: %s (retrying quietly; logged again only if the reason changes)%s"), + *Reason, *PreviousRepeats)); + LastInitFailureReason = Reason; + SuppressedInitFailures = 0; return true; } #endif // !PLATFORM_MAC @@ -384,7 +414,9 @@ void UFrameGrabberHelper::TriggerCapture(const FString& InFileName) MobiusLog(TEXT("[FrameGrabber] FrameGrabber not initialized, attempting initialization...")); if (!TryInitialize()) { - MobiusLog(TEXT("[FrameGrabber] Error: Could not initialize FrameGrabber - viewport not ready.")); + // TryInitialize may have stayed quiet (the same reason Tick already logged), so name it here. + MobiusLog(FString::Printf(TEXT("[FrameGrabber] Error: Could not initialize FrameGrabber - viewport not ready (%s)."), + *LastInitFailureReason)); MobiusReportError( FText::FromString(TEXT("Initialization Failed")), FText::FromString(TEXT("Could not initialize screenshot capture. The viewport may not be ready yet.")), @@ -548,10 +580,16 @@ void UFrameGrabberHelper::Tick(float DeltaTime) // The OnScreenshotCaptured callback handles everything #else // Windows/other: Poll FFrameGrabber - // Try lazy initialization if not yet initialized + // Try lazy initialization if not yet initialized. Backed off while it keeps failing: under -nullrhi / + // headless the viewport is 0x0 for the whole session, and a retry every frame bought nothing. + // TriggerCapture still initializes on demand without waiting for this. if (!FrameGrabber.IsValid()) { - TryInitialize(); + InitRetryCooldown -= DeltaTime; + if (InitRetryCooldown <= 0.0f && !TryInitialize()) + { + InitRetryCooldown = InitRetryIntervalSeconds; + } } if (!bIsCapturing || !FrameGrabber.IsValid()) diff --git a/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Subsystems/MobiusCustomLoggerSubsystem.h b/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Subsystems/MobiusCustomLoggerSubsystem.h index b22c027a1..bdbbaa86f 100644 --- a/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Subsystems/MobiusCustomLoggerSubsystem.h +++ b/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Subsystems/MobiusCustomLoggerSubsystem.h @@ -13,7 +13,8 @@ DECLARE_MULTICAST_DELEGATE_OneParam(FOnMobiusLogLine, const FString&); /** - * Engine-level logger that opens a text file next to the launched executable and writes messages via a queue. + * Engine-level logger that writes messages via a queue to a per-session text file, + * MobiusCustomLog_.txt, next to the launched executable (Saved/Logs in the editor). * Designed to be cheap to call from BeginPlay (including blueprints) to spot startup stalls. */ UCLASS() @@ -46,6 +47,9 @@ class MOBIUSCORE_API UMobiusCustomLoggerSubsystem : public UEngineSubsystem /** Native delegate fired when a log line is flushed. */ FOnMobiusLogLine& OnLogLine(); + /** Full path of this session's log file. Each session (process) writes its own; see Initialize. */ + const FString& GetLogFilePath() const { return LogFilePath; } + /** Enable or disable logging (thread-safe). When disabled, EnqueueLogMessage becomes a no-op. */ UFUNCTION(BlueprintCallable, Category = "Mobius|Logging") void SetLoggingEnabled(bool bEnabled); diff --git a/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Subsystems/MobiusUserFeedbackSubsystem.h b/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Subsystems/MobiusUserFeedbackSubsystem.h index cbbc781ec..a1537a45c 100644 --- a/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Subsystems/MobiusUserFeedbackSubsystem.h +++ b/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Subsystems/MobiusUserFeedbackSubsystem.h @@ -92,7 +92,15 @@ class MOBIUSCORE_API UMobiusUserFeedbackSubsystem : public UGameInstanceSubsyste bool bLogWindowOpen = false; TArray PendingErrors; + + /** + * Ring buffer of the newest MaxCachedLogLines lines, so HandleLogLine is O(1) instead of shifting the + * whole cache down on every line. Plain append order until it fills; from then on CachedLogHead is the + * slot of the OLDEST line, i.e. the next one to be overwritten. GetCachedLogLines unrolls it. + */ TArray CachedLogLines; + int32 CachedLogHead = 0; + TMap RecentPromptTimes; FDelegateHandle LogLineHandle; diff --git a/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Util/FrameGrabberHelper.h b/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Util/FrameGrabberHelper.h index 4c7ca0fbd..cdec2fd18 100644 --- a/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Util/FrameGrabberHelper.h +++ b/UnrealFolder/ProjectMobius/Source/MobiusCore/Public/Util/FrameGrabberHelper.h @@ -48,7 +48,27 @@ class MOBIUSCORE_API UFrameGrabberHelper : public UObject /** Attempt to initialize the frame grabber from the game engine's scene viewport */ bool TryInitialize(); + /** + * Log a TryInitialize failure only when its reason differs from the last one logged; repeats are + * counted instead. Returns true if this call logged, so the caller escalates (ReportError) once per + * reason rather than once per retry. + */ + bool NoteInitFailure(const FString& Reason); + TUniquePtr FrameGrabber; + + /** Reason of the last TryInitialize failure that was logged. Cleared by a successful init, so a + * failure after a later (resize) re-init logs again. */ + FString LastInitFailureReason; + + /** Identical failures swallowed since LastInitFailureReason was logged. */ + int32 SuppressedInitFailures = 0; + + /** Seconds until Tick may retry TryInitialize. Tick-side only: TriggerCapture never waits on it. */ + float InitRetryCooldown = 0.0f; + + /** Tick's retry period while the viewport is unusable (0x0 for a whole -nullrhi / headless session). */ + static constexpr float InitRetryIntervalSeconds = 0.5f; #endif #if PLATFORM_MAC diff --git a/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Private/Logging/LogWindow.cpp b/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Private/Logging/LogWindow.cpp index 1c2e713f1..3f6769365 100644 --- a/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Private/Logging/LogWindow.cpp +++ b/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Private/Logging/LogWindow.cpp @@ -13,9 +13,10 @@ #include "Engine/GameInstance.h" #include "Widgets/Input/SButton.h" #include "Widgets/Layout/SBorder.h" -#include "Widgets/Layout/SScrollBox.h" #include "Widgets/SBoxPanel.h" #include "Widgets/Text/STextBlock.h" +#include "Widgets/Views/SListView.h" +#include "Widgets/Views/STableRow.h" namespace { @@ -83,26 +84,25 @@ void SLogWindowWidget::AppendLine(const FString& Line) return; } - LogLines.Add(Line); - if (LogLines.Num() > MaxLines) - { - const int32 Overflow = LogLines.Num() - MaxLines; - LogLines.RemoveAt(0, Overflow, EAllowShrinking::No); - } + // O(1) per line whatever the backlog. The logger's flush calls this once per queued line, so a + // heatmap load arrives as thousands of calls in one frame; all of them together cost one list + // regeneration (visible rows only) on the next paint. + LogItems.Add(MakeShared(Line)); + const int32 Removed = LogItems.Num() > MaxLines ? TrimBacklog() : 0; - RebuildLogText(); + RefreshLogList(Removed, /*bForceToEnd=*/false); } void SLogWindowWidget::SetLines(const TArray& Lines) { - LogLines = Lines; - if (LogLines.Num() > MaxLines) + const int32 First = FMath::Max(0, Lines.Num() - MaxLines); + LogItems.Reset(Lines.Num() - First); + for (int32 Index = First; Index < Lines.Num(); ++Index) { - const int32 Overflow = LogLines.Num() - MaxLines; - LogLines.RemoveAt(0, Overflow, EAllowShrinking::No); + LogItems.Add(MakeShared(Lines[Index])); } - RebuildLogText(); + RefreshLogList(/*RemovedFromFront=*/0, /*bForceToEnd=*/true); } void SLogWindowWidget::ShowLogWindow() @@ -148,13 +148,22 @@ void SLogWindowWidget::ReleaseWindowState() // Unconditional: the binding outlives the window pointer, and OpenLogWindow re-binds on reopen. UnbindThemeChanged(); - // ScrollBox / LogTextBlock / CloseButton live inside the destroyed window's tree and are rebuilt by - // OpenLogWindow. LogLines and LogText are NOT cleared: they are the viewer's contents, and - // OpenLogWindow seeds the new STextBlock from LogText, so a reopen shows the same backlog. + // LogListView holds a raw pointer to LogItems. Detach it before letting go: on the OnWindowClosed + // route this widget - and LogItems with it - is destroyed inside the callback while the window's tree + // is still alive, so a list left pointing at LogItems would outlive the array it reads. + if (LogListView.IsValid()) + { + LogListView->ClearItemsSource(); + } + + // LogListView / CloseButton live inside the destroyed window's tree and are rebuilt by + // OpenLogWindow. LogItems is NOT cleared: it is the viewer's contents, and OpenLogWindow builds the + // new list over it, so a reopen shows the same backlog. LogWindowPtr.Reset(); - ScrollBox.Reset(); - LogTextBlock.Reset(); + LogListView.Reset(); CloseButton.Reset(); + LogSettleTimer.Reset(); + LogSettleFramesLeft = 0; } void SLogWindowWidget::SetEnabled(bool bEnabled) @@ -195,9 +204,27 @@ void SLogWindowWidget::OpenLogWindow() { LogFont = FSlateFontInfo(Inter, 10, FName("Regular")); } - const FTextBlockStyle LogTextStyle = FTextBlockStyle(FCoreStyle::Get().GetWidgetStyle("NormalText")) + LogTextStyle = FTextBlockStyle(FCoreStyle::Get().GetWidgetStyle("NormalText")) .SetFont(LogFont); + // Rows and list draw nothing of their own - no hover, selection or alternating-row brushes, no list + // background - so the themed RibbonBg border shows through exactly as it did behind the old single + // text block. + const FSlateNoResource NoBrush; + LogRowStyle = FTableRowStyle(FCoreStyle::Get().GetWidgetStyle("TableView.Row")) + .SetSelectorFocusedBrush(NoBrush) + .SetActiveHoveredBrush(NoBrush) + .SetActiveBrush(NoBrush) + .SetInactiveHoveredBrush(NoBrush) + .SetInactiveBrush(NoBrush) + .SetEvenRowBackgroundHoveredBrush(NoBrush) + .SetEvenRowBackgroundBrush(NoBrush) + .SetOddRowBackgroundHoveredBrush(NoBrush) + .SetOddRowBackgroundBrush(NoBrush) + .SetActiveHighlightedBrush(NoBrush) + .SetInactiveHighlightedBrush(NoBrush); + LogListViewStyle = FTableViewStyle().SetBackgroundBrush(NoBrush); + // Themed Close button (stored as a member: SButton caches raw pointers into the style's brushes). // Background brushes are stamped for the theme current at construct; the label colour follows live. CloseButtonStyle = MobiusWindowButtonStyle::MakeWindowButtonStyle(FindThemeSubsystemForLogWindow()); @@ -216,19 +243,12 @@ void SLogWindowWidget::OpenLogWindow() + SVerticalBox::Slot() .FillHeight(1.0f) [ - SAssignNew(ScrollBox, SScrollBox) - + SScrollBox::Slot() - [ - SAssignNew(LogTextBlock, STextBlock) - .Text(FText::FromString(LogText)) - .AutoWrapText(true) - .TextStyle(&LogTextStyle) - // Mono log text follows the theme (readable on either surface) via a live poll. - .ColorAndOpacity_Lambda([]() - { - return FSlateColor(PollLogWindowColor(EMobiusPaletteRole::InputMonoText, MobiusThemePalette::Color(EMobiusPaletteRole::InputMonoText, /*bLight=*/true))); - }) - ] + SAssignNew(LogListView, SListView>) + .ListItemsSource(&LogItems) + .OnGenerateRow(this, &SLogWindowWidget::GenerateLogRow) + .OnListViewScrolled(this, &SLogWindowWidget::HandleLogListScrolled) + .SelectionMode(ESelectionMode::None) + .ListViewStyle(&LogListViewStyle) ] + SVerticalBox::Slot() .AutoHeight() @@ -253,6 +273,9 @@ void SLogWindowWidget::OpenLogWindow() ] ]; + // A fresh list starts at the top; open on the newest line, as the old text block's ScrollToEnd did. + RefreshLogList(/*RemovedFromFront=*/0, /*bForceToEnd=*/true); + // Live re-theme for the Close button, event-driven off OnThemeChangedNative. The bind normally succeeds // right here; if it does not, fall back to a 2 Hz bootstrap timer that stops the moment it binds. // @@ -298,26 +321,108 @@ FReply SLogWindowWidget::HandleCloseClicked() return FReply::Handled(); } -void SLogWindowWidget::RebuildLogText() +TSharedRef SLogWindowWidget::GenerateLogRow(TSharedPtr Line, const TSharedRef& OwnerTable) +{ + return SNew(STableRow>, OwnerTable) + .Style(&LogRowStyle) + .ShowSelection(false) + [ + SNew(STextBlock) + .Text(FText::FromString(Line.IsValid() ? *Line : FString())) + .AutoWrapText(true) + .TextStyle(&LogTextStyle) + // Mono log text follows the theme (readable on either surface) via a live poll. + .ColorAndOpacity_Lambda([]() + { + return FSlateColor(PollLogWindowColor(EMobiusPaletteRole::InputMonoText, MobiusThemePalette::Color(EMobiusPaletteRole::InputMonoText, /*bLight=*/true))); + }) + ]; +} + +int32 SLogWindowWidget::TrimBacklog() { - LogText.Reset(); - for (int32 Index = 0; Index < LogLines.Num(); ++Index) + const int32 Keep = FMath::Max(0, MaxLines - LogTrimBatch); + const int32 Removed = FMath::Max(0, LogItems.Num() - Keep); + if (Removed > 0) { - LogText.Append(LogLines[Index]); - if (Index + 1 < LogLines.Num()) - { - LogText.Append(TEXT("\n")); - } + LogItems.RemoveAt(0, Removed, EAllowShrinking::No); + } + return Removed; +} + +void SLogWindowWidget::RefreshLogList(int32 RemovedFromFront, bool bForceToEnd) +{ + // Closed: nothing to refresh. OpenLogWindow builds a fresh list over LogItems. + if (!LogListView.IsValid()) + { + return; } - if (LogTextBlock.IsValid()) + if (bForceToEnd) { - LogTextBlock->SetText(FText::FromString(LogText)); + bFollowLogEnd = true; } + else if (!bFollowLogEnd && !LogListView->IsUserScrolling() + && LogListView->GetScrollDistanceRemaining().Y <= KINDA_SMALL_NUMBER) + { + // The reader scrolled back down to the end (scrollbar state from the list's last layout): follow again. + bFollowLogEnd = true; + } + + LogListView->RequestListRefresh(); + + if (bFollowLogEnd) + { + PinLogListToEnd(); + ArmLogListSettle(); + } + else if (RemovedFromFront > 0) + { + // The scroll offset is an item index: shift it with the trim so the lines being read stay put. + TGuardValue OwnScroll(bScrollingLogListProgrammatically, true); + LogListView->SetScrollOffset(FMath::Max(0.0f, LogListView->GetScrollOffset() - static_cast(RemovedFromFront))); + } +} + +void SLogWindowWidget::PinLogListToEnd() +{ + if (LogListView.IsValid()) + { + TGuardValue OwnScroll(bScrollingLogListProgrammatically, true); + LogListView->ScrollToBottom(); + } +} - if (ScrollBox.IsValid()) +void SLogWindowWidget::ArmLogListSettle() +{ + // Three frames: the one that generates the new rows (measured unwrapped), the one that re-measures + // them wrapped, and one spare. A burst re-arms rather than stacking timers. + LogSettleFramesLeft = 3; + if (!LogSettleTimer.IsValid() && LogListView.IsValid()) + { + LogSettleTimer = LogListView->RegisterActiveTimer(0.0f, FWidgetActiveTimerDelegate::CreateSP(this, &SLogWindowWidget::SettleLogListEnd)); + } +} + +EActiveTimerReturnType SLogWindowWidget::SettleLogListEnd(double InCurrentTime, float InDeltaTime) +{ + // Runs in the list's Paint before its Tick, so the re-pin is laid out in this same frame. + if (!bFollowLogEnd || !LogListView.IsValid() || LogSettleFramesLeft <= 0) + { + LogSettleFramesLeft = 0; + return EActiveTimerReturnType::Stop; + } + + --LogSettleFramesLeft; + PinLogListToEnd(); + return LogSettleFramesLeft > 0 ? EActiveTimerReturnType::Continue : EActiveTimerReturnType::Stop; +} + +void SLogWindowWidget::HandleLogListScrolled(double InScrollOffset) +{ + if (!bScrollingLogListProgrammatically && FSlateApplication::IsInitialized() && FSlateApplication::Get().IsProcessingInput()) { - ScrollBox->ScrollToEnd(); + bFollowLogEnd = false; } } diff --git a/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Private/UI/MobiusSettingsWindowWidget.cpp b/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Private/UI/MobiusSettingsWindowWidget.cpp index 35e87eb2a..94bbf08bc 100644 --- a/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Private/UI/MobiusSettingsWindowWidget.cpp +++ b/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Private/UI/MobiusSettingsWindowWidget.cpp @@ -33,6 +33,7 @@ #include "Components/SizeBox.h" #include "Components/TextBlock.h" #include "Engine/Engine.h" +#include "Misc/Paths.h" #include "Slate/Components/SMoveableWindow.h" #include "Styling/SlateBrush.h" #include "UI/MobiusPanelWindowHost.h" @@ -57,9 +58,6 @@ namespace */ constexpr float GSettingsCardWidth = 700.0f; - /** The logger's file name, single-sourced from MobiusCustomLoggerSubsystem::Initialize. */ - const FString GMobiusLogFileName = TEXT("MobiusCustomLog.txt"); - UUserProjectSettings* GetMobiusUserSettings() { return GEngine ? Cast(GEngine->GetGameUserSettings()) : nullptr; @@ -75,13 +73,21 @@ void UMobiusSettingsWindowWidget::NativePreConstruct() { Super::NativePreConstruct(); - // The brief's footer line says "MobiusLog.txt"; the logger actually appends to MobiusCustomLog.txt. - // Set it here so the UI can never claim a file the logger does not write. + // Name the file the logger is actually writing this session, read from the logger itself, so the UI can + // never claim a file it does not write. The pattern stands in when there is no logger yet. if (LogFileNoteText) { + FString LogFileName = TEXT("MobiusCustomLog_.txt"); + if (const UMobiusCustomLoggerSubsystem* Logger = GetMobiusLogger()) + { + if (!Logger->GetLogFilePath().IsEmpty()) + { + LogFileName = FPaths::GetCleanFilename(Logger->GetLogFilePath()); + } + } LogFileNoteText->SetText(FText::Format( - NSLOCTEXT("MobiusSettings", "LogFileNote", "Every session appends to the same file: {0}"), - FText::FromString(GMobiusLogFileName))); + NSLOCTEXT("MobiusSettings", "LogFileNote", "Each session writes its own log file. This session: {0}"), + FText::FromString(LogFileName))); } } diff --git a/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Public/Logging/LogWindow.h b/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Public/Logging/LogWindow.h index 0ddc1b98b..4fea528fa 100644 --- a/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Public/Logging/LogWindow.h +++ b/UnrealFolder/ProjectMobius/Source/MobiusWidgets/Public/Logging/LogWindow.h @@ -6,11 +6,13 @@ #include "Widgets/SCompoundWidget.h" #include "Styling/SlateTypes.h" +class FActiveTimerHandle; +class ITableRow; class SButton; class SMoveableWindow; -class SScrollBox; -class STextBlock; +class STableViewBase; class UUIThemeSubsystem; +template class SListView; enum class EMobiusUITheme : uint8; DECLARE_DELEGATE(FOnLogWindowClosed); @@ -49,7 +51,41 @@ class MOBIUSWIDGETS_API SLogWindowWidget final : public SCompoundWidget private: void OpenLogWindow(); FReply HandleCloseClicked(); - void RebuildLogText(); + + /** Row factory for LogListView: one STextBlock per line, built only for rows on screen. */ + TSharedRef GenerateLogRow(TSharedPtr Line, const TSharedRef& OwnerTable); + + /** Drop the oldest lines down to MaxLines - LogTrimBatch. Returns how many were removed. */ + int32 TrimBacklog(); + + /** + * Tell the list its items changed. Cheap and idempotent - the list regenerates once, on its next + * paint, for the visible rows only, however many lines arrived in between. Follows the end of the + * log unless the reader has scrolled away from it (bForceToEnd overrides that); a reader who has + * scrolled up keeps their place even when RemovedFromFront lines were trimmed above them. + */ + void RefreshLogList(int32 RemovedFromFront, bool bForceToEnd); + + /** ScrollToBottom, flagged as ours so HandleLogListScrolled does not read it as the reader's. */ + void PinLogListToEnd(); + + /** Start (or extend) the SettleLogListEnd timer on LogListView. */ + void ArmLogListSettle(); + + /** + * Re-pins the end for a few frames after a change. A new row's text is measured before it has a wrap + * width (AutoWrapText learns it from the row's first paint), so the frame that adds wrapping lines + * fits too many rows and the newest ones slide below the fold once they wrap. Registered on + * LogListView, which is painted - this widget never is. + */ + EActiveTimerReturnType SettleLogListEnd(double InCurrentTime, float InDeltaTime); + + /** + * Bound to the list's OnListViewScrolled. Only the reader's own scroll input (wheel, thumb drag, + * keys) stops the follow: our own scrolls are flagged, and the list's end-of-list clamp fires this + * from inside its Tick, i.e. during paint, never while Slate is processing input. + */ + void HandleLogListScrolled(double InScrollOffset); /** * Re-theme the Close button for the current theme. Event-driven: bound to @@ -93,13 +129,45 @@ class MOBIUSWIDGETS_API SLogWindowWidget final : public SCompoundWidget FOnLogWindowClosed OnLogWindowClosed; TSharedPtr LogWindowPtr; - TSharedPtr ScrollBox; - TSharedPtr LogTextBlock; - FString LogText; - TArray LogLines; + + /** + * Virtualized: one row per line, laid out only while on screen. This used to be a single STextBlock + * holding the whole backlog, rebuilt and re-laid out on EVERY appended line - with a full 1000-line + * backlog that is ~90 KB of text per line, and the logger delivers its queue one line at a time. + */ + TSharedPtr>> LogListView; + + /** The viewer's contents, oldest first; LogListView's items source. Survives a close/reopen. */ + TArray> LogItems; + + /** Hard cap on LogItems. */ int32 MaxLines = 1000; + + /** + * Trim granularity: overflowing MaxLines drops this many extra lines at once, so the front-of-array + * memmove is paid once per batch instead of once per appended line. + */ + int32 LogTrimBatch = 128; + + /** True while the view follows the newest line. Cleared only by the reader scrolling; restored when + * they scroll back to the end, and by every (re)open. */ + bool bFollowLogEnd = true; + + /** Set around our own scroll calls, so HandleLogListScrolled can tell them from the reader's. */ + bool bScrollingLogListProgrammatically = false; + + /** Frames SettleLogListEnd still has to re-pin, and its timer (owned by LogListView, dies with it). */ + int32 LogSettleFramesLeft = 0; + TWeakPtr LogSettleTimer; + bool bIsEnabled = true; FWindowStyle LogWindowStyle; + + /** Members, not locals: OnGenerateRow builds rows long after OpenLogWindow returns, and STableRow / + * SListView keep a pointer to the style they were given. */ + FTextBlockStyle LogTextStyle; + FTableRowStyle LogRowStyle; + FTableViewStyle LogListViewStyle; /** Themed Close-button style (member so SButton's cached brush pointers stay valid). */ FButtonStyle CloseButtonStyle; diff --git a/UnrealFolder/ProjectMobius/Source/ProjectMobius/Private/MassAI/SubSystems/AgentDataSubsystem.cpp b/UnrealFolder/ProjectMobius/Source/ProjectMobius/Private/MassAI/SubSystems/AgentDataSubsystem.cpp index 4aa7f4b52..f2d347afa 100644 --- a/UnrealFolder/ProjectMobius/Source/ProjectMobius/Private/MassAI/SubSystems/AgentDataSubsystem.cpp +++ b/UnrealFolder/ProjectMobius/Source/ProjectMobius/Private/MassAI/SubSystems/AgentDataSubsystem.cpp @@ -741,11 +741,95 @@ void FProcessAgentSimulationDataRunnable::RunSimulationDataGatheringLoop(bool bC TEXT("AgentDataSubsystem - RunSimulationDataGatheringLoop")); } } +namespace +{ + /** + * Hands every entity's movement samples to Visit in timestep order, each paired with that entity's + * NEXT sample (the look-ahead both derivations below need), in one pass over SimulationData. + * + * This replaced a per-entity scan that searched every timestep's whole sample array for the entity, + * then searched it again to write the result back: O(entities x total samples), which turned a + * positions-only load of a few thousand agents into minutes. This is O(total samples), and its + * bookkeeping is O(MaxAgents) rather than O(samples) - SimulationData can already run to gigabytes. + * + * Visit(EntityId, Sample, Next) is called once per sample, in timestep order per entity, with Next == + * nullptr for the entity's final sample. Which samples are visited is exactly what the old scan did, + * so derived values stay bit-identical (ProjectMobius.SimData.DerivedMotionParity holds the old scan + * as its oracle): + * - timesteps in ascending key order, whatever order the map iterates in; + * - only ids in [0, MaxAgents) - out-of-range ids stay in the map, untouched; + * - an entity listed twice in one timestep counts once: the FIRST sample is read and written; + * - an entity with fewer than 2 samples is never visited (there is no direction to derive). + */ + template + void ForEachEntitySampleWithNext(TMap>& SimulationData, const int32 MaxAgents, + const FThreadSafeBool& bShouldStop, const TWeakObjectPtr& OwnerSubsystem, + FVisitor&& Visit) + { + struct FEntityCursor + { + /** Latest sample seen, held back until its successor (or the end of the data) is known. */ + FSimMovementSample* Pending = nullptr; + /** Ordinal of the timestep Pending came from - detects a second sample in the same timestep. */ + int32 PendingStepOrdinal = INDEX_NONE; + /** True once Pending has a predecessor, i.e. the entity has at least 2 samples. */ + bool bHasPredecessor = false; + }; + + TArray Timesteps; + SimulationData.GetKeys(Timesteps); + Timesteps.Sort(); + + TArray Cursors; + Cursors.SetNum(FMath::Max(MaxAgents, 0)); + + for (int32 StepOrdinal = 0; StepOrdinal < Timesteps.Num(); ++StepOrdinal) + { + if (bShouldStop) return; + + for (FSimMovementSample& Sample : SimulationData.FindChecked(Timesteps[StepOrdinal])) + { + if (Sample.EntityID < 0 || Sample.EntityID >= MaxAgents) continue; + + FEntityCursor& Cursor = Cursors[Sample.EntityID]; + if (Cursor.PendingStepOrdinal == StepOrdinal) continue; + + if (Cursor.Pending) + { + Visit(Sample.EntityID, *Cursor.Pending, &Sample); + Cursor.bHasPredecessor = true; + } + Cursor.Pending = &Sample; + Cursor.PendingStepOrdinal = StepOrdinal; + } + + // Report progress to the subsystem for UI feedback (every 100 timesteps to avoid flooding) + if (StepOrdinal % 100 == 0) + { + float CurrentPercentage = static_cast(StepOrdinal) / static_cast(Timesteps.Num()); + if (UAgentDataSubsystem* Subsys = OwnerSubsystem.Get()) + { + Subsys->ProgressQueue.Enqueue(CurrentPercentage); + } + } + } + + // Each entity's final sample has no successor: Visit it last, so it carries the previous value + for (int32 EntityIdx = 0; EntityIdx < Cursors.Num(); ++EntityIdx) + { + if (Cursors[EntityIdx].bHasPredecessor) + { + Visit(EntityIdx, *Cursors[EntityIdx].Pending, nullptr); + } + } + } +} + /** * Calculates rotation from movement direction when the HDF5 source data lacks rotation information. * - * Algorithm: - * 1. For each entity, collect all position samples ordered by timestep + * Algorithm (one pass over all samples, see ForEachEntitySampleWithNext): + * 1. Walk each entity's position samples in timestep order * 2. For each timestep, calculate direction vector to the NEXT position (look-ahead approach) * 3. Convert direction to rotation angle in degrees using FMath::Atan2 * 4. For the last timestep, carry forward the previous calculated rotation @@ -764,50 +848,19 @@ void FProcessAgentSimulationDataRunnable::CalculateRotationFromMovement() UE_LOG(LogTemp, Log, TEXT("Calculating rotation from movement direction for %d entities"), MaxAgents); - // Process each entity independently - for (int32 EntityIdx = 0; EntityIdx < MaxAgents; ++EntityIdx) - { - if (bShouldStop) break; - - // Step 1: Collect all position samples for this entity across all timesteps - // We store (timestep index, position) pairs for later sorting - TArray> EntityPositions; + // Per entity: the rotation used for stationary steps and carried onto the last frame + TArray LastValidRotations; + LastValidRotations.Init(FRotator::ZeroRotator, FMath::Max(MaxAgents, 0)); - for (auto& Pair : *AgentMovementInfoData.SimulationData) + ForEachEntitySampleWithNext(*AgentMovementInfoData.SimulationData, MaxAgents, bShouldStop, OwnerSubsystem, + [&LastValidRotations](const int32 EntityIdx, FSimMovementSample& Sample, const FSimMovementSample* NextSample) { - for (const FSimMovementSample& Sample : Pair.Value) - { - if (Sample.EntityID == EntityIdx) - { - EntityPositions.Add(TPair(Pair.Key, Sample.Position)); - break; // Only one sample per entity per timestep, so we can exit early - } - } - } - - // Need at least 2 positions to calculate any rotation (need direction to next position) - if (EntityPositions.Num() < 2) continue; - - // Step 2: Sort positions by timestep to ensure chronological order - EntityPositions.Sort([](const TPair& A, const TPair& B) - { - return A.Key < B.Key; - }); - - // Step 3: Calculate rotation for each timestep using look-ahead to next position - // LastValidRotation is used for stationary entities or the last frame - FRotator LastValidRotation = FRotator::ZeroRotator; - - for (int32 i = 0; i < EntityPositions.Num(); ++i) - { - if (bShouldStop) break; - - FRotator NewRotation = LastValidRotation; // Default to previous rotation + FRotator& LastValidRotation = LastValidRotations[EntityIdx]; // Calculate direction from current position to NEXT position (look-ahead approach) - if (i < EntityPositions.Num() - 1) + if (NextSample) { - FVector Delta = EntityPositions[i + 1].Value - EntityPositions[i].Value; + FVector Delta = NextSample->Position - Sample.Position; // Only update rotation if there's meaningful movement (avoid jitter from tiny movements) // Threshold of 0.1 cm filters out noise while allowing detection of real movement @@ -821,35 +874,14 @@ void FProcessAgentSimulationDataRunnable::CalculateRotationFromMovement() // Apply -90 degree offset for mesh orientation correction // Unreal meshes typically face +X when yaw=0, but our coordinate system // expects facing direction to align with movement vector - NewRotation = FRotator(0.0f, YawDeg - 90.0f, 0.0f); - LastValidRotation = NewRotation; + LastValidRotation = FRotator(0.0f, YawDeg - 90.0f, 0.0f); } // If position hasn't changed significantly, keep LastValidRotation } // For the last frame, we keep LastValidRotation (carry forward previous rotation) - // Step 4: Update the rotation value in the actual simulation data - int32 Timestep = EntityPositions[i].Key; - for (FSimMovementSample& Sample : (*AgentMovementInfoData.SimulationData)[Timestep]) - { - if (Sample.EntityID == EntityIdx) - { - Sample.Rotation = NewRotation; - break; - } - } - } - - // Report progress to the subsystem for UI feedback (every 100 entities to avoid flooding) - if (EntityIdx % 100 == 0) - { - float CurrentPercentage = static_cast(EntityIdx) / static_cast(MaxAgents); - if (UAgentDataSubsystem* Subsys = OwnerSubsystem.Get()) - { - Subsys->ProgressQueue.Enqueue(CurrentPercentage); - } - } - } + Sample.Rotation = LastValidRotation; + }); UE_LOG(LogTemp, Log, TEXT("Finished calculating rotation from movement")); } @@ -857,8 +889,8 @@ void FProcessAgentSimulationDataRunnable::CalculateRotationFromMovement() /** * Calculates speed from position deltas when the HDF5 source data lacks speed information. * - * Algorithm: - * 1. For each entity, collect all position samples ordered by timestep + * Algorithm (one pass over all samples, see ForEachEntitySampleWithNext): + * 1. Walk each entity's position samples in timestep order * 2. Calculate speed using the formula: speed = distance / time * - Distance is the Euclidean distance between consecutive positions (in cm after unit conversion) * - Time is TimeBetweenSteps (the sampling interval in seconds) @@ -869,7 +901,7 @@ void FProcessAgentSimulationDataRunnable::CalculateRotationFromMovement() * this discrepancy occurs - possible causes include: * - TimeBetweenSteps not being set correctly before this function is called * - Position data not being properly converted to cm before storage - * - Entities that only appear in a single timestep (skipped due to Num() < 2 check) + * - Entities that only appear in a single timestep (skipped: they have no next position) * - Floating point precision issues with very small movements */ void FProcessAgentSimulationDataRunnable::CalculateSpeedFromMovement() @@ -882,80 +914,29 @@ void FProcessAgentSimulationDataRunnable::CalculateSpeedFromMovement() UE_LOG(LogTemp, Log, TEXT("Calculating speed from position deltas for %d entities"), MaxAgents); - // Process each entity independently - for (int32 EntityIdx = 0; EntityIdx < MaxAgents; ++EntityIdx) - { - if (bShouldStop) break; - - // Step 1: Collect all position samples for this entity across all timesteps - // We store (timestep index, position) pairs for later sorting - TArray> EntityPositions; + // Per entity: the speed carried onto the last frame + TArray LastValidSpeeds; + LastValidSpeeds.Init(0.0f, FMath::Max(MaxAgents, 0)); - for (auto& Pair : *AgentMovementInfoData.SimulationData) + ForEachEntitySampleWithNext(*AgentMovementInfoData.SimulationData, MaxAgents, bShouldStop, OwnerSubsystem, + [this, &LastValidSpeeds](const int32 EntityIdx, FSimMovementSample& Sample, const FSimMovementSample* NextSample) { - for (const FSimMovementSample& Sample : Pair.Value) - { - if (Sample.EntityID == EntityIdx) - { - EntityPositions.Add(TPair(Pair.Key, Sample.Position)); - break; // Only one sample per entity per timestep, so we can exit early - } - } - } - - // Need at least 2 positions to calculate any speed - if (EntityPositions.Num() < 2) continue; - - // Step 2: Sort positions by timestep to ensure chronological order - EntityPositions.Sort([](const TPair& A, const TPair& B) - { - return A.Key < B.Key; - }); - - // Step 3: Calculate speed for each timestep using look-ahead to next position - float LastValidSpeed = 0.0f; - - for (int32 i = 0; i < EntityPositions.Num(); ++i) - { - if (bShouldStop) break; - - float Speed = LastValidSpeed; // Default to previous speed (used for last frame) + float& LastValidSpeed = LastValidSpeeds[EntityIdx]; // Calculate speed from current position to next position - if (i < EntityPositions.Num() - 1) + if (NextSample) { - FVector Delta = EntityPositions[i + 1].Value - EntityPositions[i].Value; + FVector Delta = NextSample->Position - Sample.Position; float Distance = Delta.Size(); // Distance in cm (positions are converted to cm during loading) // Speed formula: speed (m/s) = distance (cm) / (time (s) * 100) // The * 100 converts cm to m - Speed = Distance / (TimeBetweenSteps * 100.0f); - LastValidSpeed = Speed; + LastValidSpeed = Distance / (TimeBetweenSteps * 100.0f); } // For the last frame, we keep LastValidSpeed (carry forward previous speed) - // Step 4: Update the speed value in the actual simulation data - int32 Timestep = EntityPositions[i].Key; - for (FSimMovementSample& Sample : (*AgentMovementInfoData.SimulationData)[Timestep]) - { - if (Sample.EntityID == EntityIdx) - { - Sample.Speed = Speed; - break; - } - } - } - - // Report progress to the subsystem for UI feedback (every 100 entities) - if (EntityIdx % 100 == 0) - { - float CurrentPercentage = static_cast(EntityIdx) / static_cast(MaxAgents); - if (UAgentDataSubsystem* Subsys = OwnerSubsystem.Get()) - { - Subsys->ProgressQueue.Enqueue(CurrentPercentage); - } - } - } + Sample.Speed = LastValidSpeed; + }); UE_LOG(LogTemp, Log, TEXT("Finished calculating speed from movement")); } @@ -1230,6 +1211,8 @@ uint32 FProcessAgentSimulationDataRunnable:: Run() return 0; } + const double DeriveStart = FPlatformTime::Seconds(); + // Calculate rotation from movement if rotation data is missing if (!AgentSimulationData.Metadata.bHasRotationData) { @@ -1255,6 +1238,11 @@ uint32 FProcessAgentSimulationDataRunnable:: Run() CalculateSpeedFromMovement(); } + if (!AgentSimulationData.Metadata.bHasRotationData || !AgentSimulationData.Metadata.bHasSpeedData) + { + ImportTimings.DeriveSeconds = FPlatformTime::Seconds() - DeriveStart; + } + if (bShouldStop) { return 0; diff --git a/UnrealFolder/ProjectMobius/Source/ProjectMobius/Public/MassAI/SubSystems/AgentDataSubsystem.h b/UnrealFolder/ProjectMobius/Source/ProjectMobius/Public/MassAI/SubSystems/AgentDataSubsystem.h index 260029612..bfaf6ac8e 100644 --- a/UnrealFolder/ProjectMobius/Source/ProjectMobius/Public/MassAI/SubSystems/AgentDataSubsystem.h +++ b/UnrealFolder/ProjectMobius/Source/ProjectMobius/Public/MassAI/SubSystems/AgentDataSubsystem.h @@ -250,6 +250,8 @@ struct FMobiusImportTimings double ParseSeconds = 0.0; /** RunSimulationDataGatheringLoop: dense per-timestep map build + unit conversions. */ double ConvertSeconds = 0.0; + /** CalculateRotationFromMovement + CalculateSpeedFromMovement (positions-only sources; 0 otherwise). */ + double DeriveSeconds = 0.0; /** CalcSmoothedStepMovementBrackets. */ double BracketSeconds = 0.0; /** WriteCacheForImport (A3; includes the reuse-check when it skips). */ diff --git a/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/DerivedMotionTest.cpp b/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/DerivedMotionTest.cpp new file mode 100644 index 000000000..e1d7fb9a6 --- /dev/null +++ b/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/DerivedMotionTest.cpp @@ -0,0 +1,569 @@ +// Copyright (c) 2025 ProjectMobius contributors. Licensed under MIT. +// +// DerivedMotionTest.cpp +// +// Guards FProcessAgentSimulationDataRunnable::CalculateRotationFromMovement() and +// CalculateSpeedFromMovement(), which derive heading and speed for sources that carry positions only +// (every Juelich file, and Mobius-HDF5 written positions-only). Both were rewritten from a per-entity +// scan, O(entities x total samples), to one O(total samples) pass; no derived value may move by a bit. +// +// The oracle is the old scan itself, copied below from AgentDataSubsystem.cpp at 7dc1b050. Only how it +// reaches the map, MaxAgents and TimeBetweenSteps changed; logging, progress and cancellation were +// dropped. Every sample's Rotation and Speed get a unique sentinel first, so a write the old scan never +// made - or one it made that the new pass skips - shows up as a mismatch, not as an equal value. +// +// ProjectMobius.SimData.DerivedMotionParity - real Juelich export + a hand-built edge-case map +// Mobius.Timing.DerivedMotionScaling - old vs new wall-clock at two sizes. Set +// MOBIUS_DERIVED_MOTION_FILE to a positions-only file +// to time a real import as well. +// +// Run: MobiusPerf\RunTests.ps1 -Filter "ProjectMobius.SimData.DerivedMotion+Mobius.Timing.DerivedMotion" +// +#if !UE_BUILD_SHIPPING + +#include "CoreMinimal.h" +#include "HAL/IConsoleManager.h" +#include "HAL/PlatformMisc.h" +#include "HAL/PlatformTime.h" +#include "Misc/AutomationTest.h" +#include "Misc/Paths.h" +#include "MassAI/SubSystems/AgentDataSubsystem.h" +#include "MassAI/Fragments/SharedFragments/SimulationFragment.h" + +namespace DerivedMotionOracle +{ + using FSimMap = TMap>; + + // FProcessAgentSimulationDataRunnable::CalculateRotationFromMovement() as of 7dc1b050. + void CalculateRotationFromMovement(FSimMap& SimulationData, const int32 MaxAgents) + { + for (int32 EntityIdx = 0; EntityIdx < MaxAgents; ++EntityIdx) + { + TArray> EntityPositions; + + for (auto& Pair : SimulationData) + { + for (const FSimMovementSample& Sample : Pair.Value) + { + if (Sample.EntityID == EntityIdx) + { + EntityPositions.Add(TPair(Pair.Key, Sample.Position)); + break; + } + } + } + + if (EntityPositions.Num() < 2) continue; + + EntityPositions.Sort([](const TPair& A, const TPair& B) + { + return A.Key < B.Key; + }); + + FRotator LastValidRotation = FRotator::ZeroRotator; + + for (int32 i = 0; i < EntityPositions.Num(); ++i) + { + FRotator NewRotation = LastValidRotation; + + if (i < EntityPositions.Num() - 1) + { + FVector Delta = EntityPositions[i + 1].Value - EntityPositions[i].Value; + + if (!Delta.IsNearlyZero(0.1f)) + { + float YawRad = FMath::Atan2(Delta.Y, Delta.X); + float YawDeg = FMath::RadiansToDegrees(YawRad); + + NewRotation = FRotator(0.0f, YawDeg - 90.0f, 0.0f); + LastValidRotation = NewRotation; + } + } + + int32 Timestep = EntityPositions[i].Key; + for (FSimMovementSample& Sample : SimulationData[Timestep]) + { + if (Sample.EntityID == EntityIdx) + { + Sample.Rotation = NewRotation; + break; + } + } + } + } + } + + // FProcessAgentSimulationDataRunnable::CalculateSpeedFromMovement() as of 7dc1b050. + void CalculateSpeedFromMovement(FSimMap& SimulationData, const int32 MaxAgents, const float TimeBetweenSteps) + { + for (int32 EntityIdx = 0; EntityIdx < MaxAgents; ++EntityIdx) + { + TArray> EntityPositions; + + for (auto& Pair : SimulationData) + { + for (const FSimMovementSample& Sample : Pair.Value) + { + if (Sample.EntityID == EntityIdx) + { + EntityPositions.Add(TPair(Pair.Key, Sample.Position)); + break; + } + } + } + + if (EntityPositions.Num() < 2) continue; + + EntityPositions.Sort([](const TPair& A, const TPair& B) + { + return A.Key < B.Key; + }); + + float LastValidSpeed = 0.0f; + + for (int32 i = 0; i < EntityPositions.Num(); ++i) + { + float Speed = LastValidSpeed; + + if (i < EntityPositions.Num() - 1) + { + FVector Delta = EntityPositions[i + 1].Value - EntityPositions[i].Value; + float Distance = Delta.Size(); + + Speed = Distance / (TimeBetweenSteps * 100.0f); + LastValidSpeed = Speed; + } + + int32 Timestep = EntityPositions[i].Key; + for (FSimMovementSample& Sample : SimulationData[Timestep]) + { + if (Sample.EntityID == EntityIdx) + { + Sample.Speed = Speed; + break; + } + } + } + } + } +} + +// Named, not anonymous: helper names like DoubleBits/MakeSample already exist in other test files' +// anonymous namespaces, which would collide if this module is ever unity-built. +namespace DerivedMotionTestUtil +{ + using DerivedMotionOracle::FSimMap; + + const TCHAR* JuelichFixtureRelative = TEXT("MobiusDataImporter/Hdf5TestData/JuelichTestCases/trajectories_hdf5/030_c_56_h0.h5"); + + uint64 DoubleBits(const double Value) + { + uint64 Bits = 0; + FMemory::Memcpy(&Bits, &Value, sizeof(Bits)); + return Bits; + } + + uint32 FloatBits(const float Value) + { + uint32 Bits = 0; + FMemory::Memcpy(&Bits, &Value, sizeof(Bits)); + return Bits; + } + + bool RotatorsBitEqual(const FRotator& A, const FRotator& B) + { + return DoubleBits(A.Pitch) == DoubleBits(B.Pitch) && DoubleBits(A.Yaw) == DoubleBits(B.Yaw) && DoubleBits(A.Roll) == DoubleBits(B.Roll); + } + + bool VectorsBitEqual(const FVector& A, const FVector& B) + { + return DoubleBits(A.X) == DoubleBits(B.X) && DoubleBits(A.Y) == DoubleBits(B.Y) && DoubleBits(A.Z) == DoubleBits(B.Z); + } + + /** A Speed no derivation can produce (negative) and that is unique per sample. */ + float SentinelSpeed(const int32 SampleOrdinal) + { + return -1.0f - static_cast(SampleOrdinal); + } + + /** Overwrite every Rotation/Speed with a per-sample sentinel, so "not written" is observable. */ + void StampSentinels(FSimMap& Map) + { + int32 SampleOrdinal = 0; + for (TPair>& Pair : Map) + { + for (FSimMovementSample& Sample : Pair.Value) + { + Sample.Rotation = FRotator(1000.0 + SampleOrdinal, 2000.0 + SampleOrdinal, 3000.0 + SampleOrdinal); + Sample.Speed = SentinelSpeed(SampleOrdinal); + ++SampleOrdinal; + } + } + } + + /** Runs the shipping derivation on Map through a bare runnable; Run() is never called. */ + void DeriveWithRunnable(const FString& AnyExistingFile, UAgentDataSubsystem* Owner, const TSharedPtr& Map, + const int32 MaxAgents, const float TimeBetweenSteps) + { + // The constructor only checks the file exists; nothing reads it unless Run() is called. + FProcessAgentSimulationDataRunnable Runnable(AnyExistingFile, Owner, /*bAutoStartThread*/ false); + Runnable.MaxAgents = MaxAgents; + Runnable.TimeBetweenSteps = TimeBetweenSteps; + Runnable.AgentSimulationData.Metadata.bHasRotationData = false; + Runnable.AgentSimulationData.Metadata.bHasSpeedData = false; + Runnable.AgentMovementInfoData.SimulationData = Map; + Runnable.CalculateRotationFromMovement(); + Runnable.CalculateSpeedFromMovement(); + } + + void DeriveWithOracle(FSimMap& Map, const int32 MaxAgents, const float TimeBetweenSteps) + { + DerivedMotionOracle::CalculateRotationFromMovement(Map, MaxAgents); + DerivedMotionOracle::CalculateSpeedFromMovement(Map, MaxAgents, TimeBetweenSteps); + } + + struct FParity + { + int32 Samples = 0; + /** Samples the oracle wrote (Speed no longer its sentinel). */ + int32 Derived = 0; + int32 Mismatches = 0; + FString FirstMismatch; + }; + + /** Bit-for-bit comparison of two derivations of the same sentinel-stamped source map. */ + FParity CompareDerivations(const FSimMap& Oracle, const FSimMap& Candidate) + { + FParity Result; + auto NoteMismatch = [&Result](const FString& What) + { + if (Result.Mismatches++ == 0) + { + Result.FirstMismatch = What; + } + }; + + if (Oracle.Num() != Candidate.Num()) + { + NoteMismatch(FString::Printf(TEXT("timestep count %d vs %d"), Oracle.Num(), Candidate.Num())); + return Result; + } + + int32 SampleOrdinal = 0; + for (const TPair>& Pair : Oracle) + { + const TArray* CandidateSamples = Candidate.Find(Pair.Key); + if (!CandidateSamples || CandidateSamples->Num() != Pair.Value.Num()) + { + NoteMismatch(FString::Printf(TEXT("timestep %d missing or resized"), Pair.Key)); + SampleOrdinal += Pair.Value.Num(); + continue; + } + + for (int32 Index = 0; Index < Pair.Value.Num(); ++Index, ++SampleOrdinal) + { + const FSimMovementSample& A = Pair.Value[Index]; + const FSimMovementSample& B = (*CandidateSamples)[Index]; + ++Result.Samples; + if (FloatBits(A.Speed) != FloatBits(SentinelSpeed(SampleOrdinal))) + { + ++Result.Derived; + } + + if (A.EntityID != B.EntityID || !VectorsBitEqual(A.Position, B.Position) + || !RotatorsBitEqual(A.Rotation, B.Rotation) || FloatBits(A.Speed) != FloatBits(B.Speed)) + { + NoteMismatch(FString::Printf(TEXT("timestep %d sample %d entity %d: oracle yaw %.9g speed %.9g, candidate yaw %.9g speed %.9g"), + Pair.Key, Index, A.EntityID, A.Rotation.Yaw, A.Speed, B.Rotation.Yaw, B.Speed)); + } + } + } + return Result; + } + + /** Oracle vs shipping derivation on copies of Source (which is sentinel-stamped first). */ + FParity RunParity(FSimMap Source, const int32 MaxAgents, const float TimeBetweenSteps, + const FString& AnyExistingFile, UAgentDataSubsystem* Owner, + double* OutOracleSeconds = nullptr, double* OutCandidateSeconds = nullptr) + { + StampSentinels(Source); + FSimMap OracleMap = Source; + TSharedPtr CandidateMap = MakeShared(MoveTemp(Source)); + + const double OracleStart = FPlatformTime::Seconds(); + DeriveWithOracle(OracleMap, MaxAgents, TimeBetweenSteps); + const double OracleSeconds = FPlatformTime::Seconds() - OracleStart; + + const double CandidateStart = FPlatformTime::Seconds(); + DeriveWithRunnable(AnyExistingFile, Owner, CandidateMap, MaxAgents, TimeBetweenSteps); + const double CandidateSeconds = FPlatformTime::Seconds() - CandidateStart; + + if (OutOracleSeconds) { *OutOracleSeconds = OracleSeconds; } + if (OutCandidateSeconds) { *OutCandidateSeconds = CandidateSeconds; } + return CompareDerivations(OracleMap, *CandidateMap); + } + + /** FNV-1a over every sample's derived fields in timestep order - comparable across builds. */ + uint64 DigestDerivedValues(const FSimMap& Map) + { + uint64 Hash = 0xcbf29ce484222325ull; + auto Mix = [&Hash](const uint64 Value) + { + for (int32 Byte = 0; Byte < 8; ++Byte) + { + Hash ^= (Value >> (Byte * 8)) & 0xffull; + Hash *= 0x100000001b3ull; + } + }; + + TArray Timesteps; + Map.GetKeys(Timesteps); + Timesteps.Sort(); + for (const int32 Timestep : Timesteps) + { + for (const FSimMovementSample& Sample : Map.FindChecked(Timestep)) + { + Mix(static_cast(Timestep)); + Mix(static_cast(static_cast(Sample.EntityID))); + Mix(DoubleBits(Sample.Rotation.Pitch)); + Mix(DoubleBits(Sample.Rotation.Yaw)); + Mix(DoubleBits(Sample.Rotation.Roll)); + Mix(FloatBits(Sample.Speed)); + } + } + return Hash; + } + + FSimMovementSample MakeDerivedMotionSample(const int32 EntityId, const double X, const double Y) + { + FSimMovementSample Sample; + Sample.EntityID = EntityId; + Sample.Position = FVector(X, Y, 0.0); + return Sample; + } + + /** Pins both .msc cache cvars off, so an import always parses and derives, and never writes a cache. */ + struct FScopedSimCacheOff + { + IConsoleVariable* WriteCVar = IConsoleManager::Get().FindConsoleVariable(TEXT("mobius.SimCache.WriteOnImport")); + IConsoleVariable* FastReloadCVar = IConsoleManager::Get().FindConsoleVariable(TEXT("mobius.SimCache.FastReload")); + int32 SavedWrite = WriteCVar ? WriteCVar->GetInt() : 1; + int32 SavedFastReload = FastReloadCVar ? FastReloadCVar->GetInt() : 1; + + FScopedSimCacheOff() + { + if (WriteCVar) { WriteCVar->Set(0, ECVF_SetByCode); } + if (FastReloadCVar) { FastReloadCVar->Set(0, ECVF_SetByCode); } + } + + ~FScopedSimCacheOff() + { + if (WriteCVar) { WriteCVar->Set(SavedWrite, ECVF_SetByCode); } + if (FastReloadCVar) { FastReloadCVar->Set(SavedFastReload, ECVF_SetByCode); } + } + }; + + /** + * Positions-only map shaped like an evacuation: agents enter and leave at staggered times, so + * timestep arrays vary in length and each agent sits at a different index in each one. + */ + FSimMap MakeSyntheticPositionsOnly(const int32 NumAgents, const int32 NumTimesteps) + { + FSimMap Map; + for (int32 Timestep = 0; Timestep < NumTimesteps; ++Timestep) + { + TArray& Samples = Map.Add(Timestep); + for (int32 Agent = 0; Agent < NumAgents; ++Agent) + { + const int32 Enter = (Agent * 37) % (NumTimesteps / 3); + const int32 Leave = Enter + NumTimesteps / 2 + (Agent * 53) % (NumTimesteps / 2); + if (Timestep < Enter || Timestep >= Leave) + { + continue; + } + const double Heading = Agent * 0.7; + const double Walked = 12.0 * (Timestep - Enter); + Samples.Add(MakeDerivedMotionSample(Agent, 50.0 * Agent + Walked * FMath::Cos(Heading), Walked * FMath::Sin(Heading))); + } + } + return Map; + } +} + +IMPLEMENT_SIMPLE_AUTOMATION_TEST( + FDerivedMotionParityTest, + "ProjectMobius.SimData.DerivedMotionParity", + EAutomationTestFlags::EditorContext | EAutomationTestFlags::ProductFilter) + +bool FDerivedMotionParityTest::RunTest(const FString& Parameters) +{ + using namespace DerivedMotionTestUtil; + + const FString JuelichPath = FPaths::ConvertRelativePathToFull(FPaths::Combine(FPaths::ProjectPluginsDir(), JuelichFixtureRelative)); + if (!TestTrue(FString::Printf(TEXT("Juelich fixture present (%s)"), *JuelichPath), FPaths::FileExists(JuelichPath))) + { + return false; + } + + UAgentDataSubsystem* Owner = NewObject(); + + // ---------- 1. Real Juelich export through the real import ---------- + { + FScopedSimCacheOff CacheOff; + FProcessAgentSimulationDataRunnable Import(JuelichPath, Owner, /*bAutoStartThread*/ false); + Import.Run(); + + // A cache hit would skip derivation altogether and make every comparison below trivially true. + TestFalse(TEXT("import parsed the source, not a cache"), Import.ImportTimings.bUsedFastReload); + TestEqual(TEXT("file format"), static_cast(Import.AgentFileFormat), static_cast(EMobiusAgentFileFormat::JuelichHdf5)); + TestTrue(TEXT("import ran the derivation"), Import.ImportTimings.DeriveSeconds > 0.0); + TestTrue(TEXT("Juelich has agents"), Import.MaxAgents > 0); + + const FSimMap& Imported = *Import.AgentMovementInfoData.SimulationData; + AddInfo(FString::Printf(TEXT("Juelich 030_c_56_h0: %d agents, %d timesteps, TimeBetweenSteps %.6f s, derive %.4f s; derived-value digest 0x%016llx"), + Import.MaxAgents, Imported.Num(), Import.TimeBetweenSteps, Import.ImportTimings.DeriveSeconds, DigestDerivedValues(Imported))); + + const FParity Parity = RunParity(Imported, Import.MaxAgents, Import.TimeBetweenSteps, JuelichPath, Owner); + TestTrue(TEXT("Juelich: oracle derived something"), Parity.Derived > 0); + TestEqual(FString::Printf(TEXT("Juelich: bit-identical to the old scan (%d samples, %d derived; first mismatch: %s)"), + Parity.Samples, Parity.Derived, *Parity.FirstMismatch), Parity.Mismatches, 0); + } + + // ---------- 2. Edge cases a clean export never hits ---------- + { + const int32 MaxAgents = 6; + const float TimeBetweenSteps = 0.1f; + + // Keys inserted out of order: both derivations must walk timesteps by key, not by insertion. + FSimMap Map; + Map.Add(3, TArray{ + MakeDerivedMotionSample(3, 400.0, 0.0), + MakeDerivedMotionSample(0, 30.0, 30.0), + MakeDerivedMotionSample(1, 100.0, 0.0), // first move of a stationary-from-start agent: +X, yaw -90 + MakeDerivedMotionSample(6, 1.0, 1.0), // id == MaxAgents: never derived + }); + Map.Add(0, TArray{ + MakeDerivedMotionSample(0, 0.0, 0.0), + MakeDerivedMotionSample(1, 0.0, 0.0), + MakeDerivedMotionSample(3, 0.0, 0.0), + MakeDerivedMotionSample(4, 0.0, 0.0), + MakeDerivedMotionSample(-1, 5.0, 5.0), // negative id: never derived + MakeDerivedMotionSample(7, 2.0, 2.0), // id > MaxAgents: never derived + }); + Map.Add(4, TArray()); // empty timestep + Map.Add(2, TArray{ + MakeDerivedMotionSample(2, 9.0, 9.0), // the only sample of entity 2: never derived + MakeDerivedMotionSample(0, 20.0, 20.0), + MakeDerivedMotionSample(1, 0.0, 0.0), + }); + Map.Add(1, TArray{ + MakeDerivedMotionSample(1, 0.0, 0.0), + MakeDerivedMotionSample(3, 100.0, 0.0), + MakeDerivedMotionSample(0, 10.0, 10.0), + MakeDerivedMotionSample(3, -500.0, -500.0), // second sample of entity 3 in one timestep: only the first counts + MakeDerivedMotionSample(6, 3.0, 3.0), + }); + Map.Add(5, TArray{ + MakeDerivedMotionSample(1, 100.0, 0.05), // sub-0.1 cm step along +Y: heading holds at -90, not 0 + MakeDerivedMotionSample(4, 0.0, 300.0), // entity 4 skipped timesteps 1-4 + MakeDerivedMotionSample(0, 60.0, 30.0), // last move +X: yaw -90, carried onto this final frame + }); + + const FParity Parity = RunParity(Map, MaxAgents, TimeBetweenSteps, JuelichPath, Owner); + TestEqual(FString::Printf(TEXT("edge cases: bit-identical to the old scan (%d samples, %d derived; first mismatch: %s)"), + Parity.Samples, Parity.Derived, *Parity.FirstMismatch), Parity.Mismatches, 0); + // 0: 5 samples, 1: 5, 3: 3 (duplicate excluded), 4: 2. Entities 2, 5 and the out-of-range ids: none. + TestEqual(TEXT("edge cases: exactly the expected samples were derived"), Parity.Derived, 15); + + // The parity check above proves "same as before"; these pin down what "before" was. + TSharedPtr Derived = MakeShared(Map); + StampSentinels(*Derived); + DeriveWithRunnable(JuelichPath, Owner, Derived, MaxAgents, TimeBetweenSteps); + const FSimMovementSample& StationaryStart = (*Derived)[0][1]; + TestTrue(TEXT("stationary-from-start agent holds yaw 0 (not the importer's -90)"), RotatorsBitEqual(StationaryStart.Rotation, FRotator::ZeroRotator)); + TestEqual(TEXT("stationary-from-start agent speed 0"), StationaryStart.Speed, 0.0f); + const FSimMovementSample& Duplicate = (*Derived)[1][3]; + TestTrue(TEXT("duplicate sample left untouched"), Duplicate.Speed < 0.0f); + TestTrue(TEXT("single-sample entity left untouched"), (*Derived)[2][0].Speed < 0.0f); + TestTrue(TEXT("negative id left untouched"), (*Derived)[0][4].Speed < 0.0f); + TestTrue(TEXT("id > MaxAgents left untouched"), (*Derived)[0][5].Speed < 0.0f); + TestTrue(TEXT("id == MaxAgents left untouched"), (*Derived)[3][3].Speed < 0.0f && (*Derived)[1][4].Speed < 0.0f); + TestEqual(TEXT("look-ahead speed = |next - this| cm / (dt * 100)"), (*Derived)[0][0].Speed, + static_cast(FVector(10.0, 10.0, 0.0).Size()) / (TimeBetweenSteps * 100.0f)); + TestEqual(TEXT("last frame carries the previous heading"), (*Derived)[5][2].Rotation.Yaw, -90.0); + TestTrue(TEXT("last frame carries the previous heading (bitwise)"), RotatorsBitEqual((*Derived)[5][2].Rotation, (*Derived)[3][1].Rotation)); + TestEqual(TEXT("sub-threshold step holds the heading"), (*Derived)[3][2].Rotation.Yaw, -90.0); + TestTrue(TEXT("sub-threshold step holds the heading (bitwise)"), RotatorsBitEqual((*Derived)[3][2].Rotation, (*Derived)[2][2].Rotation)); + } + + return true; +} + +IMPLEMENT_SIMPLE_AUTOMATION_TEST( + FDerivedMotionScalingTimingTest, + "Mobius.Timing.DerivedMotionScaling", + EAutomationTestFlags::EditorContext | EAutomationTestFlags::ProductFilter) + +bool FDerivedMotionScalingTimingTest::RunTest(const FString& Parameters) +{ + using namespace DerivedMotionTestUtil; + + const FString JuelichPath = FPaths::ConvertRelativePathToFull(FPaths::Combine(FPaths::ProjectPluginsDir(), JuelichFixtureRelative)); + UAgentDataSubsystem* Owner = NewObject(); + const float TimeBetweenSteps = 0.1f; + + // Doubling the agents doubles the samples: the old scan should grow ~4x, the new pass ~2x. + const int32 NumTimesteps = 1000; + const int32 AgentCounts[] = { 500, 1000 }; + double OracleSeconds[2] = {}; + double CandidateSeconds[2] = {}; + for (int32 Size = 0; Size < 2; ++Size) + { + const FSimMap Map = MakeSyntheticPositionsOnly(AgentCounts[Size], NumTimesteps); + const FParity Parity = RunParity(Map, AgentCounts[Size], TimeBetweenSteps, JuelichPath, Owner, &OracleSeconds[Size], &CandidateSeconds[Size]); + TestEqual(FString::Printf(TEXT("%d agents: bit-identical (first mismatch: %s)"), AgentCounts[Size], *Parity.FirstMismatch), Parity.Mismatches, 0); + AddInfo(FString::Printf(TEXT("%d agents x %d timesteps (%d samples): old scan %.3f s, new pass %.4f s, %.0fx"), + AgentCounts[Size], NumTimesteps, Parity.Samples, OracleSeconds[Size], CandidateSeconds[Size], + CandidateSeconds[Size] > 0.0 ? OracleSeconds[Size] / CandidateSeconds[Size] : 0.0)); + } + AddInfo(FString::Printf(TEXT("2x agents -> old scan %.1fx slower, new pass %.1fx slower"), + OracleSeconds[0] > 0.0 ? OracleSeconds[1] / OracleSeconds[0] : 0.0, + CandidateSeconds[0] > 0.0 ? CandidateSeconds[1] / CandidateSeconds[0] : 0.0)); + + // Loose gate, as in MobiusTimingTests: observed is orders of magnitude, so background load can't flake it. + TestTrue(FString::Printf(TEXT("new pass at least 10x the old scan at %d agents (%.4f s vs %.3f s)"), AgentCounts[1], CandidateSeconds[1], OracleSeconds[1]), + CandidateSeconds[1] * 10.0 <= OracleSeconds[1]); + + // Optional: a real positions-only file, through the real import first. + const FString FilePath = FPlatformMisc::GetEnvironmentVariable(TEXT("MOBIUS_DERIVED_MOTION_FILE")); + if (FilePath.IsEmpty()) + { + AddInfo(TEXT("MOBIUS_DERIVED_MOTION_FILE not set - real-file timing skipped")); + return true; + } + if (!TestTrue(FString::Printf(TEXT("MOBIUS_DERIVED_MOTION_FILE exists (%s)"), *FilePath), FPaths::FileExists(FilePath))) + { + return false; + } + + FScopedSimCacheOff CacheOff; + FProcessAgentSimulationDataRunnable Import(FilePath, Owner, /*bAutoStartThread*/ false); + Import.Run(); + TestFalse(TEXT("file import parsed the source, not a cache"), Import.ImportTimings.bUsedFastReload); + TestFalse(TEXT("file is positions-only (no rotation)"), Import.AgentSimulationData.Metadata.bHasRotationData); + TestFalse(TEXT("file is positions-only (no speed)"), Import.AgentSimulationData.Metadata.bHasSpeedData); + + const FMobiusImportTimings& Timings = Import.ImportTimings; + const FSimMap& Imported = *Import.AgentMovementInfoData.SimulationData; + double FileOracleSeconds = 0.0; + double FileCandidateSeconds = 0.0; + const FParity Parity = RunParity(Imported, Import.MaxAgents, Import.TimeBetweenSteps, FilePath, Owner, &FileOracleSeconds, &FileCandidateSeconds); + TestEqual(FString::Printf(TEXT("file: bit-identical (first mismatch: %s)"), *Parity.FirstMismatch), Parity.Mismatches, 0); + AddInfo(FString::Printf(TEXT("%s: %d agents, %d timesteps, %d samples. Import %.2f s (parse %.2f, convert %.2f, derive %.3f, brackets %.2f). Derivation on the imported map: old scan %.2f s, new pass %.3f s, %.0fx"), + *FPaths::GetCleanFilename(FilePath), Import.MaxAgents, Imported.Num(), Parity.Samples, + Timings.TotalSeconds, Timings.ParseSeconds, Timings.ConvertSeconds, Timings.DeriveSeconds, Timings.BracketSeconds, + FileOracleSeconds, FileCandidateSeconds, FileCandidateSeconds > 0.0 ? FileOracleSeconds / FileCandidateSeconds : 0.0)); + return true; +} + +#endif // !UE_BUILD_SHIPPING diff --git a/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/LogWindowPerfTest.cpp b/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/LogWindowPerfTest.cpp new file mode 100644 index 000000000..9d7bcf6b6 --- /dev/null +++ b/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/LogWindowPerfTest.cpp @@ -0,0 +1,594 @@ +// Copyright (c) 2026 ProjectMobius contributors. Licensed under MIT. +// +// LogWindowPerfTest.cpp +// +// Frame cost of the Mobius log window (SLogWindowWidget) once its backlog is full - the "Mobius gets +// slow after 30+ minutes with the log window open" report. Against a 1000-line backlog it measures: +// - idle : a redraw with nothing new (the window's standing per-frame cost) +// - trickle : one line per frame (the low-FPS watchdog and other steady producers) +// - burst N : N lines in one logger flush, then the next two frames. +// UMobiusCustomLoggerSubsystem::FlushToDisk broadcasts every queued line separately, +// 4x a second, so a heatmap load (~3,600 CreateMeshSection lines) lands as one burst. +// Each scenario times AppendLine (what the logger flush pays, synchronously) and +// FSlateApplication::ForceRedrawWindow on the log window alone - prepass, widget tick and paint, which +// is where text layout and list-row generation actually happen. Timing only AppendLine, or only the +// redraw, would hide whichever half a given implementation defers. +// +// Uses only the widget's public API and finds the window by title, so this one file measures any +// implementation of the window, before and after a change. Every scenario also checks the view is +// still pinned to the newest line - a cheap frame that stopped showing the end would not be a win. +// +// FollowsEndUnlessScrolledUp is the behavioural half: a reader who scrolls up (real wheel events routed +// through FSlateApplication, so they count as input) is not yanked back by new lines or a backlog trim, +// and following resumes once they scroll back down to the end. +// +// Needs a real renderer. Under -nullrhi Slate skips widget drawing entirely +// (GSlateSkipWidgetDrawingInHeadlessMode), so the redraw would time nothing; the test fails rather than +// report that. Run: +// MobiusPerf\RunTests.ps1 -Filter "Mobius.RenderedTiming.LogWindow." -Rendered +// Named "Mobius.RenderedTiming." - deliberately NOT containing "ProjectMobius." or "Mobius.Timing." - so +// neither the default correctness run nor -IncludeTiming picks it up: both run under -nullrhi, where these +// tests can only fail. Rows are appended to +// /MobiusPerfTimings/logwindow.csv; no absolute number is asserted. +// +#if !UE_BUILD_SHIPPING + +#include "CoreMinimal.h" +#include "Framework/Application/SlateApplication.h" +#include "HAL/FileManager.h" +#include "HAL/PlatformProcess.h" +#include "HAL/PlatformTime.h" +#include "Logging/LogWindow.h" +#include "Misc/App.h" +#include "Misc/AutomationTest.h" +#include "Misc/DateTime.h" +#include "Misc/FileHelper.h" +#include "Misc/Paths.h" +#include "Input/Events.h" +#include "Layout/WidgetPath.h" +#include "Rendering/SlateRenderer.h" +#include "Widgets/SWindow.h" +#include "Widgets/Text/STextBlock.h" + +namespace MobiusLogWindowTiming +{ + const FString LogWindowTitle = TEXT("Mobius Log"); + constexpr int32 BacklogLines = 1000; + + /** + * Two real line shapes from MobiusCustomLog.txt, alternated so the mean (~92 chars) sits near the + * ~89-char mean of the 1.26 M-line log on the dev machine. The index makes every line distinct. + */ + FString MakeLine(int32 Index) + { + if (Index % 2 == 0) + { + return FString::Printf(TEXT("[2026-10-22 11:23:45.%03d] [Heatmap floor=0] CreateMeshSection section=%d verts=1056 tris=1984 in 0.05 ms"), + Index % 1000, Index); + } + return FString::Printf(TEXT("[2026-10-22 11:23:45.%03d] Low FPS detected: 27.4 FPS (threshold 30.0) line %d"), + Index % 1000, Index); + } + + /** The "Mobius Log" window that was not already up before the widget was constructed. */ + TSharedPtr FindNewLogWindow(const TSet& Before) + { + for (const TSharedRef& Window : FSlateApplication::Get().GetTopLevelWindows()) + { + if (!Before.Contains(&Window.Get()) && Window->GetTitle().ToString().Equals(LogWindowTitle)) + { + return Window; + } + } + return nullptr; + } + + /** Drain the render thread. Through the Slate renderer (FSlateRHIRenderer::FlushCommands is + * FlushRenderingCommands) so this module needs no RenderCore link dependency. */ + void FlushSlateRendering() + { + if (FSlateRenderer* Renderer = FSlateApplication::Get().GetRenderer()) + { + Renderer->FlushCommands(); + } + } + + /** + * FSlateApplicationBase::UpdateAnyActiveTimersArePending is protected; a real Slate frame calls it + * once before drawing, and an active timer only runs in a paint that follows it. ForceRedrawWindow + * alone would therefore never run the window's timers. Re-exposed here - not re-implemented - so a + * redraw below behaves like a real frame. + */ + struct FActiveTimerPump : FSlateApplicationBase + { + using FSlateApplicationBase::UpdateAnyActiveTimersArePending; + }; + + void PumpActiveTimers() + { + FSlateApplicationBase& App = FSlateApplication::Get(); + (App.*(&FActiveTimerPump::UpdateAnyActiveTimersArePending))(); + } + + /** Game-thread cost of one frame of this window: active-timer pump + prepass + tick + paint. + * Render-thread work is flushed outside the timer so it neither counts nor piles up across + * iterations (a full draw-buffer pool would otherwise make the next draw block inside the timer). */ + double RedrawMs(const TSharedRef& Window) + { + FlushSlateRendering(); + const double Start = FPlatformTime::Seconds(); + PumpActiveTimers(); + FSlateApplication::Get().ForceRedrawWindow(Window); + const double Ms = (FPlatformTime::Seconds() - Start) * 1000.0; + FlushSlateRendering(); + return Ms; + } + + int32 CountTextBlocks(const TSharedRef& Root) + { + int32 Count = Root->GetType() == TEXT("STextBlock") ? 1 : 0; + FChildren* Children = Root->GetChildren(); + for (int32 Index = 0; Children && Index < Children->Num(); ++Index) + { + Count += CountTextBlocks(Children->GetChildAt(Index)); + } + return Count; + } + + /** First widget, depth-first, whose type name starts with Prefix ("SListView<...>" is templated). */ + TSharedPtr FindWidgetOfType(const TSharedRef& Root, const TCHAR* Prefix) + { + if (Root->GetType().ToString().StartsWith(Prefix)) + { + return Root; + } + FChildren* Children = Root->GetChildren(); + for (int32 Index = 0; Children && Index < Children->Num(); ++Index) + { + if (const TSharedPtr Found = FindWidgetOfType(Children->GetChildAt(Index), Prefix)) + { + return Found; + } + } + return nullptr; + } + + /** A painted STextBlock ending with Line. EndsWith, not Equals, so it matches both a single + * whole-backlog text block and a one-row-per-line list. */ + TSharedPtr FindPaintedTextEndingWith(const TSharedRef& Root, const FString& Line) + { + if (Root->GetType() == TEXT("STextBlock") + && StaticCastSharedRef(Root)->GetText().ToString().EndsWith(Line) + && Root->GetPaintSpaceGeometry().GetLocalSize().Y > 0.0) + { + return Root; + } + FChildren* Children = Root->GetChildren(); + for (int32 Index = 0; Children && Index < Children->Num(); ++Index) + { + if (const TSharedPtr Found = FindPaintedTextEndingWith(Children->GetChildAt(Index), Line)) + { + return Found; + } + } + return nullptr; + } + + double AbsoluteTop(const FGeometry& Geometry) + { + return Geometry.LocalToAbsolute(FVector2D::ZeroVector).Y; + } + + double AbsoluteBottom(const FGeometry& Geometry) + { + return Geometry.LocalToAbsolute(FVector2D(0.0, Geometry.GetLocalSize().Y)).Y; + } + + /** The scrolling viewport: the list (one row per line) or the scroll box (one whole-backlog text). */ + TSharedPtr FindLogViewport(const TSharedRef& Window) + { + const TSharedPtr List = FindWidgetOfType(Window, TEXT("SListView")); + return List.IsValid() ? List : FindWidgetOfType(Window, TEXT("SScrollBox")); + } + + /** + * Line is painted with its bottom on the viewport's bottom edge: the view is pinned to it, not merely + * holding a row for it somewhere below the fold. (Holding-but-not-showing is exactly what a freshly + * generated wrapping row does if nothing re-pins once it learns its wrap width.) + */ + bool IsLineAtViewportBottom(const TSharedRef& Window, const FString& Line, FString& OutDetail) + { + const TSharedPtr Viewport = FindLogViewport(Window); + if (!Viewport.IsValid()) + { + OutDetail = TEXT("no list or scroll box in the log window"); + return false; + } + const TSharedPtr Text = FindPaintedTextEndingWith(Viewport.ToSharedRef(), Line); + if (!Text.IsValid()) + { + OutDetail = TEXT("the line has no painted text block (not generated, or never drawn)"); + return false; + } + const double ViewportBottom = AbsoluteBottom(Viewport->GetPaintSpaceGeometry()); + const double TextBottom = AbsoluteBottom(Text->GetPaintSpaceGeometry()); + OutDetail = FString::Printf(TEXT("line bottom %.1f px, viewport bottom %.1f px"), TextBottom, ViewportBottom); + return FMath::Abs(TextBottom - ViewportBottom) <= 3.0; + } + + /** Texts of the painted text blocks lying wholly inside the viewport, top to bottom. */ + void CollectVisibleLines(const TSharedRef& Root, double ViewportTop, double ViewportBottom, TArray& OutLines) + { + if (Root->GetType() == TEXT("STextBlock")) + { + const FGeometry& Geometry = Root->GetPaintSpaceGeometry(); + if (Geometry.GetLocalSize().Y > 0.0 + && AbsoluteTop(Geometry) >= ViewportTop - 0.5 && AbsoluteBottom(Geometry) <= ViewportBottom + 0.5) + { + OutLines.Add(StaticCastSharedRef(Root)->GetText().ToString()); + } + return; + } + FChildren* Children = Root->GetChildren(); + for (int32 Index = 0; Children && Index < Children->Num(); ++Index) + { + CollectVisibleLines(Children->GetChildAt(Index), ViewportTop, ViewportBottom, OutLines); + } + } + + /** One wheel notch (or a big flick) routed to Target the way FSlateApplication routes real input - + * inside its input-processing scope, which is what the window keys "the reader scrolled" off. */ + bool RouteWheel(const TSharedRef& Target, float WheelDelta) + { + FWidgetPath FoundPath; + if (!FSlateApplication::Get().FindPathToWidget(Target, FoundPath)) + { + return false; + } + // FindPathToWidget leaves the per-widget virtual pointer positions empty, and pointer routing + // indexes them, so rebuild the path through the constructor that fills them (unset = no 3D widget). + TArray WidgetsAndPointers; + for (int32 Index = 0; Index < FoundPath.Widgets.Num(); ++Index) + { + WidgetsAndPointers.Add(FWidgetAndPointer(FoundPath.Widgets[Index])); + } + const FWidgetPath Path(WidgetsAndPointers); + + const FVector2D Centre = Target->GetPaintSpaceGeometry().GetAbsolutePositionAtCoordinates(FVector2D(0.5, 0.5)); + const FPointerEvent Wheel(/*PointerIndex*/ 0, Centre, Centre, TSet(), FKey(), WheelDelta, FModifierKeysState()); + return FSlateApplication::Get().RouteMouseWheelOrGestureEvent(Path, Wheel, nullptr).IsEventHandled(); + } + + double Median(TArray Values) + { + if (Values.Num() == 0) + { + return 0.0; + } + Values.Sort(); + const int32 Mid = Values.Num() / 2; + return (Values.Num() % 2 == 1) ? Values[Mid] : 0.5 * (Values[Mid - 1] + Values[Mid]); + } + + double MaxOf(const TArray& Values) + { + double Result = 0.0; + for (const double Value : Values) + { + Result = FMath::Max(Result, Value); + } + return Result; + } + + struct FScenarioResult + { + FString Scenario; + int32 LinesPerIteration = 0; + int32 Iterations = 0; + double AppendMedianMs = 0.0; + double Frame1MedianMs = 0.0; + double Frame2MedianMs = 0.0; + double TotalMedianMs = 0.0; + double TotalMaxMs = 0.0; + }; + + /** Append one row per scenario to the trend CSV (header written on first use). */ + void AppendCsvRow(const FScenarioResult& Result, int32 TextBlocks) + { + const FString Dir = FPaths::Combine(FPaths::ProjectSavedDir(), TEXT("MobiusPerfTimings")); + const FString CsvPath = FPaths::Combine(Dir, TEXT("logwindow.csv")); + FString Row; + if (!IFileManager::Get().FileExists(*CsvPath)) + { + IFileManager::Get().MakeDirectory(*Dir, /*Tree*/ true); + Row += TEXT("timestamp_utc,machine,scenario,lines_per_iteration,iterations,append_median_ms,frame1_median_ms,frame2_median_ms,total_median_ms,total_max_ms,text_blocks_in_window\n"); + } + Row += FString::Printf(TEXT("%s,%s,%s,%d,%d,%.4f,%.4f,%.4f,%.4f,%.4f,%d\n"), + *FDateTime::UtcNow().ToIso8601(), FPlatformProcess::ComputerName(), *Result.Scenario, + Result.LinesPerIteration, Result.Iterations, Result.AppendMedianMs, Result.Frame1MedianMs, + Result.Frame2MedianMs, Result.TotalMedianMs, Result.TotalMaxMs, TextBlocks); + FFileHelper::SaveStringToFile(Row, *CsvPath, FFileHelper::EEncodingOptions::ForceUTF8WithoutBOM, + &IFileManager::Get(), FILEWRITE_Append); + } +} + +IMPLEMENT_SIMPLE_AUTOMATION_TEST( + FMobiusTimingLogWindowFullBacklogTest, + "Mobius.RenderedTiming.LogWindow.FullBacklogFrameCost", + EAutomationTestFlags::EditorContext | EAutomationTestFlags::ClientContext | EAutomationTestFlags::ProductFilter) + +bool FMobiusTimingLogWindowFullBacklogTest::RunTest(const FString& Parameters) +{ + using namespace MobiusLogWindowTiming; + + if (!FSlateApplication::IsInitialized() || !FApp::CanEverRender()) + { + AddError(TEXT("Needs Slate and a real renderer: under -nullrhi Slate skips widget drawing, so the ") + TEXT("redraw half of every scenario would measure nothing. Run with RunTests.ps1 -Rendered.")); + return false; + } + + TSet WindowsBefore; + for (const TSharedRef& Window : FSlateApplication::Get().GetTopLevelWindows()) + { + WindowsBefore.Add(&Window.Get()); + } + + TArray Results; + int32 TextBlocksAtEnd = 0; + int32 NextLine = 0; + + { + const TSharedRef Widget = SNew(SLogWindowWidget); + const TSharedPtr Window = FindNewLogWindow(WindowsBefore); + if (!Window.IsValid()) + { + AddError(TEXT("Constructing SLogWindowWidget did not open a 'Mobius Log' window.")); + return false; + } + const TSharedRef WindowRef = Window.ToSharedRef(); + + // The backlog the owner's sessions reach after ~30 minutes: the cap, full. + TArray Backlog; + Backlog.Reserve(BacklogLines); + for (; NextLine < BacklogLines; ++NextLine) + { + Backlog.Add(MakeLine(NextLine)); + } + Widget->SetLines(Backlog); + + // Warm-up: the first paint establishes the wrap width, the next re-wraps against it, and any + // scroll-to-end request settles. + for (int32 Warm = 0; Warm < 3; ++Warm) + { + RedrawMs(WindowRef); + } + FString SeedDetail; + if (!IsLineAtViewportBottom(WindowRef, Backlog.Last(), SeedDetail)) + { + AddError(FString::Printf(TEXT("After seeding and three redraws the newest backlog line is not at the ") + TEXT("bottom of the view (%s) - the window is not drawing, or not following the end, so nothing ") + TEXT("below would be a measurement."), *SeedDetail)); + Widget->CloseLogWindow(); + return false; + } + + // --- idle: nothing new ---------------------------------------------------------------------- + { + TArray Frames; + for (int32 Iteration = 0; Iteration < 30; ++Iteration) + { + Frames.Add(RedrawMs(WindowRef)); + } + FScenarioResult& Result = Results.AddDefaulted_GetRef(); + Result.Scenario = TEXT("idle"); + Result.Iterations = Frames.Num(); + Result.Frame1MedianMs = Median(Frames); + Result.TotalMedianMs = Result.Frame1MedianMs; + Result.TotalMaxMs = MaxOf(Frames); + } + + // --- trickle: one line per frame ------------------------------------------------------------ + { + TArray Appends; + TArray Frames; + TArray Totals; + for (int32 Iteration = 0; Iteration < 120; ++Iteration) + { + const FString Line = MakeLine(NextLine++); + const double Start = FPlatformTime::Seconds(); + Widget->AppendLine(Line); + const double AppendMs = (FPlatformTime::Seconds() - Start) * 1000.0; + const double FrameMs = RedrawMs(WindowRef); + Appends.Add(AppendMs); + Frames.Add(FrameMs); + Totals.Add(AppendMs + FrameMs); + } + RedrawMs(WindowRef); + FString Detail; + TestTrue(TEXT("trickle: the view stays pinned to the newest line"), + IsLineAtViewportBottom(WindowRef, MakeLine(NextLine - 1), Detail)); + AddInfo(FString::Printf(TEXT("trickle end: %s"), *Detail)); + + FScenarioResult& Result = Results.AddDefaulted_GetRef(); + Result.Scenario = TEXT("trickle"); + Result.LinesPerIteration = 1; + Result.Iterations = Totals.Num(); + Result.AppendMedianMs = Median(Appends); + Result.Frame1MedianMs = Median(Frames); + Result.TotalMedianMs = Median(Totals); + Result.TotalMaxMs = MaxOf(Totals); + } + + // --- bursts: one logger flush of N lines, then two frames --------------------------------- + struct FBurst { int32 Lines; int32 Iterations; }; + const FBurst Bursts[] = { { 250, 5 }, { 3600, 3 } }; + for (const FBurst& Burst : Bursts) + { + TArray Appends; + TArray Frame1s; + TArray Frame2s; + TArray Totals; + for (int32 Iteration = 0; Iteration < Burst.Iterations; ++Iteration) + { + TArray Lines; + Lines.Reserve(Burst.Lines); + for (int32 Index = 0; Index < Burst.Lines; ++Index) + { + Lines.Add(MakeLine(NextLine++)); + } + + const double Start = FPlatformTime::Seconds(); + for (const FString& Line : Lines) + { + Widget->AppendLine(Line); + } + const double AppendMs = (FPlatformTime::Seconds() - Start) * 1000.0; + const double Frame1Ms = RedrawMs(WindowRef); + const double Frame2Ms = RedrawMs(WindowRef); + + FString Detail; + const bool bPinned = IsLineAtViewportBottom(WindowRef, Lines.Last(), Detail); + TestTrue(FString::Printf(TEXT("burst %d: the view is pinned to the newest line after two frames (%s)"), Burst.Lines, *Detail), + bPinned); + + Appends.Add(AppendMs); + Frame1s.Add(Frame1Ms); + Frame2s.Add(Frame2Ms); + Totals.Add(AppendMs + Frame1Ms + Frame2Ms); + } + + FScenarioResult& Result = Results.AddDefaulted_GetRef(); + Result.Scenario = FString::Printf(TEXT("burst_%d"), Burst.Lines); + Result.LinesPerIteration = Burst.Lines; + Result.Iterations = Totals.Num(); + Result.AppendMedianMs = Median(Appends); + Result.Frame1MedianMs = Median(Frame1s); + Result.Frame2MedianMs = Median(Frame2s); + Result.TotalMedianMs = Median(Totals); + Result.TotalMaxMs = MaxOf(Totals); + } + + // Fingerprint of the implementation under test: a single whole-backlog text block reads ~3 + // (body + Close label + title); a virtualized list reads roughly its visible row count. + TextBlocksAtEnd = CountTextBlocks(WindowRef); + + Widget->CloseLogWindow(); + } + + for (const FScenarioResult& Result : Results) + { + AddInfo(FString::Printf(TEXT("%-10s lines/iter=%4d iters=%3d | append %8.3f ms | frame1 %8.3f ms | frame2 %8.3f ms | total median %8.3f ms, max %8.3f ms"), + *Result.Scenario, Result.LinesPerIteration, Result.Iterations, Result.AppendMedianMs, + Result.Frame1MedianMs, Result.Frame2MedianMs, Result.TotalMedianMs, Result.TotalMaxMs)); + AppendCsvRow(Result, TextBlocksAtEnd); + } + AddInfo(FString::Printf(TEXT("text blocks in the log window at the end: %d"), TextBlocksAtEnd)); + + return true; +} + +IMPLEMENT_SIMPLE_AUTOMATION_TEST( + FMobiusLogWindowFollowsEndUnlessScrolledUpTest, + "Mobius.RenderedTiming.LogWindow.FollowsEndUnlessScrolledUp", + EAutomationTestFlags::EditorContext | EAutomationTestFlags::ClientContext | EAutomationTestFlags::ProductFilter) + +bool FMobiusLogWindowFollowsEndUnlessScrolledUpTest::RunTest(const FString& Parameters) +{ + using namespace MobiusLogWindowTiming; + + if (!FSlateApplication::IsInitialized() || !FApp::CanEverRender()) + { + AddError(TEXT("Needs Slate and a real renderer (geometry only exists once painted). Run with RunTests.ps1 -Rendered.")); + return false; + } + + TSet WindowsBefore; + for (const TSharedRef& Window : FSlateApplication::Get().GetTopLevelWindows()) + { + WindowsBefore.Add(&Window.Get()); + } + + { + const TSharedRef Widget = SNew(SLogWindowWidget); + const TSharedPtr Window = FindNewLogWindow(WindowsBefore); + if (!Window.IsValid()) + { + AddError(TEXT("Constructing SLogWindowWidget did not open a 'Mobius Log' window.")); + return false; + } + const TSharedRef WindowRef = Window.ToSharedRef(); + const auto Redraw = [&WindowRef](int32 Frames) + { + for (int32 Frame = 0; Frame < Frames; ++Frame) + { + RedrawMs(WindowRef); + } + }; + + int32 NextLine = 0; + TArray Backlog; + for (; NextLine < BacklogLines; ++NextLine) + { + Backlog.Add(MakeLine(NextLine)); + } + Widget->SetLines(Backlog); + Redraw(4); + + FString Detail; + TestTrue(TEXT("opens pinned to the newest line"), IsLineAtViewportBottom(WindowRef, Backlog.Last(), Detail)); + AddInfo(FString::Printf(TEXT("open: %s"), *Detail)); + + const TSharedPtr List = FindWidgetOfType(WindowRef, TEXT("SListView")); + if (!List.IsValid()) + { + AddError(TEXT("The log window has no SListView - this test drives the virtualized list.")); + Widget->CloseLogWindow(); + return false; + } + + // --- the reader scrolls up a few notches ---------------------------------------------------- + for (int32 Notch = 0; Notch < 5; ++Notch) + { + TestTrue(TEXT("wheel-up is handled by the list"), RouteWheel(List.ToSharedRef(), 1.0f)); + Redraw(2); + } + const double ViewportTop = AbsoluteTop(List->GetPaintSpaceGeometry()); + const double ViewportBottom = AbsoluteBottom(List->GetPaintSpaceGeometry()); + TArray VisibleBefore; + CollectVisibleLines(List.ToSharedRef(), ViewportTop, ViewportBottom, VisibleBefore); + TestTrue(TEXT("scrolled up: some lines are on screen"), VisibleBefore.Num() > 0); + TestFalse(TEXT("scrolled up: the newest line is no longer at the bottom"), IsLineAtViewportBottom(WindowRef, Backlog.Last(), Detail)); + + // --- new lines arrive - enough to overflow the cap and trim the front ------------------------ + FString Newest; + for (int32 Index = 0; Index < 200; ++Index) + { + Newest = MakeLine(NextLine++); + Widget->AppendLine(Newest); + } + Redraw(5); + TestFalse(TEXT("a scrolled-up reader is not yanked to the newest line"), IsLineAtViewportBottom(WindowRef, Newest, Detail)); + TArray VisibleAfter; + CollectVisibleLines(List.ToSharedRef(), ViewportTop, ViewportBottom, VisibleAfter); + if (VisibleBefore.Num() > 0) + { + TestTrue(TEXT("the lines being read stay on screen through new lines and a front trim"), + VisibleAfter.Contains(VisibleBefore[VisibleBefore.Num() / 2])); + } + + // --- the reader scrolls back down to the end: following resumes ----------------------------- + TestTrue(TEXT("wheel-down is handled by the list"), RouteWheel(List.ToSharedRef(), -1000.0f)); + Redraw(3); + Newest = MakeLine(NextLine++); + Widget->AppendLine(Newest); + Redraw(4); + TestTrue(TEXT("after scrolling back to the end, new lines are followed again"), IsLineAtViewportBottom(WindowRef, Newest, Detail)); + AddInfo(FString::Printf(TEXT("resumed: %s"), *Detail)); + + Widget->CloseLogWindow(); + } + return true; +} + +#endif // !UE_BUILD_SHIPPING diff --git a/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/MobiusLoggerInGameTest.cpp b/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/MobiusLoggerInGameTest.cpp new file mode 100644 index 000000000..5228d7afe --- /dev/null +++ b/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/MobiusLoggerInGameTest.cpp @@ -0,0 +1,240 @@ +// Copyright (c) 2026 ProjectMobius contributors. Licensed under MIT. +// +// MobiusLoggerInGameTest.cpp +// +// In-game gates for the custom logger path +// (UMobiusCustomLoggerSubsystem -> UMobiusUserFeedbackSubsystem -> log window). +// +// - FrameGrabberRetryIsQuiet. UFrameGrabberHelper::Tick retries TryInitialize until the game viewport +// is usable. Under -game -nullrhi - every headless in-game automation run - the viewport stays 0x0 for +// the whole session, and the helper used to log "[FrameGrabber] TryInitialize failed: Invalid +// viewport size 0x0" on every frame: 1,070,058 of the 1,259,848 lines in the dev machine's +// MobiusCustomLog.txt, at up to ~3,000 lines/s. This counts those lines over a fixed idle window and +// fails above a small bound. Where the viewport IS valid the count is ~0 with or without the fix, so +// only a -nullrhi run discriminates; the test reports which viewport it saw. +// +// - FeedbackCacheKeepsNewestOldestFirst. UMobiusUserFeedbackSubsystem caches the newest lines to seed +// a (re)opened log window. It is a ring buffer; this pushes 1.5x its capacity of marker lines through +// the REAL logger flush and checks GetCachedLogLines() returns exactly the newest ones, oldest first - +// the wrap-around seam is where a ring buffer gets the order wrong. +// +// Both force logging on for their duration (it is a persisted user toggle and may be off) and restore it. +// ClientContext only: they need a game world and its GameInstance. Run: +// UnrealEditor-Cmd -game -nullrhi -unattended -ExecCmds="Automation RunTests Mobius.InGame.Logger.; Quit" +// Named "Mobius.InGame." so the default correctness filter never runs them. +// +#if !UE_BUILD_SHIPPING + +#include "CoreMinimal.h" +#include "Engine/Engine.h" +#include "Engine/GameInstance.h" +#include "Engine/GameViewportClient.h" +#include "Engine/World.h" +#include "HAL/PlatformTime.h" +#include "Misc/AutomationTest.h" +#include "Misc/Guid.h" +#include "Subsystems/MobiusCustomLoggerSubsystem.h" +#include "Subsystems/MobiusUserFeedbackSubsystem.h" +#include "Tests/AutomationCommon.h" +#include "UnrealClient.h" + +namespace MobiusLoggerInGameTest +{ + /** Idle window for the FrameGrabber count. At the ~120 fps a -nullrhi run reaches, the pre-fix code + * logs well over a thousand lines in this time. */ + constexpr float IdleSeconds = 10.0f; + + /** Generous: the fixed helper logs a failure reason once, when it CHANGES - normally before this + * test even starts, so 0 lines here. */ + constexpr int32 MaxFrameGrabberFailureLines = 3; + + /** UMobiusUserFeedbackSubsystem::MaxCachedLogLines (private). If that changes, so must this. */ + constexpr int32 FeedbackCacheCapacity = 1000; + + UWorld* FindGameWorld() + { + for (const FWorldContext& Context : GEngine->GetWorldContexts()) + { + if ((Context.WorldType == EWorldType::Game || Context.WorldType == EWorldType::PIE) && Context.World()) + { + return Context.World(); + } + } + return nullptr; + } + + FIntPoint GameViewportSize() + { + if (GEngine && GEngine->GameViewport && GEngine->GameViewport->Viewport) + { + return GEngine->GameViewport->Viewport->GetSizeXY(); + } + return FIntPoint::ZeroValue; + } + + struct FFrameGrabberSpamState + { + int32 FrameGrabberFailureLines = 0; + int32 AllLines = 0; + FDelegateHandle LogLineHandle; + bool bLoggingWasEnabled = true; + double StartSeconds = 0.0; + }; +} + +IMPLEMENT_SIMPLE_AUTOMATION_TEST( + FMobiusLoggerFrameGrabberRetryIsQuietTest, + "Mobius.InGame.Logger.FrameGrabberRetryIsQuiet", + EAutomationTestFlags::ClientContext | EAutomationTestFlags::ProductFilter) + +bool FMobiusLoggerFrameGrabberRetryIsQuietTest::RunTest(const FString& Parameters) +{ + using namespace MobiusLoggerInGameTest; + + if (!FindGameWorld()) + { + AddError(TEXT("no game world - run under UnrealEditor-Cmd -game")); + return false; + } + UMobiusCustomLoggerSubsystem* Logger = GEngine->GetEngineSubsystem(); + if (!Logger) + { + AddError(TEXT("UMobiusCustomLoggerSubsystem is missing")); + return false; + } + + const TSharedRef State = MakeShared(); + // SetLoggingEnabled UE_LOGs a warning, so only touch it when logging is actually off. + State->bLoggingWasEnabled = Logger->IsLoggingEnabled(); + if (!State->bLoggingWasEnabled) + { + Logger->SetLoggingEnabled(true); + } + + // Deliver everything queued so far first, so only lines produced inside the window are counted. + Logger->FlushQueuedMessages(); + State->LogLineHandle = Logger->OnLogLine().AddLambda([State](const FString& Line) + { + ++State->AllLines; + if (Line.Contains(TEXT("[FrameGrabber] TryInitialize failed"))) + { + ++State->FrameGrabberFailureLines; + } + }); + State->StartSeconds = FPlatformTime::Seconds(); + + ADD_LATENT_AUTOMATION_COMMAND(FEngineWaitLatentCommand(IdleSeconds)); + ADD_LATENT_AUTOMATION_COMMAND(FFunctionLatentCommand([this, State]() + { + UMobiusCustomLoggerSubsystem* LoggerNow = GEngine->GetEngineSubsystem(); + if (!LoggerNow) + { + AddError(TEXT("UMobiusCustomLoggerSubsystem went away during the idle window")); + return true; + } + LoggerNow->FlushQueuedMessages(); + LoggerNow->OnLogLine().Remove(State->LogLineHandle); + const double Elapsed = FPlatformTime::Seconds() - State->StartSeconds; + const FIntPoint Viewport = GameViewportSize(); + + AddInfo(FString::Printf(TEXT("%d '[FrameGrabber] TryInitialize failed' lines in %.1f s (%.1f/s), %d logger lines in total; game viewport %dx%d%s"), + State->FrameGrabberFailureLines, Elapsed, Elapsed > 0.0 ? State->FrameGrabberFailureLines / Elapsed : 0.0, + State->AllLines, Viewport.X, Viewport.Y, + (Viewport.X > 0 && Viewport.Y > 0) ? TEXT(" (valid viewport: this run cannot reproduce the 0x0 retry spam)") : TEXT(""))); + + TestTrue(FString::Printf(TEXT("FrameGrabber logged at most %d TryInitialize failures in %.0f s idle"), MaxFrameGrabberFailureLines, IdleSeconds), + State->FrameGrabberFailureLines <= MaxFrameGrabberFailureLines); + + if (!State->bLoggingWasEnabled) + { + LoggerNow->SetLoggingEnabled(false); + } + return true; + })); + return true; +} + +IMPLEMENT_SIMPLE_AUTOMATION_TEST( + FMobiusLoggerFeedbackCacheOrderTest, + "Mobius.InGame.Logger.FeedbackCacheKeepsNewestOldestFirst", + EAutomationTestFlags::ClientContext | EAutomationTestFlags::ProductFilter) + +bool FMobiusLoggerFeedbackCacheOrderTest::RunTest(const FString& Parameters) +{ + using namespace MobiusLoggerInGameTest; + + UWorld* World = FindGameWorld(); + UGameInstance* GameInstance = World ? World->GetGameInstance() : nullptr; + UMobiusUserFeedbackSubsystem* Feedback = GameInstance ? GameInstance->GetSubsystem() : nullptr; + UMobiusCustomLoggerSubsystem* Logger = GEngine->GetEngineSubsystem(); + if (!Feedback || !Logger) + { + AddError(TEXT("needs a game world with UMobiusUserFeedbackSubsystem and the custom logger - run under UnrealEditor-Cmd -game")); + return false; + } + + const bool bLoggingWasEnabled = Logger->IsLoggingEnabled(); + if (!bLoggingWasEnabled) + { + Logger->SetLoggingEnabled(true); + } + Logger->FlushQueuedMessages(); + + // 1.5x capacity, through the real path: EnqueueLogMessage -> FlushToDisk broadcast -> HandleLogLine. + // Unique tag so lines from any other run (or another producer mid-test) cannot be mistaken for ours. + const FString Tag = FString::Printf(TEXT("[LoggerCacheTest %s]"), *FGuid::NewGuid().ToString(EGuidFormats::Short)); + const int32 Pushed = FeedbackCacheCapacity + FeedbackCacheCapacity / 2; + for (int32 Index = 0; Index < Pushed; ++Index) + { + Logger->EnqueueLogMessage(FString::Printf(TEXT("%s marker %05d"), *Tag, Index)); + } + Logger->FlushQueuedMessages(); + + const TArray Cached = Feedback->GetCachedLogLines(); + if (!bLoggingWasEnabled) + { + Logger->SetLoggingEnabled(false); + } + + TestEqual(TEXT("cache holds exactly its capacity"), Cached.Num(), FeedbackCacheCapacity); + if (Cached.Num() == 0) + { + return false; + } + + // Marker indices in cache order. + TArray Markers; + const FString MarkerPrefix = Tag + TEXT(" marker "); + for (const FString& Line : Cached) + { + const int32 At = Line.Find(MarkerPrefix); + if (At != INDEX_NONE) + { + Markers.Add(FCString::Atoi(*Line.Mid(At + MarkerPrefix.Len()))); + } + } + + TestTrue(TEXT("the newest cached line is the newest marker"), + Cached.Last().EndsWith(FString::Printf(TEXT("marker %05d"), Pushed - 1))); + bool bStrictlyIncreasing = Markers.Num() > 0; + for (int32 Index = 1; Index < Markers.Num(); ++Index) + { + bStrictlyIncreasing &= Markers[Index] == Markers[Index - 1] + 1; + } + TestTrue(TEXT("markers come back oldest first and contiguous - no seam where the ring wrapped"), bStrictlyIncreasing); + if (Markers.Num() > 0) + { + TestTrue(TEXT("the oldest markers were evicted"), Markers[0] >= Pushed - FeedbackCacheCapacity); + } + // Nothing else is normally logged between the two flushes, in which case the cache is exactly the + // newest FeedbackCacheCapacity markers. + if (Markers.Num() == FeedbackCacheCapacity) + { + TestEqual(TEXT("first cached line is marker Pushed - capacity"), Markers[0], Pushed - FeedbackCacheCapacity); + } + AddInfo(FString::Printf(TEXT("cache %d lines, %d of them markers, first marker %d, last %d"), + Cached.Num(), Markers.Num(), Markers.Num() ? Markers[0] : -1, Markers.Num() ? Markers.Last() : -1)); + return true; +} + +#endif // !UE_BUILD_SHIPPING diff --git a/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/MobiusLoggerSessionFileTest.cpp b/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/MobiusLoggerSessionFileTest.cpp new file mode 100644 index 000000000..a78e3d724 --- /dev/null +++ b/UnrealFolder/ProjectMobius/Source/ProjectMobiusTests/Private/Tests/MobiusLoggerSessionFileTest.cpp @@ -0,0 +1,107 @@ +// Copyright (c) 2026 ProjectMobius contributors. Licensed under MIT. +// +// MobiusLoggerSessionFileTest.cpp +// +// UMobiusCustomLoggerSubsystem writes one file per session (process), MobiusCustomLog_.txt, +// instead of appending every session forever to one MobiusCustomLog.txt (115 MB on the dev machine, +// with side-by-side processes interleaved in it). Checks, against THIS process's logger: +// - the file is per-session by name: MobiusCustomLog_yyyy.mm.dd-hh.mm.ss[_n].txt, never the old name; +// - it starts with the "=== Mobius session started