Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Project Mobius is an Unreal Engine 5.5 application for loading pedestrian trajec
## Why HDF5

For larger trajectory files, convert JSON to HDF5 with the bundled
[`json_to_hdf5_converter.py`](UnrealFolder/ProjectMobius/Plugins/Hdf5DataPlugin/Scripts/json_to_hdf5_converter.py)
[`json_to_hdf5_converter.py`](UnrealFolder/ProjectMobius/Plugins/MobiusDataImporter/Scripts/json_to_hdf5_converter.py)
script. In one 5,000-agent sample, HDF5 cut load time from `21.29 s` to `3.18 s`
and reduced memory use from `317,555 KB` to `55,283 KB`.

Expand All @@ -88,6 +88,11 @@ and reduced memory use from `317,555 KB` to `55,283 KB`.
More on supported formats, the conversion command, and the benchmark:
[docs/DATA-PIPELINE.md](docs/DATA-PIPELINE.md)

To export trajectories from your own evacuation or crowd model, see
[Pedestrian Data Format](docs/PEDESTRIAN-DATA-FORMAT.md) for the exact file layout and the
[HDF5 Export Guide](docs/HDF5-EXPORT-GUIDE.md) for writing it from Python or with the bundled
HDF5 C library ([buildingEXODUS users](docs/EXODUS-EXPORT.md)).

## Quick Start

### Prerequisites
Expand Down Expand Up @@ -169,6 +174,9 @@ Full build, packaging, and automation notes live in
- [Architecture](docs/ARCHITECTURE.md)
- [Data Pipeline](docs/DATA-PIPELINE.md)
- [Agent Sample Data](docs/AGENT-SAMPLE-DATA.md)
- [Pedestrian Data Format](docs/PEDESTRIAN-DATA-FORMAT.md)
- [HDF5 Export Guide](docs/HDF5-EXPORT-GUIDE.md)
- [Exporting from buildingEXODUS](docs/EXODUS-EXPORT.md)
- [Research and Publications](docs/RESEARCH.md)
- [UE Automation Guide](UnrealFolder/ProjectMobius/BuildDocs/UE-Automation-Guide.md)
- [Contributing](CONTRIBUTING.md)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,17 @@

## HDF5

> **Writing pedestrian data from another simulator?** You can build this HDF5 source on its own,
> without Unreal or the superbuild: see `docs/HDF5-EXPORT-GUIDE.md` at the repository root.

## Layout

- `hdf5-2.0.0/` — full upstream HDF5 2.0.0 source (CMake-only since 2.0; autotools was dropped
upstream). Kept in-tree so the prebuilt libraries below can be rebuilt without hunting down the
exact upstream snapshot.
- `hdf5-2.0.0/install/` — the build output the UE module actually consumes:
- `include/` — public headers, **including the generated `H5pubconf.h`** (deliberately tracked so
a fresh clone compiles without running CMake; it carries harmless machine stamps from whoever
generated it last).
upstream). This source is what git tracks; the libraries below are built from it.
- `hdf5-2.0.0/install/` — the build output the UE module actually consumes. **None of it is in
git** (see `.gitignore`): a fresh clone gets it by running the superbuild
(`python superbuild.py` from `UnrealFolder/ProjectMobius`).
- `include/` — public headers, including the generated `H5pubconf.h`.
- `lib/libhdf5.lib`, `lib/libhdf5_hl.lib`, `lib/zlib-static.lib` — static libs, linked by
`UeHdf5Library.Build.cs`.
- `bin/` — MSVC runtime redistributables (`UeHdf5Library.Build.cs` checks the dir exists; nothing
Expand All @@ -33,7 +35,7 @@

## What is intentionally NOT tracked (see repo `.gitignore`)

`install/cmake/`, `install/lib/pkgconfig/`, `install/lib/libhdf5.settings`, `hdf5-2.0.0/build*` —
The whole `install/` tree and `hdf5-2.0.0/build*` —
machine-generated at configure/install time and re-stamped with absolute local paths on every
`cmake --install`. If they show up as modified/untracked after a rebuild, that is expected.

Expand Down Expand Up @@ -66,8 +68,7 @@ the Windows one for the Mac libs.
Notes:
- `ZLIB_USE_LOCALCONTENT=OFF` makes the superbuild download the zlib TGZ at configure time
(URLs in `config/CacheURLs.cmake`) — network access required.
- After an install, only `install/include/H5pubconf.h` may show a real tracked diff (path/date
stamps). Revert it unless the rebuild was intentional; commit it together with the new libs when
it was.
- `install/` is regenerated as a whole (headers, including the platform-generated `H5pubconf.h`,
and libs together), so nothing under it needs reverting or committing after a rebuild.
- Keep the record layout expectations of the importer in mind: the libs must stay ABI-compatible
with `H5pubconf.h` as committed.
with the `H5pubconf.h` installed beside them.
Original file line number Diff line number Diff line change
Expand Up @@ -71,10 +71,12 @@ Reporting a problem
Please send these two files along with what you were doing:

~/Library/Logs/ProjectMobius/ProjectMobius.log
~/Library/Logs/ProjectMobius/MobiusCustomLog.txt
~/Library/Logs/ProjectMobius/MobiusCustomLog_<date-time>.txt (the newest one)

ProjectMobius.log is replaced each time the app starts, so copy it before reopening
the app after a problem. Older runs are kept beside it as ProjectMobius-backup-*.log.
MobiusCustomLog_<date-time>.txt is a new file for each run, named after when that run
started; the last 20 are kept.


Licences
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,48 @@ namespace
{
return GEngine ? GEngine->GetEngineSubsystem<UMobiusCustomLoggerSubsystem>() : nullptr;
}

/** Per-session files are MobiusCustomLog_<yyyy.mm.dd-hh.mm.ss>[_n].txt; the stamp makes name order chronological. */
const TCHAR* const SessionLogPrefix = TEXT("MobiusCustomLog_");

/** Session files kept, the current one included. The pre-rotation MobiusCustomLog.txt never matches. */
constexpr int32 MaxSessionLogFiles = 20;

FString MakeSessionLogFilePath(const FString& LogDir, const FDateTime& SessionStart)
{
const FString Stamp = SessionStart.ToString(TEXT("%Y.%m.%d-%H.%M.%S"));
FString Path = FPaths::Combine(LogDir, FString::Printf(TEXT("%s%s.txt"), SessionLogPrefix, *Stamp));

// Two processes started in the same second (an editor and a scripted headless run, say) get a file
// each rather than interleaving into one.
for (int32 Suffix = 2; IFileManager::Get().FileExists(*Path); ++Suffix)
{
Path = FPaths::Combine(LogDir, FString::Printf(TEXT("%s%s_%d.txt"), SessionLogPrefix, *Stamp, Suffix));
}
return Path;
}

/** Delete the oldest session files beyond MaxSessionLogFiles. Returns how many went. */
int32 PruneOldSessionLogs(const FString& LogDir, const FString& CurrentLogPath)
{
TArray<FString> SessionFiles;
IFileManager::Get().FindFiles(SessionFiles, *FPaths::Combine(LogDir, FString(SessionLogPrefix) + TEXT("*.txt")),
/*Files=*/true, /*Directories=*/false);
SessionFiles.Remove(FPaths::GetCleanFilename(CurrentLogPath));
SessionFiles.Sort();

const int32 ToDelete = SessionFiles.Num() - (MaxSessionLogFiles - 1);
int32 Deleted = 0;
for (int32 Index = 0; Index < ToDelete; ++Index)
{
if (IFileManager::Get().Delete(*FPaths::Combine(LogDir, SessionFiles[Index]), /*RequireExists=*/false,
/*EvenReadOnly=*/false, /*Quiet=*/true))
{
++Deleted;
}
}
return Deleted;
}
}

UMobiusCustomLoggerSubsystem::UMobiusCustomLoggerSubsystem()
Expand All @@ -45,17 +87,36 @@ void UMobiusCustomLoggerSubsystem::Initialize(FSubsystemCollectionBase& Collecti
// Packaged builds expect the log next to the launched executable.
const FString LogDir = FPaths::ConvertRelativePathToFull(FPaths::LaunchDir());
#endif
LogFilePath = FPaths::Combine(LogDir, TEXT("MobiusCustomLog.txt"));

// One file per session (process). A single shared MobiusCustomLog.txt grew without bound (115 MB on the
// dev machine) and interleaved processes running side by side. The newest MaxSessionLogFiles are kept;
// the old shared file is left alone.
IPlatformFile& PlatformFile = FPlatformFileManager::Get().GetPlatformFile();
PlatformFile.CreateDirectoryTree(*FPaths::GetPath(LogFilePath));
PlatformFile.CreateDirectoryTree(*LogDir);

const FDateTime SessionStart = FDateTime::Now();
LogFilePath = MakeSessionLogFilePath(LogDir, SessionStart);

// Claim the name now, so a process starting a moment later sees it taken.
TUniquePtr<IFileHandle> Claim(PlatformFile.OpenWrite(*LogFilePath, /*bAppend=*/true));
Claim.Reset();

const int32 PrunedSessionLogs = PruneOldSessionLogs(LogDir, LogFilePath);

FlushTickerHandle = FTSTicker::GetCoreTicker().AddTicker(
FTickerDelegate::CreateUObject(this, &UMobiusCustomLoggerSubsystem::PumpLogs),
0.25f); // flush 4x per second to keep overhead tiny

UE_LOG(LogTemp, Display, TEXT("MobiusCustomLogger initialised. Writing to %s"), *LogFilePath);

// First line of every file: when this session started, matching the stamp in the file name.
EnqueueLogMessage(FString::Printf(TEXT("=== Mobius session started %s (pid %u) ==="),
*SessionStart.ToString(TEXT("%Y-%m-%d %H:%M:%S")), FPlatformProcess::GetCurrentProcessId()));
EnqueueLogMessage(FString::Printf(TEXT("Custom logger initialised. Writing to %s"), *LogFilePath));
if (PrunedSessionLogs > 0)
{
EnqueueLogMessage(FString::Printf(TEXT("Removed %d old session log(s); keeping the newest %d"),
PrunedSessionLogs, MaxSessionLogFiles));
}
}

void UMobiusCustomLoggerSubsystem::Deinitialize()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ void UMobiusUserFeedbackSubsystem::Deinitialize()

PendingErrors.Reset();
CachedLogLines.Reset();
CachedLogHead = 0;
RecentPromptTimes.Reset();

Super::Deinitialize();
Expand Down Expand Up @@ -226,7 +227,24 @@ TArray<FMobiusErrorMessage> UMobiusUserFeedbackSubsystem::DrainPendingErrors()

TArray<FString> UMobiusUserFeedbackSubsystem::GetCachedLogLines() const
{
return CachedLogLines;
// Oldest first, which is what the log window's SetLines expects. The head only moves once the ring
// is full, so a zero head means the buffer is still in plain append order.
if (CachedLogHead == 0)
{
return CachedLogLines;
}

TArray<FString> Ordered;
Ordered.Reserve(CachedLogLines.Num());
for (int32 Index = CachedLogHead; Index < CachedLogLines.Num(); ++Index)
{
Ordered.Add(CachedLogLines[Index]);
}
for (int32 Index = 0; Index < CachedLogHead; ++Index)
{
Ordered.Add(CachedLogLines[Index]);
}
return Ordered;
}

FOnMobiusErrorReported& UMobiusUserFeedbackSubsystem::OnErrorReported()
Expand Down Expand Up @@ -310,11 +328,18 @@ void UMobiusUserFeedbackSubsystem::ReportErrorInternal(const FMobiusErrorMessage

void UMobiusUserFeedbackSubsystem::HandleLogLine(const FString& Line)
{
CachedLogLines.Add(Line);
if (CachedLogLines.Num() > MaxCachedLogLines)
if (MaxCachedLogLines > 0)
{
const int32 Overflow = CachedLogLines.Num() - MaxCachedLogLines;
CachedLogLines.RemoveAt(0, Overflow, EAllowShrinking::No);
if (CachedLogLines.Num() < MaxCachedLogLines)
{
CachedLogLines.Add(Line);
}
else
{
// Full: overwrite the oldest slot in place (reuses its allocation) and advance the head.
CachedLogLines[CachedLogHead] = Line;
CachedLogHead = (CachedLogHead + 1) % MaxCachedLogLines;
}
}

LogLineDelegate.Broadcast(Line);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -172,42 +172,45 @@ bool UFrameGrabberHelper::TryInitialize()

if (!GEngine)
{
MobiusLog(TEXT("[FrameGrabber] TryInitialize failed: GEngine is null"));
NoteInitFailure(TEXT("GEngine is null"));
return false;
}

if (!GEngine->GameViewport)
{
MobiusLog(TEXT("[FrameGrabber] TryInitialize failed: GameViewport is null"));
NoteInitFailure(TEXT("GameViewport is null"));
return false;
}

TSharedPtr<SViewport> GameViewportWidget = GEngine->GameViewport->GetGameViewportWidget();
if (!GameViewportWidget.IsValid())
{
MobiusLog(TEXT("[FrameGrabber] TryInitialize failed: GameViewportWidget not constructed yet"));
NoteInitFailure(TEXT("GameViewportWidget not constructed yet"));
return false;
}

// The real viewport interface backing the SViewport
TSharedPtr<ISlateViewport> SlateViewportInterface = GameViewportWidget->GetViewportInterface().Pin();
if (!SlateViewportInterface.IsValid())
{
MobiusLog(TEXT("[FrameGrabber] TryInitialize failed: SlateViewportInterface not wired up yet"));
NoteInitFailure(TEXT("SlateViewportInterface not wired up yet"));
return false;
}

// This is typically an FSceneViewport for the game viewport
TSharedPtr<FSceneViewport> SceneViewport = StaticCastSharedPtr<FSceneViewport>(SlateViewportInterface);
if (!SceneViewport.IsValid())
{
MobiusLog(TEXT("[FrameGrabber] TryInitialize failed: Unexpected viewport type (not FSceneViewport)"));
MobiusReportError(
FText::FromString(TEXT("Viewport Error")),
FText::FromString(TEXT("Could not access scene viewport for screenshot capture.")),
FText::FromString(TEXT("FrameGrabberHelper::TryInitialize")),
EMobiusErrorSeverity::Warning,
false); // Don't show popup for this, it may resolve on retry
// Escalate once per reason, not per retry: ReportError UE_LOGs and re-enqueues into the custom log too.
if (NoteInitFailure(TEXT("Unexpected viewport type (not FSceneViewport)")))
{
MobiusReportError(
FText::FromString(TEXT("Viewport Error")),
FText::FromString(TEXT("Could not access scene viewport for screenshot capture.")),
FText::FromString(TEXT("FrameGrabberHelper::TryInitialize")),
EMobiusErrorSeverity::Warning,
false); // Don't show popup for this, it may resolve on retry
}
return false;
}

Expand All @@ -217,8 +220,7 @@ bool UFrameGrabberHelper::TryInitialize()
// If either dimension is 0, we can't capture yet
if (ViewportSize.X <= 0 || ViewportSize.Y <= 0)
{
MobiusLog(FString::Printf(TEXT("[FrameGrabber] TryInitialize failed: Invalid viewport size %dx%d"),
ViewportSize.X, ViewportSize.Y));
NoteInitFailure(FString::Printf(TEXT("Invalid viewport size %dx%d"), ViewportSize.X, ViewportSize.Y));
return false;
}

Expand All @@ -227,23 +229,51 @@ bool UFrameGrabberHelper::TryInitialize()
// Clamp/validate target too (avoid passing nonsense to frame grabber)
if (TargetSize.X <= 0 || TargetSize.Y <= 0)
{
MobiusLog(FString::Printf(TEXT("[FrameGrabber] TryInitialize failed: Invalid target size %dx%d"),
TargetSize.X, TargetSize.Y));
MobiusReportError(
FText::FromString(TEXT("Configuration Error")),
FText::FromString(FString::Printf(TEXT("Invalid screenshot target size: %dx%d"), TargetSize.X, TargetSize.Y)),
FText::FromString(TEXT("FrameGrabberHelper::TryInitialize")),
EMobiusErrorSeverity::Error,
true);
if (NoteInitFailure(FString::Printf(TEXT("Invalid target size %dx%d"), TargetSize.X, TargetSize.Y)))
{
MobiusReportError(
FText::FromString(TEXT("Configuration Error")),
FText::FromString(FString::Printf(TEXT("Invalid screenshot target size: %dx%d"), TargetSize.X, TargetSize.Y)),
FText::FromString(TEXT("FrameGrabberHelper::TryInitialize")),
EMobiusErrorSeverity::Error,
true);
}
return false;
}

FrameGrabber = MakeUnique<FFrameGrabber>(SceneViewport.ToSharedRef(), ViewportSize);
FrameGrabber->StartCapturingFrames();

MobiusLog(FString::Printf(TEXT("[FrameGrabber] Initialized successfully. ViewportSize: %dx%d, TargetSize: %dx%d"),
ViewportSize.X, ViewportSize.Y, TargetSize.X, TargetSize.Y));
const FString QuietRetries = SuppressedInitFailures > 0
? FString::Printf(TEXT(" (after %d unlogged repeats of: %s)"), SuppressedInitFailures, *LastInitFailureReason)
: FString();
MobiusLog(FString::Printf(TEXT("[FrameGrabber] Initialized successfully. ViewportSize: %dx%d, TargetSize: %dx%d%s"),
ViewportSize.X, ViewportSize.Y, TargetSize.X, TargetSize.Y, *QuietRetries));
LastInitFailureReason.Reset();
SuppressedInitFailures = 0;

return true;
}

bool UFrameGrabberHelper::NoteInitFailure(const FString& Reason)
{
// Tick retries until the viewport is usable, and in editor / -game -nullrhi / headless automation runs
// it stays 0x0 for the whole session. Logging every attempt put one identical line per frame into
// MobiusCustomLog.txt - up to ~3,000 a second, 1.07 M of its 1.26 M lines - and each one went through
// the log window too. A reason is logged once, when it changes; repeats are only counted.
if (Reason == LastInitFailureReason)
{
++SuppressedInitFailures;
return false;
}

const FString PreviousRepeats = SuppressedInitFailures > 0
? FString::Printf(TEXT(" - previous reason '%s' repeated %d more times"), *LastInitFailureReason, SuppressedInitFailures)
: FString();
MobiusLog(FString::Printf(TEXT("[FrameGrabber] TryInitialize failed: %s (retrying quietly; logged again only if the reason changes)%s"),
*Reason, *PreviousRepeats));
LastInitFailureReason = Reason;
SuppressedInitFailures = 0;
return true;
}
#endif // !PLATFORM_MAC
Expand Down Expand Up @@ -384,7 +414,9 @@ void UFrameGrabberHelper::TriggerCapture(const FString& InFileName)
MobiusLog(TEXT("[FrameGrabber] FrameGrabber not initialized, attempting initialization..."));
if (!TryInitialize())
{
MobiusLog(TEXT("[FrameGrabber] Error: Could not initialize FrameGrabber - viewport not ready."));
// TryInitialize may have stayed quiet (the same reason Tick already logged), so name it here.
MobiusLog(FString::Printf(TEXT("[FrameGrabber] Error: Could not initialize FrameGrabber - viewport not ready (%s)."),
*LastInitFailureReason));
MobiusReportError(
FText::FromString(TEXT("Initialization Failed")),
FText::FromString(TEXT("Could not initialize screenshot capture. The viewport may not be ready yet.")),
Expand Down Expand Up @@ -548,10 +580,16 @@ void UFrameGrabberHelper::Tick(float DeltaTime)
// The OnScreenshotCaptured callback handles everything
#else
// Windows/other: Poll FFrameGrabber
// Try lazy initialization if not yet initialized
// Try lazy initialization if not yet initialized. Backed off while it keeps failing: under -nullrhi /
// headless the viewport is 0x0 for the whole session, and a retry every frame bought nothing.
// TriggerCapture still initializes on demand without waiting for this.
if (!FrameGrabber.IsValid())
{
TryInitialize();
InitRetryCooldown -= DeltaTime;
if (InitRetryCooldown <= 0.0f && !TryInitialize())
{
InitRetryCooldown = InitRetryIntervalSeconds;
}
}

if (!bIsCapturing || !FrameGrabber.IsValid())
Expand Down
Loading