Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion assets/css/main.css
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
--accent: #86a7d8;
--copper: #9e6339;
--sage: #54724b;
--teal: #23746e;
--shadow: 0 18px 42px rgba(13, 31, 60, 0.08);
--shadow-soft: 0 10px 24px rgba(13, 31, 60, 0.06);
--radius: 1.35rem;
Expand Down Expand Up @@ -1031,7 +1032,7 @@ a.chip {
}

.card-grid-wide {
grid-template-columns: repeat(3, minmax(0, 1fr));
grid-template-columns: repeat(2, minmax(0, 1fr));
}

.accent-copper {
Expand All @@ -1046,6 +1047,10 @@ a.chip {
border-top: 4px solid var(--primary);
}

.accent-teal {
border-top: 4px solid var(--teal);
}

.person-card {
display: flex;
flex-direction: column;
Expand Down Expand Up @@ -1217,6 +1222,13 @@ button:disabled { cursor: default; opacity: 0.6; }
font-size: 1.9rem;
}

.research-support-logos { display: flex; flex-wrap: wrap; align-items: center; gap: 2rem; margin-block: 1rem; }
.research-support-logos img { display: block; object-fit: contain; }
.research-diagrams { display: grid; grid-template-columns: repeat(auto-fit, minmax(min(100%, 20rem), 1fr)); gap: 2rem; margin-block: 2rem; }
.research-diagrams figure { margin: 0 auto; max-width: 25rem; }
.research-diagrams img { display: block; width: 100%; height: auto; }
.research-diagrams figcaption { color: var(--muted); font-size: 0.9rem; margin-top: 0.75rem; }

/* Long display equations scroll within their panel, including on phones. */
.math-block {
max-width: 100%;
Expand Down
2 changes: 1 addition & 1 deletion content/_index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
+++
title = "Home"
hero_subtitle = "The Middleware Systems Research Group at the University of Toronto"
hero_tagline = "Systems research that spans data, learning, and quantum computing"
hero_tagline = "Systems research that spans data, learning, quantum computing, and reliability"
+++

Traditionally, the “M” in MSRG stood for “middleware”—the Middleware Systems Research Group at the University of Toronto. But don’t worry, “middleware is literally everywhere.” Over the years, that “M” has evolved to mean “mobile,” “microservices,” “machine learning,” and, last but not least, “matter,” as in entangled particles in space-time. What *more* could you ask for?
2 changes: 1 addition & 1 deletion content/people/chenkai-zhang.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,6 @@ email = "zhangchenkai.zhang@mail.utoronto.ca"
interests = ["AI Agent Reliability", "Fault Diagnosis"]
author_names = ["Chenkai Zhang"]
portrait = ""
research = ["distributed-machine-learning"]
research = ["distributed-machine-learning", "rca-aiops"]
linkedin = "https://www.linkedin.com/in/chenkaizhang1/"
+++
2 changes: 1 addition & 1 deletion content/people/hans-arno-jacobsen.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ role = "Professor"
status = "current"
start_date = "2000-01"
location = "Toronto, Canada"
research = ["quantum-computing-systems", "distributed-machine-learning", "data-management"]
research = ["quantum-computing-systems", "distributed-machine-learning", "data-management", "rca-aiops"]
email = "jacobsen@eecg.utoronto.ca"
homepage = "https://www.eecg.toronto.edu/~jacobsen"
orcid = "https://orcid.org/0000-0003-0813-0101"
Expand Down
2 changes: 1 addition & 1 deletion content/people/mengli-duan.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ role = "PhD Student"
status = "current"
start_date = "2026-09"
location = "Toronto, Canada"
research = ["distributed-machine-learning"]
research = ["distributed-machine-learning", "rca-aiops"]
email = "mengli.duan@mail.utoronto.ca"
interests = ["Machine Learning Systems"]
author_names = ["Mengli Duan", "Dawn (Mengli) Duan", "Dawn Duan"]
Expand Down
2 changes: 1 addition & 1 deletion content/people/michalis-bachras.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ role = "PhD Student"
status = "current"
start_date = "2023-09"
location = "Toronto, Canada"
research = ["data-management"]
research = ["data-management", "rca-aiops"]
interests = ["Data Management", "Distributed Systems"]
portrait = "/images/people/michalis-bachras.jpg"
linkedin = "https://www.linkedin.com/in/michalis-bachras/"
Expand Down
12 changes: 7 additions & 5 deletions content/people/rev-liu.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
+++
name = "Rev Liu"
name = "Ruiwu Liu"
slug = "rev-liu"
role = "MSc Student"
role = "PhD Student"
status = "current"
start_date = "2025-09"
start_date = "2026-09"
location = "Toronto, Canada"
research = ["distributed-machine-learning", "rca-aiops"]
email = "rev.liu@mail.utoronto.ca"
interests = []
interests = ["VLM", "RL", "AIOps"]
author_names = []
portrait = ""
portrait = "/images/people/ruiwu-liu.jpg"
linkedin = "https://www.linkedin.com/in/ruiwu-liu-b1534827b/"
+++
2 changes: 1 addition & 1 deletion content/people/yifang-tian.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ role = "PhD Student"
status = "current"
start_date = "2022-01"
location = "Toronto, Canada"
research = ["distributed-machine-learning"]
research = ["distributed-machine-learning", "rca-aiops"]
interests = ["Machine Learning Systems", "Distributed Systems"]
portrait = "/images/people/yifang-tian.jpg"
email = "yifang.tian@mail.utoronto.ca"
Expand Down
1 change: 1 addition & 0 deletions content/people/yifeng-he.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ role = "MSc Student"
status = "current"
start_date = "2025-09"
location = "Toronto, Canada"
research = ["rca-aiops"]
email = "yifeng.he@mail.utoronto.ca"
interests = []
author_names = ["Yifeng He", "Yifeng (Jason) He"]
Expand Down
2 changes: 1 addition & 1 deletion content/people/yiran-li.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ role = "Post-Doc"
status = "current"
start_date = "2025-11"
location = "Toronto, Canada"
research = ["quantum-computing-systems", "distributed-machine-learning", "data-management"]
research = ["quantum-computing-systems", "distributed-machine-learning", "data-management", "rca-aiops"]
interests = ["Graph Analytics", "Graph Learning", "Quantum Data Management", "AI Agent Reliability"]
email = ""
homepage = "https://cyanidecentral.github.io/"
Expand Down
15 changes: 15 additions & 0 deletions content/people/yuanchen-gao.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
+++
name = "Yuanchen Gao"
slug = "yuanchen-gao"
role = "PhD Student"
status = "current"
location = "HKUST, Hong Kong"
research = ["rca-aiops"]
interests = ["Root Cause Analysis", "Automated Program Repair", "AIOps"]
author_names = ["Yuanchen Gao"]
email = "ygaocv@connect.ust.hk"
# PhD role and current RCA participation confirmed by the maintainer, 2026-11-01.
# HKUST group: https://cse.hkust.edu.hk/~charlesz/
# Affiliation and public email: https://arxiv.org/html/2608.17018v1
# MSRG visit dates, portrait, and personal academic accounts are unverified.
+++
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,12 @@
title = "FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection"
year = 2026
authors = ["Huanchi Wang", "Zihang Huang", "Yifang Tian", "Kristina Dzeparoska", "Hans-Arno Jacobsen", "Alberto Leon-Garcia"]
venue = "arXiv"
publication_type = "ArXiv Preprint"
research = ["distributed-machine-learning"]
venue = "37th IEEE International Symposium on Software Reliability Engineering (ISSRE)"
publication_type = "Conference Paper"
research = ["distributed-machine-learning", "rca-aiops"]
external_url = "https://arxiv.org/abs/2605.22779"
abstract = "Production systems generate millions of log lines daily, yet most anomaly detectors operate at the session or window-level, flagging groups of lines rather than identifying the specific message responsible. This coarse granularity forces operators to inspect many routine lines per alert. Message-level detection offers finer granularity, but remains challenging. A single event template may correspond to both normal and anomalous messages, failures arise from heterogeneous subsystems, and line-level labeling at scale is impractical. Although large language models (LLMs) can reason over log semantics, applying them to every line is too costly for continuous monitoring. We present FAME (Failure-Aware Mixture-of-Experts), a label-efficient message-level mixture-of-experts framework that uses an LLM only once offline. We annotate at most K labeled lines per template to derive binary normal/anomaly indicators and representative examples. The LLM proposes a partition of templates into failure domains, and a certification step validates the proposal before training. FAME trains a lightweight router and domain experts that run on-premise and output anomaly predictions and failure-domain labels. On BGL, FAME achieves F1 = 98.16 at K = 100 reducing annotation effort by 76x and detects 97.7% of anomalies from unseen EventIDs. On Thunderbird, FAME reaches F1 = 99.95 with perfect recall."
+++

Presented at [ISSRE 2026](https://cyprusconferences.org/issre2026/program/).
The full-text link points to the arXiv version.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,10 +1,18 @@
+++
title = "GALA: Graph-Augmented LLM Agents for Root Cause Analysis and Incident Response in Microservices"
aliases = ["/publications/gala-can-graph-augmented-large-language-model-agentic-workflows-elevate-root-cause-analysis/"]
year = 2026
authors = ["Yifang Tian", "Yaming Liu", "Zichun Chong", "Zihang Huang", "Yiran Li", "Hans-Arno Jacobsen"]
venue = "arXiv"
publication_type = "ArXiv Preprint"
research = ["distributed-machine-learning", "data-management"]
venue = "41st IEEE/ACM International Conference on Automated Software Engineering (ASE)"
publication_type = "Conference Paper"
research = ["distributed-machine-learning", "data-management", "rca-aiops"]
external_url = "https://arxiv.org/abs/2608.08968"
abstract = "Microservice root cause analysis (RCA) requires correlating failures across heterogeneous telemetry within complex service dependency graphs. Existing methods often rely on a single telemetry modality; recent LLM-based approaches can suffer from unconstrained exploration and hallucination; and most systems stop at fault ranking without producing actionable incident response. We present GALA+, a graph-augmented LLM agentic framework centered on graph-guided investigation, which uses service dependencies to bound exploration and refine diagnosis through localized multi-modal evidence. For initial hypothesis generation, GALA+ combines complementary telemetry signals with STRIX, a novel trace- and graph-structure-aware scoring module. GALA+ then produces ranked diagnoses, incident summaries, and stratified action recommendations. We further introduce SURE-Score, a human-guided evaluation framework co-developed with industry SRE experts for assessing RCA-specific output quality beyond conventional text similarity metrics. On two microservice benchmarks, GALA+ consistently achieves the strongest overall results, surpassing the best LLM-based baseline by more than 25 percentage points in AC@1, while also receiving the highest ratings from both SURE-Score and independent human SRE evaluation."
+++

[Published version](https://doi.org/10.1145/3832783.3834335).
The full-text link points to the arXiv version, which describes the GALA+ framework.

This record consolidates the GALA work and its updated GALA+ framework.
The [earlier preprint](https://arxiv.org/abs/2508.12472) circulated under the title
*GALA: Can Graph-Augmented Large Language Model Agentic Workflows Elevate Root Cause Analysis?*.
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ year = 2026
authors = ["Yuanchen Gao", "Yifang Tian", "Yiran Li", "Charles Zhang", "Hans-Arno Jacobsen"]
venue = "arXiv"
publication_type = "ArXiv Preprint"
research = ["data-management"]
research = ["data-management", "rca-aiops"]
external_url = "https://arxiv.org/abs/2608.17018"
abstract = "Microservice failures are often diagnosed from operational telemetry. However, automated program repair systems usually start from issue reports, localized code context, or failing tests. This mismatch leaves a gap between telemetry-based diagnosis and patch generation. We present ORCA, an observability-grounded APR pipeline for microservice incidents. ORCA first distills the differences in paired failure and reference telemetry into a fault signature, then uses the signature to identify candidate code and deployment-configuration locations. Repair graph agents and an Exploration agent generate unified-diff patch candidates from these locations. ORCA evaluates generated patches with a Telemetry-Grounded Patch Verifier that separates patch validity, syntactic and semantic correctness, test-oracle integrity, and telemetry replay. On a 575-case benchmark, ORCA outperforms all evaluated baselines in terms of cost-effectiveness. Results show that operational telemetry can be transformed from diagnostic evidence into actionable repair context: paired telemetry supports repair-oriented localization, while repair graph agents convert localized code and configuration evidence into constrained patch-generation context for the LLM. Telemetry-grounded verification then exposes repair outcomes that issue- or test-only evaluation would miss."
+++
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
+++
title = "SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agents"
year = 2026
authors = ["Yifang Tian", "Yingjian Bai", "Yifeng He", "Zichun Chong", "Yuanchen Gao", "Yiran Li", "Hans-Arno Jacobsen"]
venue = "arXiv"
publication_type = "ArXiv Preprint"
research = ["distributed-machine-learning", "rca-aiops"]
external_url = "https://arxiv.org/abs/2609.33023"
+++

This preprint studies continuous SRE operation with overlapping incidents in a
live Kubernetes environment. Its evaluation follows the progression from
correlation and localization to repair using recorded system evidence.

[Project website and leaderboard](https://bigbigbai.github.io/sre-marathon-leaderboard/).
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,13 @@
title = "SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios"
year = 2026
authors = ["Jackson Clark", "Yiming Su", "Saad Mohammad Rafid Pial", "Yifang Tian", "Lily Gniedziejko", "Hans-Arno Jacobsen", "Yinfang Chen", "Tianyin Xu"]
venue = "arXiv"
publication_type = "ArXiv Preprint"
research = ["distributed-machine-learning"]
venue = "NeurIPS Evaluations and Datasets Track (accepted; forthcoming)"
publication_type = "Conference Paper"
research = ["distributed-machine-learning", "rca-aiops"]
external_url = "https://arxiv.org/abs/2605.07161"
abstract = "AI agents are increasingly used to diagnose and mitigate failures in production systems, known as agentic Site Reliability Engineering (SRE). Current SRE benchmarks are limited to oversimplistic SRE tasks and are unfortunately hard to extend due to bespoke designs. We present SREGym, a high-fidelity benchmark for SRE agents. SREGym exposes a live system environment built atop real-world cloud-native system stacks, where high-fidelity failure scenarios are simulated through fault injectors. SREGym models the complexity of production environments by simulating (1) a wide range of faults at different layers, (2) various ambient noises, and (3) diverse failure modes such as metastable failures and correlated failures. SREGym is architected as a modular, extensible framework that orchestrates fault and noise injectors across stacks. SREGym currently includes 90 realistic, challenging SRE problems. We use SREGym to evaluate frontier agents and show that their capabilities varies significantly in addressing different kinds of failures, with up to 40% differences in end-to-end results. SREGym is actively maintained as an open-source project and has been used by researchers and practitioners."
+++

Accepted to NeurIPS 2026, as listed on the
[co-author's publication page](https://yinfangchen.github.io/publications/).
The full-text link points to the arXiv version; proceedings are forthcoming.
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
+++
title = "When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry"
title = "AgentChaosBench: Benchmarking External Fault Detection and Localization in Agentic Systems"
slug = "when-agentic-executions-fail-detecting-and-localizing-runtime-faults-from-telemetry"
year = 2026
authors = ["Chenkai Zhang", "Yiran Li", "Yifang Tian", "Michalis Bachras", "Hans-Arno Jacobsen"]
venue = "arXiv"
publication_type = "ArXiv Preprint"
research = ["distributed-machine-learning"]
authors = ["Chenkai Zhang", "Yiran Li", "Yifang Tian", "Michail Bachras", "Hans-Arno Jacobsen"]
venue = "International Workshop on Trustworthy and Responsible Autonomous Systems (TRUST)"
publication_type = "Workshop Paper"
research = ["distributed-machine-learning", "rca-aiops"]
external_url = "https://arxiv.org/abs/2608.14680"
abstract = "Reliability in LLM-based agentic systems is a property of the whole execution (its tool calls, model calls, guardrails, and inter-agent messages), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails. We present AGENTCHAOSBENCH, a benchmark for detecting and localizing runtime faults in agentic systems from their execution telemetry. We run five heterogeneous applications that coordinate agents over the Agent-to-Agent protocol and call tools through the Model Context Protocol, and inject ten types of operational fault (unavailable or slow tools, corrupted or oversized responses, and delayed, looped, or misrouted delegations and bypassed guardrails) at their tool, model, guardrail, and inter-agent boundaries, alongside a no-fault control. The resulting dataset contains 275 sanitized traces: 250 faulty executions spanning ten fault types and 25 no-fault controls. Each faulty trace is aligned with the no-fault execution of the same input; fault-type labels and, where applicable, location labels are held out from diagnosis. On structured single-trace inputs, a first set of zero-shot LLM baselines shows the task is far from solved: local detectors up to 14B parameters reach only 13.6-19.2% top-1 fault-type accuracy and the frontier DeepSeek-v4-pro only 24.8%, while jointly identifying the fault type and its location tops out at 22%; reference-dependent faults (above all a bypassed guardrail) stay near-unsolved from a single trace. An aligned reference improves selected relative faults but does not resolve guardrail bypass. The held-out labels and compact prediction format support reproducible comparison of LLM-based and non-LLM diagnosis methods."
+++

[Published version](https://doi.org/10.1145/3843782.3844662).
The linked arXiv version uses the earlier title *When Agentic Executions Fail:
Detecting and Localizing Runtime Faults from Telemetry*.
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
+++
title = "When Log Parsing Removes What Matters: Detecting Failures in Distributed GPU Workloads"
year = 2026
authors = ["Boyuan Cui", "Yuyang Zeng", "Yifang Tian", "Hans-Arno Jacobsen"]
venue = "1st International Workshop on Reliable and trustworthy Automated Software Engineering (RASE), pp. 60–63"
publication_type = "Workshop Paper"
research = ["distributed-machine-learning", "rca-aiops"]
external_url = "https://arxiv.org/abs/2610.14009"
+++

[Published version](https://doi.org/10.1145/3820756.3844919).
The arXiv version was revised after publication and differs from the RASE 2026 paper.

The paper introduces [GF-Bench](https://github.com/cuiboyuan/nccl_log_rca_bench)
for generating controlled faults in distributed GPU workloads and examines how
log parsing affects failure detection.
2 changes: 1 addition & 1 deletion content/research/_index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
+++
title = "Research"
layout = "overview"
summary = "Three active research themes connect MSRG's work in systems, AI infrastructure, and quantum computing."
summary = "Four active research themes connect MSRG's work in data management, distributed machine learning, quantum computing, and reliable systems."
+++
Loading
Loading