Graduate Client Project · Army Research Laboratory · Multimodal Situation Awareness
MUMOSA Situation Awareness Dashboard
A graduate client project for Army Research Laboratory that helps analysts compare reports, images, timelines, and reconstructed scenes after a crisis.
At a glance
Role
Led research and spatial prototyping; contributed to the dashboard information architecture
Team
Three-person graduate team: Georgi Tsvetanski, Kelly Ehrlich, and Kamilah S.
Client Need
Help analysts reconstruct incidents, compare evidence, and train responders
Artifacts
Client report, paper prototype, Axure dashboard package, and Unreal VR proof of concept.
From scattered evidence to a traceable incident timeline
Analysts ask a question, inspect the supporting evidence, place events on a timeline, and open a reconstructed scene when location and physical context are important. The same flow can later accept new evidence as an incident develops.
- - Short term: post-crisis investigation, lessons-learned review, and responder training.
- - Long term: real-time situational awareness with dynamic timelines, hazard cues, and guided next actions.
- - My lane: research, spatial interaction design, and the Unreal scene-review prototype.
Featured implementation
Unreal spatial evidence prototype
I turned the paper interaction model into a PC-first Unreal proof of concept where analysts can select evidence in the scene, inspect its confidence and timeline context, and follow it back to source material.
PC-First, VR-Ready
Keyboard-and-mouse walkthrough first; OpenXR follows after the core interaction is proven.
Evidence Marker System
Reusable C++ markers store the claim, confidence, timeline state, discrepancies, and linked sources.
Source-Grounded Panel
Selecting a marker opens its interpretation, confidence, timeline context, and direct source links.
Project In Short
The team designed one evidence workflow spanning questions, sources, timelines, and reconstructed scenes. I led the research and spatial prototype, contributed to the dashboard structure, and built the Unreal proof of concept.
Short-Term Use
Help investigators and instructors compare reports, images, events, and timelines after a crisis.
Long-Term Direction
Extend the same structure to incoming evidence, changing timelines, hazard alerts, and role-specific views.
Design Response
Organize questions, sources, conflicts, timelines, and spatial context in one evidence workspace.
Closer look
Research, paper prototypes, dashboard work, and deliverables
Open the full process only if you want the detailed rationale, supporting artifacts, testing plan, and authored research.
Open +Close −
Closer look
Research, paper prototypes, dashboard work, and deliverables
Open the full process only if you want the detailed rationale, supporting artifacts, testing plan, and authored research.
Research Findings That Shaped The Direction
Cognitive Load Comes First
Responders and investigators already manage too many information streams. The interface should reduce fragmentation and visual noise.
AI Claims Need Sources
Every AI summary should link directly to the report, image, event, or scene evidence that supports it.
Post-Crisis Review Comes First
Post-crisis reconstruction and training provide the clearest near-term use for documents, event relationships, and 3D scene review.
Finished Low-Fidelity VR Paper Prototype
The finished low-fidelity prototype translates the research into something concrete: a paper headset window, controller annotations, a sketched reconstruction of the crisis site, and evidence notes that appear in-scene when the investigator asks to verify a claim.
Instead of claiming a full VR build, I focused on the interaction questions that actually matter first. Can users orient themselves in the scene? Can AI summaries be checked against evidence? Does the interface support a resolve-phase workflow without burying the person in more complexity?
Scene-First Orientation
I started with a panoramic sketch of the site so investigators can understand place and hazard layout before chasing UI chrome.
Evidence In Context
The sticky-note overlays simulate AI summaries, timestamps, and next actions anchored directly to the place being inspected.
Low-Tech, Testable Controls
Annotated paper controllers let us test teleportation, source reveal, LiDAR measurement, and zoom/select behavior before building software.
Interaction Model
The controls are intentionally modest. I wanted the paper prototype to prove the interaction logic before promising any technical implementation. The model stays focused on navigation, evidence verification, and selective deep inspection.
- - Teleport between scene zones instead of forcing the user through menu-heavy navigation.
- - Use a visible "show source" action so AI summaries can always lead back to evidence.
- - Reserve LiDAR measurement, zoom, and alternate view controls for deeper inspection moments.
- - Keep VR as a review surface paired with the web dashboard, not a replacement for the broader system.
Evidence Grounding In The Scene
These notes are the core of the concept. They show how the interface could present AI-generated findings without asking users to trust floating summaries blindly. Each card has a timestamped claim, a quick interpretation, and an action prompt that leads back to source evidence.
Revised Dashboard: From Role Picker To Q/A Workspace
The original dashboard prototype started with a role-picker and presented a generic event overview. After aligning more tightly with the MUMOSA source paper, the team redesigned the flow around a structured investigative workflow. The result is a 5-screen Q/A workspace where every interaction has a clear purpose.
Incident + Question Entry
Replaced the old role-picker-first flow with a natural-language Q/A entry. Users select an incident and timeframe, then ask a grounded question from suggested prompts.
Q/A Results Workspace
The central screen shows the AI answer with ranked confidence, textual evidence panel, visual evidence panel, and source metadata — all visible without clicking away.
Evidence Comparison
Side-by-side textual and visual source review with discrepancy callouts, source chains, and investigator notes. Every AI claim is traceable back to source material.
Timeline + Event Map
The most heavily revised screen. A horizontal timeline with event nodes, a schema-backed event relationship map with legend, and a selected-node detail panel showing participants, sources, and conflicts.
Simulation Evidence
The spatial review page, relabeled from "VR Scene Review" to match the MUMOSA paper. Shows 3D reconstruction with question-driven annotation overlays and linked source panels.
Team Dashboard Direction
Even though my emphasis is the VR and spatial simulation lane, the full project was broader than that. The shared planning board tracked flatscreen improvements alongside my lane, and those dashboard ideas matter because spatial review only makes sense as one mode in a larger investigative workflow.
Information Architecture Reset
The flatscreen direction moved from a generic event overview dashboard to a structured Q/A workspace: Ask → Q/A Results → Compare Evidence → Timeline + Event Map → Simulation Evidence. Each screen has a clear investigative purpose.
Source-Grounded AI
The revised dashboard never shows an AI answer without attached source cues. Confidence badges, ranked evidence scores, and discrepancy warnings are built into every answer card.
Timeline/Schema Critique Response
The original timeline view had no legend, unclear node meaning, and hard-to-follow side panels. The revision adds a full legend, source-linked event nodes, participant roles (agent/causer, affected entity, location), and a selected-node detail panel.
Investigator Notes
My teammates also carried forward the note-taking and saved annotations concept so investigators can preserve findings during deeper review.
Design Brief Translation
Reviewing the design brief helped tighten the page narrative. The board makes it clear that the project is not only about adding VR, but about restructuring the whole experience around cognition, role, and investigation flow.
Role-Adaptive Information
The design brief reframed the dashboard around dynamic filtering and role-based views so investigators, responders, and coordinators can enter the same incident from different cognitive starting points.
Overview To Detail To Overview
One of the clearest patterns in the brief is hierarchical exploration: start broad, drill into specific evidence, then move back out to re-establish context.
Training Is Not Secondary
The board treats simulation-based learning, pattern recognition, and decision rehearsal as core outcomes, not side benefits layered on after the fact.
Evidence Capture Pipeline
The spatial review prototype is backed by a realistic data pipeline. Drones, robots, and body cameras capture the scene; AI processes it into grounded evidence; and Nanite renders raw photogrammetry without costly retopology. The result is an end-to-end workflow from crisis site to clickable spatial evidence.
Fidelity Tiers
Active response needs answers in minutes — low-fidelity Gaussian splatting (~5-15 min) shows danger zones and blocked routes immediately. Post-crisis investigation uses full photogrammetry (~30 min - 2+ hrs) for forensic-grade detail. The prototype proves the review layer; the processing speed is an engineering curve, not a research question.
Nanite & Raw Scans
Photogrammetry produces messy scans with holes and artifacts. Nanite renders the raw mesh at full detail — no retopology needed for static evidence review. Key objects (railcar, ignition zone) can get AI-assisted cleanup; everything else renders directly. The prototype uses photoscanned Megascans debris to demonstrate the visual quality the pipeline would produce.
Query & AI Integration
The user asks a natural-language question through the dashboard. The AI (local or API) queries the evidence store, determines relevant markers, highlights them in the Unreal scene via the MCP bridge, and populates the source panel. The prototype mocks this with structured JSON data — the same shape a real AI query would return — so the interaction model is proven regardless of backend.
Planned Usability Test
The paper-prototype package included a usability script with a concrete evaluation plan for orientation, timeline understanding, and deeper evidence review.
Task 1: Initial Orientation
Participants first explore the dashboard freely, then explain what they would do first to understand the event. This tests whether overview information and entry points are discoverable.
Task 2: Timeline Reconstruction
The script asks users to find when the event happened and reconstruct the sequence leading up to it, focusing on timeline discoverability and information hierarchy.
Task 3: Evidence And Deeper Investigation
Participants are asked to locate supporting evidence and describe how they would inspect the scene more closely, which directly probes whether VR mode and deeper analysis tools feel legible.
Course Deliverables And Project Status
Completed
Literature Review
Authored research document grounding the redesign in situational awareness, cognitive load, and multimodal crisis-response heuristics.
Completed
Low-Fidelity Team Prototype
The low-fi package included dashboard wireframes, the VR paper prototype, and a usability-testing script for an incident-analysis scenario.
Completed
Revised Dashboard Spec + Axure Prototype
The team finalized the revised 5-screen Q/A workspace flow. I contributed the Axure build spec, design system, and evidence-review interaction model for the digital prototype.
Completed Proof Of Concept
Unreal Spatial Simulation (My Lane)
Built an early Unreal spatial review proof of concept with evidence markers, source-grounded UI behavior, hazard-oriented scene context, and a mock dashboard handoff model.
Completed
Client Report + Final Presentation
Documented the research, paper prototype, electronic prototype, testing approach, lessons learned, and recommendations in a client-facing final report.
Research Grounding
This is the authored research document behind my part of the project. It covers user groups, heuristics, multimodal crisis-response design, and the reasoning that eventually shaped the resolve-phase and VR framing.
The client paper still matters as context, but it is not my portfolio artifact. What belongs in this case study is the bridge from research into the finished low-fidelity prototype and the digital implementation that followed.
Earlier VR Direction Note
Reference Documents
These links are here for context and coursework documentation. The client paper is supporting reference, not presented as my authored portfolio work. The GitHub repository contains the full project working materials.
Outcome So Far
What started as a literature review and a paper VR sketch became a complete client-facing project package: a revised Q/A dashboard direction, an Axure electronic prototype, a final report, and an Unreal proof of concept for spatial evidence review. The strongest through-line is source grounding. Whether the user is reading an AI answer, comparing visual evidence, reconstructing a timeline, or stepping into a 3D scene, the system should make it clear what evidence supports the claim and where uncertainty still exists.