Sehyeon Park
프로젝트 목록
2025년 5월 27일
2 min read

StreamSplat

웹 3DGS 장면의 color/depth 스트리밍과 depth-aware Gaussian/mesh hybrid rendering.
KIST
기간
2025.03–2026.04
내 역할
제1저자 · 시스템 설계/구현
담당 작업

서버 3DGS/4DGS 렌더러와 브라우저 WebGL 렌더러 분리

RGB-D/camera interface, WebCodecs decode, WebGL depth fusion 구현

client-only 대비 hybrid streaming 성능 실험 및 논문 작성

성과

Web3D 2025 Best Paper

English page 데모 보기

개요

웹에서 재구성 3DGS 장면을 스트리밍하고, 브라우저 안에서 로컬 mesh와 합성·상호작용하는 하이브리드 뉴럴 렌더링 시스템.

서버의 무거운 Gaussian 렌더링과 클라이언트의 WebGL/WebCodecs 기반 합성을 분리해 웹 접근성, 성능, 상호작용을 함께 다룬 연구.

문제 정의

  • Client-only 3DGS 렌더링의 모바일 메모리·GPU 제약
  • 서버 렌더 RGB-D와 브라우저 로컬 mesh 사이의 depth/좌표계 정렬 문제
  • 뉴럴 장면과 브라우저 객체가 같은 화면에서 물리적으로 상호작용해야 하는 요구

접근

  • 서버에서 RGB와 log-depth를 함께 렌더링하고 H.264/WebSocket 경로로 전송
  • WebCodecs decode, WebGL mesh rendering, screen-space depth fusion을 브라우저에서 수행
  • Coordinate-space alignment와 depth normalization으로 Gaussian scene과 local mesh를 같은 view에서 합성
  • Depth-based collision으로 browser-side object interaction 검증

담당 작업

  • Web neural graphics용 server-client rendering/transport architecture 설계
  • Color/depth streaming interface와 screen-space fusion pipeline 구현
  • Browser-side local mesh interaction, collision, constrained-client evaluation 수행

결과

  • Web3D 2025 Best Paper Award
  • JPEG+RAW 4197 KB/frame → JPEG-stack 219 KB/frame
  • 8.0 FPS baseline → 35.2 FPS compressed-depth path
  • Client-only OOM scene을 iPhone hybrid streaming에서 17–21 FPS로 실행

기술 키워드

3D Gaussian Splatting Neural Rendering Web3D Depth Fusion WebXR

검증 지표

Payload
4197 KB/fr 219 KB/fr
19.2× smaller

JPEG+RAW depth transfer → JPEG-stack compressed depth

Throughput
8.0 FPS 35.2 FPS
4.4× faster

1080p LivingLab, Mac Studio Chrome

iPhone scenes
Client-only OOM 17–21 FPS hybrid
Runnable on constrained clients

Garden/Bicycle failure cases recover under hybrid streaming

자료

Graphical abstract showing cross-platform web access, hybrid neural rendering, coordinate-space alignment, depth-based fusion, and depth-based collision
Graphical abstract showing cross-platform web access, hybrid neural rendering, coordinate-space alignment, depth-based fusion, and depth-based collision
Final main figure showing server-side neural renderer, color/depth H.264 streaming, WebCodecs/WebGL client fusion, and WebGL mesh interaction
Final main figure showing server-side neural renderer, color/depth H.264 streaming, WebCodecs/WebGL client fusion, and WebGL mesh interaction
Animated RGB-D fusion preview with local RGB, server RGB, fusion RGB, and fusion depth views
Animated RGB-D fusion preview with local RGB, server RGB, fusion RGB, and fusion depth views
Animated credit-card collision test with neural stream, local Open3D mesh, and fused view
Animated credit-card collision test with neural stream, local Open3D mesh, and fused view
Animated credit-card depth-fusion test showing local object movement and fused neural view
Animated credit-card depth-fusion test showing local object movement and fused neural view
Animated close-range credit-card collision test showing depth-aware local mesh fusion
Animated close-range credit-card collision test showing depth-aware local mesh fusion
Animated ruler interaction test showing browser-side mesh movement and fused depth response
Animated ruler interaction test showing browser-side mesh movement and fused depth response
Depth normalization failure case showing incorrect server/client depth ordering before fusion correction
Depth normalization failure case showing incorrect server/client depth ordering before fusion correction
Depth-based collision demonstration and viewpoint failure heatmap from follow-up depth-fusion experiments
Depth-based collision demonstration and viewpoint failure heatmap from follow-up depth-fusion experiments
이전
속도계 인터페이스 개발
다음
Guacamole — LLM 기반 영상 침해 탐지