Magnetic Tracking System for Dexterous 3D Interaction and Motion

We are developing a novel approach to reconstructing 3D motion data from a magnetic flux sensor array using deep learning. Computing the 3D configuration of markers (inductor-capacitor (LC) coils) from flux sensor data is achieved by using deep neural networks to learn the regression from the simulation flux values to the LC coils’ 3D configuration, which can be applied to the actual LC coils at any location and orientation within the capture volume.  Our method can track various movements, including fingers that manipulate objects, beetles that move inside a vivarium with leaves and soil, and the flow of opaque fluid. Since no power supply is needed for the lightweight wireless markers, our method can robustly track movements for a very long time, making it suitable for various types of observations whose tracking is difficult with existing motion-tracking systems.

Publications

International

  • Jiawei Huang, Mori Tsuyoshi, Kazuki Takashima, Shuichiro Hashi, Yoshifumi Kitamura: 
    IM6D: Magnetic Tracking System with 6-DOF Passive Markers for Dexterous 3D Interaction and Motion
    ACM Transactions on Graphics (TOG), Volume 34, Issue 6 (Proceedings of ACM SIGGRAPH Asia), pp. 217:1-217:10, November 2015. [PDF]
    (This paper has been selected and listed on the top page of the ACM SIGGRAPH website as one of the three papers that they feel might be of particular interest to SIGGRAPH members (April 1st – July 25th, 2016).)
  • Jiawei Huang, Mori Tsuyoshi, Kazuki Takashima, Shuichiro Hashi, Yoshifumi Kitamura: 
    6-DOF Computation and Marker Design for Magnetic 3D Dexterous Motion-Tracking System
    ACM Symposium on Virtual Reality Software and Technology (VRST), pp. 211-217, November 2016.
  • Jiawei Huang, Ryo Sugawara, Kinfung Chu, Taku Komura, and Yoshifumi Kitamura,
    “Reconstruction of Dexterous 3D Motion Data from a Flexible Magnetic Sensor with Deep Learning and Structure-Aware Filtering,”
    IEEE Transactions on Visualization and Computer Graphics (2020)
    [PDF] [Press Release]

Invited Talk

  • 黄 佳維, 森 健, 高嶋 和毅, 枦 修一郎, 北村 喜文:
    複雑な3Dインタラクションとモーションのための6自由度パッシブマーカによる磁気式トラッキングシステム
    情報処理学会研究報告 グラフィクスとCAD (CG), 2015-CG-161, 8. SIGGRAPH ASIA招待講演セッション，2015年11月6日，神戸 [PDF]

Demonstrations

  • Jiawei Huang, Kazuki Takashima, Shuichiro Hashi, Yoshifumi Kitamura:
    IM3D: Magnetic Motion Tracking System for Dexterous 3D Interactions
    SIGGRAPH 2014 Emerging Technologies, Vancouver, Canada. [INFO]