Rhizomatiks Research, ELEVENPLAY
evala, Takayuki Fujimoto

border

Work Details

「border」 は、インスタレーション空間内で、ダンサーと観客が参加して成立する、斬新なダンス公演。観客は、動きをプログラムで完全制御されたパーソナルモビリティに各自が座って体験する。観客はまず、ヘッドセット型のバーチャルリアリティのディスプレイを装着し、パーソナルモビリティ「WHILL」に乗り込む。会場には無線で制御された10台のWHILLがあり、自動生成によるコンピュータープログラムによってステージ上を個別の速さや動きでバラバラに移動。ダンスが始まると、観客の装着したディスプレイの視界には、バーチャルのステージ空間とそれに重ね合わされるAR(オーグメンテッドリアリティ:拡張現実)そして装着したカメラ映像による実写風景が映し出され、各自が動き回りながらそれぞれの視点からステージを鑑賞することになる。
ステージ上ではELEVENPLAYの複数のダンサーがWHILLと空間を共有しながらダンスを展開する。ARの中で、リアルタイムのダンサーの存在とバーチャルなデータ上のダンサーの動きがシームレスにトランジションする様子が目の前に繰り広げられていき、WHILLに乗った観客は、リアルとバーチャルの区別がつかなくなる迷宮的な空間表現と身体表現の世界に酔いしれることになる。時折、ダンサーが観客に接触する仕草をおこなうことで、観客はリアルな身体存在に引き戻される。このインスタレーション空間は、ステージの外から鑑賞することができる形でも公開される。
"border" is an innovative style of dance performance since it consists of the participation of both dancer and audience. Audiences are able to feel the installation space with their senses by riding personal mobilities all controlled by the system. 10 WHILLs -the personal mobility audiences ride- are prepared, so that 10 people can join at one time of the performance. This means, audiences who do not ride WHILL can enjoy the stage from outside of the performance area. Those who rides WHILL, put on headset formed display of virtual reality at first, and the small camera is set up on the surface of that display. These WHILLs are controlled by the system through radio wave, so that people cannot control them directly, and 10 WHILLs move on the stage in each speed and movement by the control of automatic computer programs. "WHILL" is produced by the WHILL Inc. With the start of dance performance, audiences can see the virtual space of the stage through the display they put on, and also the real scene through the camera set up on the headset formed display.
AR, augmented reality, enables people to enjoy both scenes of virtual and real, so that they can appreciate the stage from each point of view while each WHILL moves around. On the stage, some dancers of ELEVENPLAY share the space with WHILL, and perform their dance. In the AR, the seamless transition of real movements of real dancers and virtual movements of virtual dancers on the data, are showed to audiences on the WHILL. Then audiences can be greatly excited by the both worlds of spatial and physical expression. Since real dancers touch audiences sometimes, they may be pulled back to the real space, and realized themselves as the real physical existence. This installation space will be presented as one that can be appreciated from outside of the stage as well.

Work Technical Details

software

borderで体験者が装着するヘッドマウントディスプレイとそのソフトウェアは、
以下の要件を満たすためにOculus Rift DK2をベースに新規に開発した。

・従来のHMDは机の前など極めて狭いエリアのトラッキングしかできない。そのため反射マーカーを取り付け、光学式モーションキャプチャーシステムを統合することで、アクティングエリア(約8m x 14m)全領域をカバーできるHMDヘッドトラッキングシステムを開発。HMDにステレオカメラを装着することでARを実現。具体的には、PlayStation EyeのレンズをAR用に広角レンズに交換しカスタマイズしたものを2つ使用し、ステレオカメラを構成した。さらに、アクティングエリアにおいて実写とCGが完全に一致するように、10組すべてのステレオカメラについてそれぞれ厳密なキャリブレーションを行った。
・ヘッドトラッキングはモーションキャプチャーシステムとOculusRiftのモーションセンサー出力値を統合して使用することで、精度と安定性を担保した。更に、体験者のHMDのVR酔いを避け、ステレオカメラのレイテンシーを感じさせない工夫として、ステレオカメラのレイテンシーを計測したうえで、その間発生した姿勢の移動量の分だけカメラの映像の描画平面(rendering plane)を逆方向にずらすという処理を実行した。
・VR酔いを避けるために、たとえCGのみのVRシーンにおいても、現実の加速度を無視したCG空間内のみでの移動は行わないようにした。
・体験者がきちんとしたコンテンツを視聴できていることを担保するためにHMDの出力画像を圧縮して伝送し、スタッフが監視することで、体験者の体験の質の担保にも務めた。

HMD, a head mount display, and its software used
for border was newly developed based on Oculus Rift DK2 to satisfy following criteria.

Large scale tracking: Conventional HMD can only be tracked in narrow area such as a desk space. To realize large scale tracking that covers whole acting area, 8 meters in length and 14 meters in width, we attached retroreflective markers and integrated optical motion capture system into HMD tracking system.
Immersive AR with stereo cameras: We realized AR expressions by attaching stereo camera to HMD. For the stereo camera, we utilized two PlayStation Eye cameras. Their lenses are replaced by wide angle lens for AR. Moreover, for perfectly matching the real object and the virtual object within acting area, we carried out exhaustive calibration for all 10 stereo camera pairs.
Comfort head tracking & No VR sickness: For head tracking, both motion sensor output built in Oculus Rift DK2 and motion capture system output are integrally utilized and thus achieved high accuracy and stability. Moreover, to avoid VR sickness and not to feel latency of stereo camera, we measured stereo camera latency, and stereo camera rendering plane is shifted by inverse of head rotation amount during the latency period. In addition, we carefully created the CG scenes not to include shaky movement in VR space that does not match to real acceleration.
Guarantee experience quality under remote control condition: We developed a watchdog system that transfers compressed HMD screen output and check whether there are any troubles.

hardware

We control ten WHILLs and five white cubes by serial communication thorough radio wave. They are measured their position and orientation by capturing retroreflective markers, which are installed on the surface, through our system. Also, the system calibrates the difference between ideal and actual current position each time, and then we send them the data as a feedback through the wireless communication. As a result, WHILLs and cubes can change their position and move in different speed. Under the white boxes, they are customized and installed 4WD omni wheel. We develop its device as a machinery so that wheel can freely rotate on 2 axis. The WHILL is controlled the movement of back­forward and the turn of right­left by using a steering information.
On the stage, 9 meters in length and 15 meters in width, we precisely get required accuracy of their control by 1 centimeter or less for avoiding the collision between dancers, audiences and moving objects. We also install retroreflective markers on Oculas that audiences put as their headset display, and capture Oculas’s position and orientation to reflectively composite a point of view with actual space and AR/VR in each time. All the WHILLs, cubes Oculas and macbook are controlled by radio waves so that we also get requiered to build the stable wireless network.
Technical Requirements

personal mobilities "WHILL" * 10 / Oculus Rift DK2 (or, enhanced headmount despley more than that * 10)
Stereo Camera (made with Sony Playstation PS Eye) x 10 / MacBookPro Retina (on each WHILL) * 10
Stereo HeadPhone * 10

Optitrack Motion Capture System

LED Control System / 4ch Speaker System / MacBookPro Retina(Main Sequence)
MacBookPro Retina (Audio) / MacBookPro Retina (Led Sequence) / MacBookPro Retina(LED Control)
MacBookPro Retina(Visual Master) / MacBookPro Retina(Hardware controll) / Windows Laptop(Optitrack Motion Capture System)
Windows Laptop(LED Control System) / MacBookPro Retina(HMD Watchdog, Laser Control) / Omni wheel mobility device * 5
Projector (12K lumen) * 1 / Laser Projector * 3 / Smoke Machine * 1
Gigabit Ether Hub * 1 / 5GHz WIFI AccesPoint * 1

Performance

border 360AR

360度映像で「border」の世界を体験できる「border 360AR」ショートバージョン。
ドラッグ操作や、スマートフォンを動かすことで視点を変えることができる360度の3Dバーチャルリアリティ(VR)動画です。

Border at Spiral

2015年12月に東京・スパイラルホールで行われた「border」公演より

Profile

ELEVENPLAY
ELEVENPLAY
Dance company directed by choreographer & art director, MIKIKO. ELEVENPLAY was founded by MIKIKO in 2009, in the hopes of creating dancers who posses highly artistic sense and creativity on top of exquisite techniques, body and spirit. Composed of female dancers from a variety of genre, ELEVENPLAY's methods of expression are diverse, including stages, video works, and still photos.
http://elevenplay.net/
Rhizomatiks Research
Rhizomatiks Research
Rhizomatiks Research mainly takes up projects focusing on the field of research and development, and sets priority in opening a new expression for the future. The team which is leading Rhizomatiks Research, has produced projects with artists such as Perfume and Elevenplay in addition to media-art and data-art projects, and is also in charge of all processes through the projects―planning, implementation and operation.
https://rzm-research.com/
evala
evala
Debut in 2006. Evala has recorded and released cutting-edge electronic music, and created installations and concert performances at music festivals or art exhibitions in Japan and abroad. Also he works on the sound designing and producing by using stereophonic sound system or cutting-edge technology for the public spaces, performances, movies, and various advertising media.
http://evala.jp/
Takayuki Fujimoto a.k.a. Kinsei
Takayuki Fujimoto a.k.a. Kinsei
Kinsei R&D Director, Lighting Designer. Kinsei began participating in projects of the performance art group Dumb Type in 1987. After 2003 he has participated primarily as the lighting designer in performance and multimedia works by a number of overseas artists that make use of the unique potential of LED lighting. For his well-known work "true/本当のこと" in 2007, he worked put together a multi-talented technical team to focus on integrating LED lighting with other digital devices in order to create a highly organic stage work.
http://kinsei.asia/

Credit

Support received from