Holo Video Chat & Guide

Holo Video Chat is a port of WebRTC Video Chat for the HoloLens 2. It adds Universal Windows Platform support for the C# API’s ICall, IMediaNetwork and IVideoInput.

It is available in the Unity Asset Store – here:

Important folders and files

Once imported all important files should be in the HoloVideoChat folder.

  • Plugins/WSA – Contains the UWP plugins for the C# API (ICall, IMediaNetwork & similar) and processor architecture specific plugins for the webrtc-uwp-sdk. These are only used in player builds but not within the Unity Editor. Note Microsoft no longer actively supports this SDK (or any other)
  • Scenes/holovideochat – Start scene for HoloLens 2 / MRTK 2 style example apps. This scene contains a menu and the HoloGuide+ UwpCallApp examples
  • Scripts – Contains all scripts used for the example applications
  • Scripts/HoloGuideApp.cs and HoloGuideUi.cs – These are the main scripts for the HoloGuide example apps
  • Scripts/UwpCallApp.cs – A CallApp example add adds a few UWP / HoloLens 2 specific flags
  • awrtc_win – contains a lite version of WebRTC Video Chat to allow testing in the Unity Editor.
  • awrtc_win/Plugins – Contains the x64 Win32 plugins used when running in the Unity Editor
  • awrtc_win/scripts – Contains shared scripts between WebRTC Video Chat and Holo Video Chat
  • awrtc_win/callapp – This contains the cross-platform CallApp example. It is designed to work with HoloLens2/UWP if used with HoloVideoChat but can also run on all other platforms if used with WebRTC Video Chat
  • awrtc_win/scripts/UnityCallFactory.cs – This is the main access point for audio, video & network features. If run via UWP this factory will return UWP specific implementations for the ICall, IMediaNetwork and IVideoInput interfaces. If used in the Unity Editor it will return versions for classic Win32 applications

Project setup

Setting up MRTK 2

Note if both WebRTC Video Chat and Holo Video Chat is needed it is recommended to use two separate projects.

Before importing Holo Video Chat the project should be setup for MRTK 2. Microsoft publishes a great tutorial to do this: Exercise – Import and configure resources . You can follow the tutorial with only one exception: Select “Mixed Reality Toolkit Examples” as well in Step 6.

For completeness here is a list for the manual setup steps:

  1. Create project using Unity 2021 LTS, 3D template (last tested version: 2021.3.29f1).
    Switch the Platform to UWP via the top menu -> File -> Build Settings -> Select Universal Windows Platform then Switch Platform
  2. Download and run MixedRealityFeatureTool.exe. It might take a while to load its catalog
  3. Press Start, enter project path, press Discover Features
  4. Select the following:
    Under Mixed Reality Toolkit select Mixed Reality Toolkit Examples (last tested version: 2.8.3)
    Under Platform Support select Mixed Reality OpenXR Plugin (last tested version: 1.9.0)
  5. Press Get Features & Import on the next page. (Mixed Reality Toolkit Foundation/Extensions and Standard Assets are included automatically)
  6. Press Approve and Exit, then switch back to the Unity Editor
  7. If it asks about enabling the new input system package press Yes (unity will restart)
  8. The MRTK Project Configurator should open automatically. Before following the configurator: Select in the Unity Top Menu bar -> Mixed Reality -> Project Validation Settings -> Make sure “HoloLens 2 Application (UWP)” is ticked
  9. In the MRTK Project Configurator: Press “Unity OpenXR plugin”. (If the window does not open select Mixed Reality on the top menu bar -> Toolkit -> Utilities -> Configure Project for MRTK)
  10. The project settings might open. Ignore this window for now. Continue via MRTK Project Configurator and select XR Plug-in Management.
    Tick OpenXR and Microsoft HoloLens feature group then close the project settings window again
  11. In MRTK Project Configurator press Apply Settings. (NOTE: If the Configurator still shows “Unity OpenXR plugin” then press Skip To Next Step and then Apply)
  12. In Player Settings -> XR Plugin-Management -> Project Validation: Press Fix All several times until all errors / warnings disappeared. Ignore any “Scene specific” warnings. Close Project Settings again.
  13. In MRTK Project Configurator press Next and then Apply
  14. Press Import TMP Essentials

Now the project should be fully set up for use with MRTK and HoloLens2. See below for Holo Video Chat specific setup.

Project setup for Holo Video Chat

After you made sure your project is setup for MRTK2 (above) you can import Holo Video Chat into your project, then follow steps beow:

In your UWP Project setting -Publisher Settings – ensure the following capabilities are selected:

  • InternetClient
  • InternetClientServer
  • PrivateNetworkClientServer
  • WebCam
  • Microphone
  • SpatialPerception
  • GazeInput

Testing, Building and using the examples

Testing in the Unity Editor

Open the scene holovideochat and press the start button. Below are a few essential controls available. See Input simulation service — MRTK2 for more detail.

  • Press the right mouse button & move the mouse to rotate the camera
  • Press the left mouse button to trigger any UI elements in the centre of the screen
  • Use W,S,A,D to move
  • Press Left shift while scrolling the mouse wheel to move a simulated hand. This can be used to interact with the UI elements as well.
  • Press Left shift + CTRL to rotate the simulated hand. Move the Palm up to trigger the Plam Gesture menu

Testing on the HoloLens 2

In the project settings ensure holovideochat is selected as the start scene and the build platform is Universal Windows Platform. Select ARM 32-Bit as Architecture and press Build to create a Visual Studio project.

Once completed open the project in VS, make sure ARM is selected (not ARM64!) as architecture and press run. Note Debug run quite poorly. Unless you try to reproduce a specific issues use Release builds for testing.

During the first start the HoloLens 2 will ask if you allow access to camera, microphone & eye tracker. These must be answered with yes, otherwise the example applications can not run.

Pitfalls and possible problems

Note Microsoft has declared their webrtc-uwp-sdk as obsolete and also stopped updating all other WebRTC specific projects. They never added ARM64 support. Because of this Holo Video Chat will likely remain limited to ARM 32 builds until a new SDK becomes available.

A few other features likely remain limited for now:

  • Only two Camera resolutions are available at the moment: 1280×720 and 2048×1152.
  • The SetVolume method only supports 0 – for no volume and values larger than zero for full volume
  • Performance of the VideoInput API is quite poor at the moment. It is recommended to keep the resolution at 720p to ensure stability. High resolutions can cause excessive usage of CPU/Memory and cause crashes
  • UWP supports H264 encoding but the x64 windows version doesn’t include a h264 decoder. This means this setting can not be used together with the Unity Editor for testing. HoloLens2 to browser connections do generally support h264.