• HoloLens1 10.0.17763.316 / HoloLens2 22621.1399
• Windows 10 SDK 10.0.19041.0 / 10.0.22621.0
• Unity 2021.3.35f1 (Built-in Render Pipeline / OpenXR 1.11.1 / MRTK 2.8.3 / DirectX 11 / Visual Studio 2022 MSVC v143)
• Unity 2021.3.35f1 (Built-in Render Pipeline / OpenXR 1.11.1 / MRTK 3.2.2 / DirectX 11 / Visual Studio 2022 MSVC v143)
• Mixed Reality Feature Tool
• OpenCV for Unity 3.0.0+
• EnoxSoftware/HoloLensCameraStream

The setup procedure for HoloLens projects using Unity and MRTK has changed significantly. The legacy Mixed Reality Feature Tool no longer functions on the latest Windows 11 environments.

To run this project, you must import the following assets:

• Unity OpenXR Plugin 1.14.3
• Microsoft Mixed Reality OpenXR Plugin 1.11.2
• Mixed Reality Graphics Tools 0.8.1
• MRTK3

Please install the packages in the following specific order to ensure all dependencies are resolved correctly:

1. Unity OpenXR Plugin
   • Add the following line to the dependencies section of the manifest.json file: "com.unity.xr.openxr": "1.14.3",.
2. Microsoft Mixed Reality OpenXR Plugin
   • Add the following line to the dependencies section of the manifest.json file: "com.microsoft.mixedreality.openxr": "1.11.2",.
3. MRTK Graphics Tools
   • Add the following line to the dependencies section of the manifest.json file: "com.microsoft.mrtk.graphicstools.unity": "https://github.com/microsoft/MixedReality-GraphicsTools-Unity.git?path=/com.microsoft.mrtk.graphicstools.unity#v0.8.1",.
4. MRTK Core Definitions (com.microsoft.mrtk.core)
   • Download the latest tarball package from the MRTK3 GitHub repository and import it using UPM's Add package from tarball... option.
5. MRTK Input (com.microsoft.mrtk.input)
   • Download the latest tarball package from the MRTK3 GitHub repository and import it using UPM's Add package from tarball... option.
6. MRTK UX Core Scripts (com.microsoft.mrtk.uxcore)
   • Download the latest tarball package from the MRTK3 GitHub repository and import it using UPM's Add package from tarball... option.
7. MRTK Spatial Manipulation (com.microsoft.mrtk.spatialmanipulation)
   • Download the latest tarball package from the MRTK3 GitHub repository and import it using UPM's Add package from tarball... option.
8. MRTK Standard Assets (com.microsoft.mrtk.standardassets)
   • Download the latest tarball package from the MRTK3 GitHub repository and import it using UPM's Add package from tarball... option.
9. MRTK UX Components (com.microsoft.mrtk.uxcomponents)
   • Download the latest tarball package from the MRTK3 GitHub repository and import it using UPM's Add package from tarball... option.
10. MRTK UX Components (Non-Canvas) (com.microsoft.mrtk.uxcomponents.noncanvas)
    • Download the latest tarball package from the MRTK3 GitHub repository and import it using UPM's Add package from tarball... option.

• Choosing a Unity version and XR plugin

1. Download the latest release unitypackage. HoloLensWithOpenCVForUnityExample.unitypackage
2. Create a new project. (HoloLensWithOpenCVForUnityExample)
   • Change the platform to UWP in the "Build Settings" window.
3. Import and setup the Microsoft Mixed Reality Toolkit. (Recommend using MixedRealityFeatureTool)
4. Import the OpenCVForUnity.
   • Select MenuItem[Tools/OpenCV for Unity/Open Setup Tools].
   • Click the [Move StreamingAssets Folder] button.
   • Leave the following files and delete the rest. ("StreamingAssets/OpenCVForUnity/objdetect/haarcascade_frontalface_alt.xml", "lbpcascade_ frontalface.xml")
5. Clone HoloLensCameraStream repository.
   • Copy the "HoloLensCameraStream/HoloLensVideoCaptureExample/Assets/CamStream/" folder to the "Assets/" folder.
6. Import the HoloLensWithOpenCVForUnityExample.unitypackage.
7. Add the "Assets/HoloLensWithOpenCVForUnityExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
8. Configure settings in the "Project Settings" window.
   • Add Define Symbols: the following to Scripting Define Symbols depending on the XR system used in your project; Legacy built-in XR: BUILTIN_XR; XR Plugin Management (Windows Mixed Reality): XR_PLUGIN_WINDOWSMR; XR Plugin Management (OpenXR):XR_PLUGIN_OPENXR.
   • Enable WebCam Capabilties in Publishing settings tab.
9. (Optional) Setup a performance environment suitable for Holorens. (See https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/recommended-settings-for-unity)
10. Build the project: You can now build the Unity project, which generates a Visual Studio Solution (which you will then have to also build). With the Build Settings window still open, click Build. In the explorer window that appears, make a new folder called App, which should live as a sibling next to the 'Assets` folder. Then click Select Folder to generate the VS solution in that folder. Then wait for Unity to build the solution.
11. Open the VS Solution: When the solution is built, a Windows explorer folder will open. Open the newly-built VS solution, which lives in App/HoloLensWithOpenCVForUnityExample.sln. This is the solution that ultimately gets deployed to your HoloLens.
12. Configure the deploy settings: In the Visual Studio toolbar, change the solution platform from ARM to x86 if you are building for Hololens1 or to ARM64 if you are building for Hololens2; Change the deploy target (the green play button) to Device (if your HoloLens is plugged into your computer), or Remote Machine (if your HoloLens is connected via WiFi).
13. Run the app: Go to Debug > Start Debugging. Once the app is deployed to the HoloLens, you should see some confirmation output in the Output window.
    • (Print the AR marker "CanonicalMarker-d10-i1-sp500-bb1.pdf" on an A4 size paper)

• Choosing a Unity version and XR plugin
• Known issues in Unity versions and packages

1. Download the latest release unitypackage.
2. Create a new project. (HoloLensWithOpenCVForUnityExample)
3. Import the Microsoft Mixed Reality Toolkit.
   • Add MRTK2 (Mixed Reality Toolkit - Mixed Reality Toolkit Foundation) and OpenXR(Platform Support - Mixed Reality OpenXR Plugin) to the project using "Mixed Reality Feature Tool".
   • Follow the MRTK2 configuration dialog to set up the project. (XR System: OpenXR)
4. Import the OpenCVForUnity.
5. Import the HoloLensCameraStream.
6. Import the HoloLensWithOpenCVForUnityExampleMRTK2.unitypackage.
7. Add the "Assets/HoloLensWithOpenCVForUnityExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
8. Configure settings in the "Project Settings" window.
   • Add XR_PLUGIN_OPENXR to the list of Define Symbols.
   • Enable WebCam Capabilties in Publishing settings tab.
9. Build the project; open it in VS2022, set the deployment target to ARM64 and deploy it to the Hololens2 actual device.

• Set up a new OpenXR project with MRTK

1. Download the latest release unitypackage.
2. Create a new project. (HoloLensWithOpenCVForUnityExample)
3. Import the Microsoft Mixed Reality Toolkit.
   • Add MRTK3 (MRTK3 - MRTK Input, MRTK UX Components, MRTK UX Components (Non-Canvas)) and OpenXR(Platform Support - Mixed Reality OpenXR Plugin) to the project using "Mixed Reality Feature Tool".
   • Set up the MRTK3. (XR System: OpenXR)
4. Import the OpenCVForUnity.
5. Import the HoloLensCameraStream.
6. Import the HoloLensWithOpenCVForUnityExampleMRTK3.unitypackage.
7. Add the "Assets/HoloLensWithOpenCVForUnityExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
8. Configure settings in the "Project Settings" window.
   • Add XR_PLUGIN_OPENXR to the list of Define Symbols.
   • Enable WebCam Capabilties in Publishing settings tab.
9. Build the project; open it in VS2022, set the deployment target to ARM64 and deploy it to the Hololens2 actual device.

• Setting up a new Unity project with MRTK3

Project Assets	Build Settings

Player Settings