Video Chat with Unity3D, the ARFoundation Version

Many of you took steps in creating your very own video chat app using our How To: Create a Video Chat App in Unity tutorial. Now that you’ve got that covered, let’s take your app up a couple notches by adding an immersive experience to it. In this Augmented Reality (AR) tutorial, you’ll learn how to communicate and chat with friends in AR. The concept is very similar to the previous tutorial with a few changes to make it work in less than an hour.

Prerequisites

  • Unity Editor (Version 2018 or above)
  • 2 devices to test on (one to broadcast, one to view)
  • Broadcast device will be a mobile device to run AR scene: Apple device of iOS 11 or above; Android device with Android 7 or above
  • Viewing device will run the standard Video Chat demo app – pretty much any device with Windows, Mac, Android, or iOS operating systems will work
  • developer account with Agora.io

Getting Started

To start, we will need to integrate the Agora Video SDK for Unity3D into our project by searching for it in the Unity Asset Store or click this link to begin the download.

Video SDK on Asset Store

After you finish downloading and importing the SDK into your project, you should be able to see the README.md files for the different platforms the SDK supports. For your convenience, you can also access the quick start tutorials for each platform below.

Unity AR Packages

On Unity Editor, open Package Manager from the Window tab. Install the following packages:

For Unity 2018:

  • AR Foundation 1.0.0 — preview.22 (the latest for 1.0.0)
  • ARCore XR Plugin 1.0.0 — preview.24 (the latest for 1.0.0)
  • ARKit XR Plugin 1.0.0-preview.27 (the latest for 1.0.0)

For Unity 2019:

  • AR Foundation 2.0.2
  • ARCore XR Plugin 2.0.2
  • ARKit XR Plugin 2.0.2

Modify the Existing Project

Modify the Play Scene

Open up the TestSceneHelloVideo scene. Take out the Cube and Cylinder. Delete the Main Camera since we will use an AR Camera later.

Test Scene — before

On the Hierarchy panel, create a “AR Session” and “AR Session Origin”. Click on the AR Camera and change the tag to “Main Camera”, then create a Sphere 3D object. Modify the transform position to (0,0,5.67) so it can be visible in your Editor Game view and Save.

Test Scene — after

Unlike the Cube or Cylinder, the purpose of this sphere is just for positional reference. You will find this sphere in your AR view when running on a mobile device. We will need to add video view objects relative to this sphere’s position by code.

Modify Test Scene Script

Open TestHelloUnityVideo.cs, change the onUserJoin() method to generate a cube instead of a plane. We will add a function to provide a new position for each new remote user joining the chat.

Modify User Joined Delegate

In the Join() method, add the following line in the “enable video” section:

mRtcEngine.EnableLocalVideo(false)

This call disables the front camera, so it won’t get into conflict with the back camera which is used by the AR Camera on the device.

Last but not the least, fill in your APP ID for the variable declared in the beginning section of the TestHelloUnityVideo class.

If you haven’t already, go to Agora.io, log in and get an APP ID (it’s free). This will give you connectivity to the global Agora Real-time Communication network and allow you to broadcast across your living room or around the world, also your first 10,000 minutes are free every month.

Build Project

The configuration for building an ARFoundation enabled project is slightly different from the standard demo project.

Here is a quick checklist of things to set:

IOS:

  • Rendering Color Space = Linear
  • Graphics API = Metal
  • Architecture = ARM64
  • Target Minimum iOS Version = 11.0
  • A unique bundle id

Android:

  • Graphics API = GLES3
  • MultiThreaded Rendering = off
  • Minimum API Level = Android 7.0 (API level 24)
  • Create a new key store in the Publishing Settings

**Before running, to avoid colliding libraries, you will need to deactivate the files inside of either Assets/Plugins/x86 or Assets/Plugins/x86_64 folder by opening it and highlighting the files inside, unchecking them and then hitting the Apply button. The following screenshot shows that the x86_64 libraries are chosen to be deactivated.

Deactivating x86_64 library files

Now build the Application to either iOS or Android. Run the standard demo application for the remote users, from any of the four platforms that we discussed at the beginning of this tutorial. To test your demo, stand up and use the device to look around you and you should find the sphere. This indicates that you have successfully created an AR scene. A joining remote user’s video will now be placed on the cubes next to the sphere. That’s it, you can now enjoy chatting with your friends in AR!

Great job! You’ve built a simple AR world of video chatters!

The ARFoundation Demo

All Done!

Thank you for following along. If you have any questions please feel free to leave a comment on our Slack channel agoraiodev.slack/unity-help-me.

Other Resources