Skip to content
How to Add Chat with Face Filters to Your Multiplayer Unity Game Featured
← Back to Blog

How to Add Chat with Face Filters to Your Multiplayer Unity Game

By Author: Rick Cheng In Developer

Do you have any thoughts about adding Real-Time Engagement in your game? Undoubtedly, having video chat during a co-op or multiplayer online game adds jokes and laughs to the experience. Extend the fun with a face filter!

In this tutorial, we will make use of an existing project, and apply a face filter feature to the player. To follow what has been done, please check out the tutorial here. Make sure you can download the full project from GitHub to start. For comparison to the original project, consider the following preview:

Prerequisites

You need a Banuba client token to run the project. Contact info@banuba.com for a trial token and SDK package if you don’t want to pay up front at the Asset Store.

Getting Started

First, we will use the Banuba Face AR SDK to start the AR face masking. Import the package that you downloaded from the Asset Store or the unzipped SDK to the Unity project directory. You should see the current project directory structure and find the BanubaClientToken file in the Resource folder. Open the file in a text editor and paste your SDK token key there.

Next, enter your Photon project info in Window > Photon Unity Networking > PUN Wizard > Setup Project. Paste your AppID or the email address you used for your Photon account.

Note that the GitHub project already contains a copy of the Agora Video SDK. You may choose to update to the latest from the Asset Store. We recommend that you delete the folder Assets/AgoraEngine/Plugins for a clean replacement.

Create BNB_AR Prefab

Open the Sample scene from BanabuFaceAR/Scenes, and do the following:

  1. Create an empty game object and name it “BNBARStuff”.
  2. Drag and drop items Camera, Faces, and Surface into BNBARStuff.
  3. Change layers of BNBARStuff to “default” recursively.
  4. Update the Camera component to culling mask on “default” only.
  5. Drag the BNBARStuff object to the Prefabs folder to make it a prefab.
  6. Open the prefab, rename Camera to “ARCamera”, and rename Surface to “ARSurface”. Change ARCamera’s culling mask to Default only. Deselect Default from MainCamera’s culling mask list.

If your Banuba Face AR SDK has an FPS game object under Surface, you can remove it from the prefab. Also update the code in CameraController to do a null check:

if (fpsTextObj != null) fpsTextObj.text = “FPS: “ + fps;

Important: Do not save the Sample scene.

Open VikingsScene from DemoVikings and set up the BNBARStuff object using the prefab:

  1. Drag and drop the BNBARStuff prefab into the Hierarchy.
  2. Enable the one of the filters under Faces (e.g., Afro).
  3. Highlight the Camera under BNBARStuff and play the app in Editor.
  4. Save the VikingsScene scene if everything looks good so far.

Note: You can observe yourself with the AR filter applied in the Camera preview.

Create a Render Texture

Create a render texture to host the ARCamera input:

  1. Create a RenderTexture; and name it “ARRenderTexture”.
  2. Assign ARRenderTexture into ARCamera’s Target Texture field.

Create Local View Prefab

In the Vikings scene, create a UI hierarchy as following:

  1. Create a new Image and change its width to 140 and its height to 140. Name it “ViewContainerWithMask”. Add a Mask component to this object.
  2. Add a new game object under ViewContainerWithMask and name it “AvatarRenderTarget”. Make sure its scale is set to 1.
  3. Add a RawImage component to AvatarRenderTarget. Drag and drop the ARRenderTexture into the Texture field of the RawImage component.
  4. Add a button that has no text and no image as a child object of AvatarRenderTarget. Set Normal Color and Pressed Color’s alpha to 0 (transparent). Set a low alpha (e.g., 35) for the other states. This button will invoke the appearance of the FilterDropdown.
  5. Attach this AvatarViewController.cs script to the AvatarRenderTarget object.
  6. Assign the ViewContainerWithMask object into the Mask RectTan field of the AvatarViewController component on the AvatarRenderTarget object.
  7. Verify that you set up as described in steps 1 through 4.
  8. Drag the game object ViewContainerWithMask to the Prefab folder. Delete it from the Vikings scene. Save.

Set Up a Drop-Down List for the Masks

In the VikingsScene, create a UI Dropdown and place it in the upper-right corner. Make sure you set the Canvas scaler to Scale with Screen. Assign the drop-down’s layer to UI and name it “FilterDropdown”. Hide this from the scene for now, since it will be activated later only when the user taps the self-video display.

Code Modification

Spawn with the Banuba FaceFilter-Enabled Prefab

In AgoraVideoChat.cs, update the CreateUserVideoSurface method with the following code snippet:

Update Prefab Field Assignments

Open Assets/DemoVikings/Resources/Charprefab and update the fields:

  1. Assign the prefab ViewContainerWithMask into the AvatarVideoPrefab field (see line 2 in the above code snippet).
  2. Update the AppID and the Channel ID.

Send the Frames as Custom Source Video

Add the following code snippet to the end of the AgoraVideoChat.cs script. The code runs at every frame to capture what is displayed on the ARCamera and sends it as a video frame into the channel.

Last Check

Since we created a bigger rectangle for the Avatar image view (140 pt), we should also update Asset/Prefabs/UserVideo’s RectTransform to 140×140. Also, in AvatarViewController, we provided an option for the zoom level. You may want to try 1.5x zooming for a better view of yourself in that window. Here is the look after everything is connected:

That’s it! With a face mask on, you look much cooler than before. Have fun playing with friends and talking to them at the same time with your cool new look!

In Summary

  • We connected to Agora’s network to display our video chat channel.
  • We enabled other users to join our party, see their faces, and talk with them in real time.
  • We took it one step further and let users pick a face mask provided by the Banuba ARFace Filter SDK.

If you have any questions or hit a snag in the course of building your own networked group video chat, please feel free to reach out directly or via the Agora Slack Channel!

Check out the link to the full project (in the CompletedProject branch of the same repo):

Blog Cta
Blog Cta mobile