How to Embed Group Video Chat in your Unity Games

May 7, 2020 By Joel Thomas In Developer

When you’re making a video game, you want to squeeze every last drop out of performance out of your graphics, code, and any plugins you may use. Agora’s Unity SDK has a low footprint and performance cost, making it a great tool for any platform, from mobile to VR!

In this tutorial, I’m going to show you how to use Agora to create a real-time video party chat feature in a Unity MMO demo asset using the Agora SDK and Photon Unity Networking (PUN).

By the end of the demo, you should understand how to download the Agora plugin for Unity, join/leave another player’s channel, and display your player party in a scalable fashion.

For this tutorial, I’m using Unity 2018.4.18.

Before we get started, let’s touch on two important factors:

Since this is a networked demo, you have two strategies for testing:

Use two different machines, each with their own webcam (I used a PC and Mac laptop)
From the same machine, test using one client from a Unity build, and another from the Unity editor (this solution is suboptimal, because the two builds will fight for the webcam access, and may cause headache).

Getting Started

Create an Agora developer account here to get your AppID
Create a Photon developer account for their appID
Import Agora SDK into your project from the Unity Asset Store
Import Photon Viking Multiplayer Showcase into your project

Create Agora Engine

Our “Charprefab” is the default Viking character we will be using. This object lives in Assets > DemoVikings > Resources.

It is already set up with Photon to join a networked lobby/room and send messages across the network.

Create a new script called AgoraVideoChat and add it to our CharPrefab.

In AgoraVideoChat let’s add this code:

using agora_gaming_rtc;
// *NOTE* Add your own appID from console.agora.io
[SerializeField]
private string appID = "";
[SerializeField]
private string channel = "unity3d";
private string originalChannel;
private IRtcEngine mRtcEngine;
private uint myUID = 0;
void Start()
{
   if (!photonView.isMine)
     return;
   // Setup Agora Engine and Callbacks.
   if(mRtcEngine != null)
   {
       IRtcEngine.Destroy();
   }
     originalChannel = channel;
     mRtcEngine = IRtcEngine.GetEngine(appID);
     mRtcEngine.OnJoinChannelSuccess = OnJoinChannelSuccessHandler;
     mRtcEngine.OnUserJoined = OnUserJoinedHandler;
     mRtcEngine.OnLeaveChannel = OnLeaveChannelHandler;
     mRtcEngine.OnUserOffline = OnUserOfflineHandler;
     mRtcEngine.EnableVideo();
     mRtcEngine.EnableVideoObserver();
     mRtcEngine.JoinChannel(channel, null, 0);
}
private void OnApplicationQuit()
{
    if(mRtcEngine != null)
    {
        mRtcEngine.LeaveChannel();
        mRtcEngine = null;
        IRtcEngine.Destroy();
    }
}

This is a basic Agora setup protocol, and very similar if not identical to the AgoraDemo featured in the Unity SDK download. Familiarize yourself with it to take your first step towards mastering the Agora platform! You’ll notice that photon.isMine is now angry at us, and that we need to implement some Agora callback methods.

We can include the proper Photon behavior by changing public class AgoraVideoChat : MonoBehaviour to public class AgoraVideoChat : Photon.MonoBehaviour

Agora has many callback methods that we can use which can be found here, however for this case we only need these:

    // Local Client Joins Channel.
    private void OnJoinChannelSuccessHandler(string channelName, uint uid, int elapsed)
    {
        if (!photonView.isMine)
            return;
        myUID = uid;
        Debug.LogFormat("I: {0} joined channel: {1}.", uid.ToString(), channelName);
        //CreateUserVideoSurface(uid, true);
    }
    // Remote Client Joins Channel.
    private void OnUserJoinedHandler(uint uid, int elapsed)
    {
        if (!photonView.isMine)
            return;
        //CreateUserVideoSurface(uid, false);
    }
    // Local user leaves channel.
    private void OnLeaveChannelHandler(RtcStats stats)
    {
        if (!photonView.isMine)
            return;
    }
    // Remote User Leaves the Channel.
    private void OnUserOfflineHandler(uint uid, USER_OFFLINE_REASON reason)
    {
        if (!photonView.isMine)
            return;
    }

Let’s play our “VikingScene” level now, and look at the log.
(You should see something in the log like: “[your UID] joined channel: unity3d”)

Huzzah!

We are in an Agora channel with the potential to video-chat with 16 other players or broadcast to around 1 million viewers!

But what gives? Where exactly are we?

Create Agora VideoSurface

Agora’s Unity SDK uses RawImage objects to render the video feed of webcams and mobile cameras, as well as cubes and other primitive shapes (see AgoraEngine > Demo > SceneHome for an example of this in action).

Create a Raw Image (right-click Hierarchy window > UI > Raw Image) and name it “UserVideo”.
Add the VideoSurface script to it (Component > Scripts > agora_gaming_rtc > VideoSurface).
Drag the object into Assets > Prefabs folder.
Delete the UserVideo object from the hierarchy (you can leave the canvas), we only wanted the prefab.

Add this code to AgoraVideoChat.

    // add this to your other variables
    [Header("Player Video Panel Properties")]
    [SerializeField]
    private GameObject userVideoPrefab;
    private int Offset = 100;
    private void CreateUserVideoSurface(uint uid, bool isLocalUser)
    {
        // Create Gameobject holding video surface and update properties
        GameObject newUserVideo = Instantiate(userVideoPrefab);
        if (newUserVideo == null)
        {
            Debug.LogError("CreateUserVideoSurface() - newUserVideoIsNull");
            return;
        }
        newUserVideo.name = uid.ToString();
        GameObject canvas = GameObject.Find("Canvas");
        if (canvas != null)
        {
            newUserVideo.transform.parent = canvas.transform;
        }
        // set up transform for new VideoSurface
        newUserVideo.transform.Rotate(0f, 0.0f, 180.0f);
        float xPos = Random.Range(Offset - Screen.width / 2f, Screen.width / 2f - Offset);
        float yPos = Random.Range(Offset, Screen.height / 2f - Offset);
        newUserVideo.transform.localPosition = new Vector3(xPos, yPos, 0f);
        newUserVideo.transform.localScale = new Vector3(3f, 4f, 1f);
        newUserVideo.transform.rotation = Quaternion.Euler(Vector3.right * -180);
        // Update our VideoSurface to reflect new users
        VideoSurface newVideoSurface = newUserVideo.GetComponent<VideoSurface>();
        if (newVideoSurface == null)
        {
            Debug.LogError("CreateUserVideoSurface() - VideoSurface component is null on newly joined user");
        }
        if (isLocalUser == false)
        {
            newVideoSurface.SetForUser(uid);
        }
        newVideoSurface.SetGameFps(30);
    }

Add the newly created prefab to the UserPrefab slot in our Charprefab character, and uncomment CreateUserVideoSurface() from our callback. methods.
Run it again! Now we can see our local video stream rendering to our game. If we call in from another agora channel, we will see more video frames populate our screen. I use the “AgoraDemo” app on my mobile device to test this, but you can also use our 1-to-1 calling web demo to test connectivity, or connect with the same demo from another machine.

We now have our Agora module up and running, and now it’s time to create the functionality by connecting two networked players in Photon.

Photon Networking – Party Joining

To join/invite/leave a party, we are going to create a simple UI.
Inside your CharPrefab, create a canvas, and 3 buttons named InviteButton, JoinButton, and LeaveButton, respectively.

Make sure the Canvas is the first child of the Charprefab.

Next we create a new script called PartyJoiner on our base CharPrefab object. Add this to the script:

    using UnityEngine.UI;
    [Header("Local Player Stats")]
    [SerializeField]
    private Button inviteButton;
    [SerializeField]
    private GameObject joinButton;
    [SerializeField]
    private GameObject leaveButton;
    [Header("Remote Player Stats")]
    [SerializeField]
    private int remotePlayerViewID;
    [SerializeField]
    private string remoteInviteChannelName = null;
    private AgoraVideoChat agoraVideo;
    private void Awake()
    {
        agoraVideo = GetComponent<AgoraVideoChat>();
    }
    private void Start()
    {
        if(!photonView.isMine)
        {
            transform.GetChild(0).gameObject.SetActive(false);
        }
        inviteButton.interactable = false;
        joinButton.SetActive(false);
        leaveButton.SetActive(false);
    }
    private void OnTriggerEnter(Collider other)
    {
        if (!photonView.isMine || !other.CompareTag("Player"))
        {
            return;
        }
        // Used for calling RPC events on other players.
        PhotonView otherPlayerPhotonView = other.GetComponent<PhotonView>();
        if (otherPlayerPhotonView != null)
        {
            remotePlayerViewID = otherPlayerPhotonView.viewID;
            inviteButton.interactable = true;
        }
    }
    private void OnTriggerExit(Collider other)
    {
        if(!photonView.isMine || !other.CompareTag("Player"))
        {
            return;
        }
        remoteInviteChannelName = null;    
        inviteButton.interactable = false;
        joinButton.SetActive(false);
    }
    public void OnInviteButtonPress()
    {
    //PhotonView.Find(remotePlayerViewID).RPC("InvitePlayerToPartyChannel", PhotonTargets.All, remotePlayerViewID, agoraVideo.GetCurrentChannel());
    }
    public void OnJoinButtonPress()
    {
        if (photonView.isMine && remoteInviteChannelName != null)
        {
            //agoraVideo.JoinRemoteChannel(remoteInviteChannelName);
            joinButton.SetActive(false);
            leaveButton.SetActive(true);
        }
    }
    public void OnLeaveButtonPress()
    {
        if (!photonView.isMine)
            return;
    }
    [PunRPC]
    public void InvitePlayerToPartyChannel(int invitedID, string channelName)
    {
        if (photonView.isMine && invitedID == photonView.viewID)
        {
            joinButton.SetActive(true);
            remoteInviteChannelName = channelName;
        }
    }

Add the corresponding “OnButtonPress” functions into the Unity UI buttons you just created.
[Example: InviteButton -> “OnInviteButtonPress()”]

Set the CharPrefab tag to “Player”.
Add a SphereCollider component to CharPrefab (Component bar > Physics > SphereCollider, check the “Is Trigger” box to true, and set it’s radius to 1.5.

Quick Photon Tip – Local Functionality

As you can see we need to implement two more methods in our AgoraVideoChat class. Before we do that, let’s cover some code we just copied over.

    private void Start()
    {
        if (!photonView.isMine)
        {
            transform.GetChild(0).gameObject.SetActive(false);
        }
        inviteButton.interactable = false;
        joinButton.SetActive(false);
        leaveButton.SetActive(false);
    }

“If this photon view isn’t mine, set my first child to False“ - It’s important to remember that although this script is firing on the CharPrefab locally controlled by your machine/keyboard input - this script is also running on every other CharPrefab in the scene. Their canvases will display, and their print statements will show as well.

By setting the first child (my “Canvas” object) to false on all other CharPrefabs, I’m only displaying the local canvas to my screen - not every single player in the Photon “Room”.

Let’s build and run with two different clients to see what happens…

…Wait, we’re already in the same party?

If you remember, we set private string channel = "unity3d" and in our Start() method are calling mrtcEngine.JoinChannel(channel, null, 0);. We are creating and/or joining an Agora channel named “unity3d”, in every single client right at the start.

To avoid this, we have to set a new default channel name in each client, so they start off in separate Agora channels, and then can invite each other to their unique channel name.

Now let’s implement two more methods inside AgoraVideoChat:
JoinRemoteChannel(string remoteChannelName) and GetCurrentChannel().

    public void JoinRemoteChannel(string remoteChannelName)
    {
        if (!photonView.isMine)
            return;
        mRtcEngine.LeaveChannel();
        mRtcEngine.JoinChannel(remoteChannelName, null, myUID);
        mRtcEngine.EnableVideo();
        mRtcEngine.EnableVideoObserver();
        channel = remoteChannelName;
    }

public string GetCurrentChannel() => channel;

This code allows us to receive events that are called across the Photon network, bouncing off of each player, and sticking when the invited Photon ID matches the local player ID.

When the Photon event hits the correct player, they have the option to “Join Remote Channel” of another player, and connect with them via video chat using the Agora network.

Test the build to watch our PartyJoiner in action!

Finishing Touches – UI & Leaving a Party

You have now successfully used Agora to join a channel, and see the video feed of fellow players in your channel. Video containers will pop in across your screen as users join your channel.

However, it doesn’t look great, and you can’t technically leave the channel without quitting the game and rejoining. Let’s fix that!

UI Framework

Now we’ll create a ScrollView object, to hold and organize our buttons.

Inside of Charprefab > Canvas: make sure CanvasScaler UI Scale Mode is set to “Scale With Screen Size” (by default it’s at “Constant Pixel Size” which in my experience is less than ideal for most Unity UI situations).

Inside our CharPrefab object, right-click on Canvas, select UI > Scroll View.

Set “Scroll View” Rect Transform to “Stretch/Stretch” (bottom right corner) and make sure your Anchors, Pivots, and Rect Transform match the values in the red box pictured above.
Uncheck “Horizontal” and delete the HorizontalScrollbar child object
Set “Content” child object to “Top/Stretch” (rightmost column, second from the top).
I have my Height set to 300.
Min X:0 Y:1
Max X:1 Y: 1
Pivot X:0 Y: 1
Create an empty gameobject named “SpawnPoint” as a child of Content - Set the Rect Transform to “Top/Center” (Middle column, second from the top) and set the “Pos Y” value to -20.
Make sure your Anchors: Min/Max and your Pivot values equal what is displayed.

In AgoraVideoChat add:

    [SerializeField]
    private RectTransform content;
    [SerializeField]
    private Transform spawnPoint;
    [SerializeField]
    private float spaceBetweenUserVideos = 150f;
    private List<GameObject> playerVideoList;

In Start() add playerVideoList = new List();

We’re going to completely replace our CreateUserVideoSurface method to:

    // Create new image plane to display users in party
    private void CreateUserVideoSurface(uint uid, bool isLocalUser)
    {
        // Avoid duplicating Local player video screen
        for (int i = 0; i < playerVideoList.Count; i++)
        {
            if (playerVideoList[i].name == uid.ToString())
            {
                return;
            }
        }
        // Get the next position for newly created VideoSurface
        float spawnY = playerVideoList.Count * spaceBetweenUserVideos;
        Vector3 spawnPosition = new Vector3(0, -spawnY, 0);
        // Create Gameobject holding video surface and update properties
        GameObject newUserVideo = Instantiate(userVideoPrefab, spawnPosition, spawnPoint.rotation);
        if (newUserVideo == null)
        {
            Debug.LogError("CreateUserVideoSurface() - newUserVideoIsNull");
            return;
        }
        newUserVideo.name = uid.ToString();
        newUserVideo.transform.SetParent(spawnPoint, false);
        newUserVideo.transform.rotation = Quaternion.Euler(Vector3.right * -180);
        playerVideoList.Add(newUserVideo);
        // Update our VideoSurface to reflect new users
        VideoSurface newVideoSurface = newUserVideo.GetComponent<VideoSurface>();
        if(newVideoSurface == null)
        {
            Debug.LogError("CreateUserVideoSurface() - VideoSurface component is null on newly joined user");
        }
        if (isLocalUser == false)
        {
            newVideoSurface.SetForUser(uid);
        }
        newVideoSurface.SetGameFps(30);
        // Update our "Content" container that holds all the image planes
        content.sizeDelta = new Vector2(0, playerVideoList.Count * spaceBetweenUserVideos + 140);
        UpdatePlayerVideoPostions();
        UpdateLeavePartyButtonState();
    }

and add two new methods:

// organizes the position of the player video frames as they join/leave
private void UpdatePlayerVideoPostions()
{
    for (int i = 0; i < playerVideoList.Count; i++)
    {
        playerVideoList[i].GetComponent<RectTransform>().anchoredPosition = Vector2.down * 150 * i;
    }
}
// resets local players channel
public void JoinOriginalChannel()
{
    if (!photonView.isMine)
            return;
    if(channel != originalChannel || channel == myUID.ToString())
    {
        channel = originalChannel;
    }
    else if(channel == originalChannel)
    {
        channel = myUID.ToString();
    }
    JoinRemoteChannel(channel);
}

Comment out the UpdateLeavePartyButtonState() for now, and drag in your newly created ScrollView UI objects into the appropriate slots.

Almost there!

Now all we have to do is add the methods for “Leave Party” functionality in AgoraVideoChat:

    public delegate void AgoraCustomEvent();
    public static event AgoraCustomEvent PlayerChatIsEmpty;
    public static event AgoraCustomEvent PlayerChatIsPopulated;
    private void RemoveUserVideoSurface(uint deletedUID)
    {
        foreach (GameObject player in playerVideoList)
        {
            if (player.name == deletedUID.ToString())
            {
                // remove videoview from list
                playerVideoList.Remove(player);
                // delete it
                Destroy(player.gameObject);
                break;
            }
        }
        // update positions of new players
        UpdatePlayerVideoPostions();
        Vector2 oldContent = content.sizeDelta;
        content.sizeDelta = oldContent + Vector2.down * spaceBetweenUserVideos;
        content.anchoredPosition = Vector2.zero;
        UpdateLeavePartyButtonState();
    }
    private void UpdateLeavePartyButtonState()
    {
        if (playerVideoList.Count > 1)
        {
            PlayerChatIsPopulated();
        }
        else
        {
            PlayerChatIsEmpty();
        }
    }

and update our AgoraVideoChat callbacks:

    // Local Client Joins Channel.
    private void OnJoinChannelSuccessHandler(string channelName, uint uid, int elapsed)
    {
        if (!photonView.isMine)
            return;
        myUID = uid;
        CreateUserVideoSurface(uid, true);
    }
    // Remote Client Joins Channel.
    private void OnUserJoinedHandler(uint uid, int elapsed)
    {
        if (!photonView.isMine)
            return;
        CreateUserVideoSurface(uid, false);
    }
    // Local user leaves channel.
    private void OnLeaveChannelHandler(RtcStats stats)
    {
        if (!photonView.isMine)
            return;
        foreach (GameObject player in playerVideoList)
        {
            Destroy(player.gameObject);
        }
        playerVideoList.Clear();
    }
    // Remote User Leaves the Channel.
    private void OnUserOfflineHandler(uint uid, USER_OFFLINE_REASON reason)
    {
        if (!photonView.isMine)
            return;
        if (playerVideoList.Count <= 1)
        {
            PlayerChatIsEmpty();
        }
        RemoveUserVideoSurface(uid);
    }

and in PartyJoiner:

    private void OnEnable()
    {
        AgoraVideoChat.PlayerChatIsEmpty += DisableLeaveButton;
        AgoraVideoChat.PlayerChatIsPopulated += EnableLeaveButton;
    }
    private void OnDisable()
    {
        AgoraVideoChat.PlayerChatIsEmpty -= DisableLeaveButton;
        AgoraVideoChat.PlayerChatIsPopulated -= EnableLeaveButton;
    }
    public void OnLeaveButtonPress()
    {
        if(photonView.isMine)
        {
            agoraVideo.JoinOriginalChannel();
            leaveButton.SetActive(false);
        }
    }
    private void EnableLeaveButton()
    {
        if(photonView.isMine)
        {
            leaveButton.SetActive(true);
        }
    }
    private void DisableLeaveButton()
    {
        if(photonView.isMine)
        {
            leaveButton.SetActive(false);
        }
    }

Play this demo in two different editors and join a party! We start off by connecting to the same networked game lobby via the Photon network, and then connect our videochat party via Agora’s SD-RTN network!

In Summary

We connected to Agora’s network to display our video chat channel.
We enabled other users to join our party, see their faces, and talk with them in real time.
We took it one step further and built a scalable UI that houses all the people you want to chat with!

If you have any questions or hit a snag in the course of building your own networked group video chat, please feel free to reach out directly or via the Agora Slack Channel!

Check out the link to the full github project here!

Sore eyes?

Go to the “misc” section of your settings and select night theme ❤️