Testing Agora vs. Vonage for Multi-Party Web Video Calls

June 13, 2023 By Team Agora In Business, Developer

This blog summarizes the testing results comparing Agora versus Vonage for multi-party web video calls with 32 participants. The testing covers the most important considerations when assessing video SDKs, from analyzing performance under constrained network conditions to maximizing throughput given available bandwidth while keeping central processing unit (CPU) and random-access memory (RAM) usage as low as possible.

Test Configuration and Scenarios

Before we take a deep dive into the results, let’s review the test setup, configuration and scenarios covered in the testing of Agora and Vonage for multi-party web video calls.

Test Setup

	Agora	Vonage
Network Connectivity	Wi-Fi	Wi-Fi
Participants in Sessions	32	32
Versions Tested	Agora Web Demo (SDK 4.17.0)	Vonage WebRTC Demo
Receive Test PC Specification	Receiver: HP ProBook 630 G8 Notebook PC Processor: Intel Core i7-1185G7 @ 3.00 GHz Memory: 16 GB 2400 MHz DDR4	Receiver: HP ProBook 630 G8 Notebook PC Processor: Intel Core i7-1185G7 @ 3.00 GHz Memory: 16 GB 2400 MHz DDR4
Video Configuration for Receive Test PC	Receive 640×480 @ 30 FPS	Receive 640×480 @ 30 FPS
Participant Devices	Mac or Windows PCs	Mac or Windows PCs

For Agora, we tested using Agora Web Demo (SDK 4.17.0). For Vonage, we tested leveraging the most recent version of the Vonage WebRTC Demo , to ensure that Vonage best practices were properly implemented for comparison.

Test Case Scenarios

Test Case	Description
Unlimited	Normal network conditions
Limited500K	The test starts with no bandwidth limitations, then a low bandwidth 500 Kbps is applied, first on the send side, then applied on the receiver side.
UL25%PL	Uplink network with impairment of 25% packet loss
DL25%PL	Downlink network with impairment of 25% packet loss
Limited500K25%PL	The test starts with no bandwidth limitations, then a low bandwidth 500 Kbps with 25% packet loss is applied, first on the send side, then applied on the receiver side.
UL600msJitter	Uplink network with impairment of 600ms jitter
DL600msJitter	Downlink network with impairment of 600ms jitter

Results Overview

Here’s a quick overview of the results for each testing scenario before we dive into the full details for each one:

Under normal network conditions: The frame rate (FPS) for each was steady at 28 FPS for Agora and 27 FPS for Vonage.
Packet loss and jitter: Agora consistently achieved a better frame rate than Vonage under numerous conditions, including when the network had uplink/downlink packet loss of 25%, as well as in cases where the uplink/downlink experienced jitter of 600ms.
Limited bandwidth: When throttling to 500 Kbps on the send side, and then on the receive side, Agora and Vonage performed in a similar way while the limitation was applied. After the limitation was removed Agora had a more stable frame rate with an average of 27 FPS, while Vonage fluctuated more, at times dipping to 15 FPS.
25% packet loss with 500 Kbps network bandwidth limitation: While the limitation was applied Agora performed better than Vonage, recovering to an average frame rate of approximately 13 FPS while Vonage remained near 0 FPS with frozen video. Once the limitation was removed, Agora and Vonage recovered to 15 FPS and then to approximately 27 FPS. After recovery, to 27 FPS, Agora remained stable while Vonage fluctuated between near 27 FPS and 15 FPS.
CPU and RAM: Agora and Vonage had similar results for CPU consumption, while Agora consumed slightly more RAM than Vonage, at an average of 1.5% vs 1% of available RAM, respectively.

Frame Rate Testing

In these tests, we again focused primarily on the effect of packet loss and limiting network conditions on the send and receive frame rates.

Normal Network Conditions

As you can see in the graph below, with the network functioning under normal conditions, the frame rate was steady at 28 FPS for Agora and 27 FPS for Vonage. The overall experience for end users was similar under normal network conditions.

25% Uplink Packet Loss

When introducing 25% uplink packet loss, Agora remains steady at 15 FPS while Vonage begins lagging at 12 FPS.

25% Downlink Packet Loss

With with 25% downlink packet loss, Agora continues to maintain 17 FPS while the performance of Vonage is impacted significantly, down to 1 FPS resulting in choppy video.

Uplink 600ms Jitter

A significant variation in latency can occur when sending traffic over an IP network. When simulating conditions of a highly congested network having an uplink jitter of 600ms, Agora outperforms Vonage, holding steady at 8 FPS. In comparison Vonage drops to 5 FPS.

Downlink 600ms Jitter

When downlink jitter of 600ms was added, Agora held steady at 9 FPS with Vonage dropping to 5 FPS.

Frame Rate Recovery, 500 Kbps Limit

This test starts without any bandwidth limitations, and then after 60 seconds the network is throttled to 500 Kbps in the upstream direction, and then in the downstream direction for a total of 30 seconds. Under these challenging conditions, Agora and Vonage performed in a similar way while the limitation was applied. After the limitation was removed Agora had a more stable frame rate with an average of 27 FPS, while Vonage fluctuated more, at times dipping to 15 FPS.

Frame Rate Recovery, 500 Kbps Limit and 25% Packet Loss

This test starts without any bandwidth limitations or simulated packet loss. After 60 seconds, the network is throttled to 500 Kbps in the upstream direction along with 25% simulated packet loss. These conditions are then applied in the downstream direction for a total of 30 seconds. While the limitation was applied Agora performed better than Vonage, recovering to an average frame rate of approximately 13 FPS while Vonage remained near 0 FPS with frozen video. Once the limitation was removed, Agora and Vonage recovered to 15 FPS and then to approximately 27 FPS. After recovery, to 27 FPS, Agora remained stable while Vonage fluctuated between near 27 FPS and 15 FPS.

Agora has several ways to combat network disruptions and costly packet loss. Most notably, these include leveraging our SD-RTN™ network as an overlay to the public internet, which routes traffic around impairments on the internet using AI algorithms and optimally shapes real-time traffic for the best performance.

Agora also implements technologies to smooth out the effects of packet loss to optimize the end-user experience.

CPU Utilization

Processor: Intel Core i7-1185G7 @ 3.00 GHz

Test Case ID	Agora	Vonage
Unlimited	5.32%	5.51%
Limited500K	5.28%	5.46%
UL25%PL	5.22%	5.48%
DL25%PL	5.35%	5.45%
Limited500K25%PL	5.23%	5.43%
UL600msJitter	5.16%	5.37%
DL600msJitter	5.25%	5.32%

Agora consumed slightly less CPU resources vs. Vonage in all scenarios, with an average of 5.26% utilization vs. Vonage at an average of 5.43%.

RAM Utilization

Test Case ID	Agora	Vonage
Unlimited	232 MB	162 MB
Limited500K	231 MB	161 MB
UL25%PL	233 MB	154 MB
DL25%PL	223 MB	155 MB
Limited500K25%PL	254 MB	163 MB
UL600msJitter	227 MB	155 MB
DL600msJitter	228 MB	157 MB

Agora consumed slightly more RAM than Vonage, at an average of 1.5% vs. 1% of available RAM, respectively.

Conclusion

This blog summarizes the test results comparing Agora and Vonage across varying scenarios and conditions experienced when implementing video SDKs for multi-party web use cases. When it comes to providing real-time communication tools, the ability to deliver a high-quality and consistent end user experience is critical.

At Agora, our customers—ranging from some of the largest social media companies in the world to bootstrapped startups—continue to leverage our SDKs to easily integrate video, voice, and interactive live streaming experiences into any app. We empower developers to deliver best-in-class real-time experiences, with global delivery and scale.