Testing Agora vs. Vonage for 1:1 Mobile Video Calls

August 22, 2023 By Team Agora In Business, Developer

This blog summarizes the testing results of Agora versus Vonage for one-on-one mobile video calls. The testing covers the most important considerations when assessing video SDKs, from analyzing performance under constrained network conditions to maximizing throughput given available bandwidth while keeping central processing unit (CPU) and random-access memory (RAM) usage as low as possible.

For more Vonage comparisons, check out:

Test Configuration and Scenarios

Before we take a deep dive into the results, let’s review the test setup, configuration and scenarios covered in the testing of Agora and Vonage for 1:1 mobile video calls.

Test Setup

	Agora	Vonage
Network Connectivity	Wi-Fi, 5G cellular (ATT)	Wi-Fi, 5G cellular (ATT)
Participants in Sessions	1	1
Versions Tested	Agora Video SDK for iOS (version 4.2)	Vonage Video API (version 2.25.3) formerly TokBok OpenTok
Sender	iPhone 11 CPU Hexa-core (2×2.65 GHz Lightning + 4×1.8 GHz Thunder) Memory 4GB	iPhone 11 CPU Hexa-core (2×2.65 GHz Lightning + 4×1.8 GHz Thunder) Memory 4GB
Receiver	iPhone XS CPU Hexa-core (2×2.5 GHz Vortex + 4×1.6 GHz Tempest) Memory 4GB	iPhone XS CPU Hexa-core (2×2.5 GHz Vortex + 4×1.6 GHz Tempest) Memory 4GB
Video Configuration for Receive Test Device	480×544 @ 24fps	480×544 @ 24fps

For Agora, we tested using Agora Video SDK 4.2.0 for iOS. For Vonage, we tested leveraging Vonage Video API 2.25.3 for iOS, and ensured that Vonage best practices were properly implemented for comparison.

Test Case Scenarios

Test Case	Description
Unlimited	Normal network conditions
Limited500K	The test starts with no bandwidth limitations, then a low bandwidth 500Kbps is applied, first on the send side, then applied on the receiver side.
UL25%PL	Uplink network with impairment of 25% packet loss
DL25%PL	Downlink network with impairment of 25% packet loss
Limited500K25%PL	The test starts with no bandwidth limitation, then a low bandwidth 500 Kbps with 25% packet loss is applied, first on the send side, then applied on the receiver side.
UL600msJitter	Uplink network with impairment of 600ms jitter
DL600msJitter	Downlink network with impairment of 600ms jitter

Results

Here’s a quick overview of the results for each testing scenario before we dive into the full details for each one:

Under normal network conditions: Agora performed slightly better than Vonage under normal network conditions with Agora at 24 FPS and Vonage at 20 FPS.
Packet loss and jitter: Agora consistently achieved a better frame rate than Vonage under numerous packet loss and jitter conditions. Under these conditions, Agora’s FPS maintained 22 to 23 FPS in comparison with Vonage varying between 1 and 13 FPS.
Limited bandwidth: When the network was limited to low bandwidth, 500Kbps on the send side first, and then on the receiver side, Agora maintained an excellent 24 FPS during with no need for recovery. Vonage fell below 10 FPS and took 30 seconds to recover.
25% packet loss with 500 Kbps network bandwidth limitation: Agora maintained 20 to 24 FPS during the limitation with immediate recovery to stable 24 FPS while Vonage fell 0 to 5 FPS during the limitation.
CPU and RAM: Agora used less CPU resources than Vonage, but used more RAM.

Frame Rate Testing

In these tests, we again focused primarily on the effect of packet loss and limiting network conditions on the send and receive frame rates.

Normal Network Conditions

With the network functioning under normal conditions, the frame rate was steady at 24 FPS for Agora and 20 FPS for Vonage. The overall experience for end users was slightly smoother with Agora under normal network conditions.

Figure 1: FPS comparison for Agora and Vonage under a normal network condition. — **Figure 1:** FPS comparison for Agora and Vonage under a normal network condition.

25% Uplink Packet Loss

When introducing 25% uplink packet loss, Agora remains steady at 23 FPS while Vonage lags at 13 FPS.

Figure 2: FPS comparison for Agora and Vonage with network having uplink packet loss of 25%. — **Figure 2:** FPS comparison for Agora and Vonage with network having uplink packet loss of 25%.

25% Downlink Packet Loss

With 25% downlink packet loss the results are the same, with Agora maintaining 23 FPS while the performance of Vonage is impacted significantly, dropping all the way down to 1 FPS.

Figure 3: FPS comparison for Agora and Vonage with network having downlink packet loss of 25%. — **Figure 3:** FPS comparison for Agora and Vonage with network having downlink packet loss of 25%.

Uplink 600ms Jitter

A significant variation in latency can occur when sending traffic over an IP network. When simulating conditions of a highly congested network having an uplink jitter of 600ms, Agora outperforms Vonage, holding steady at 23 FPS. In comparison, Vonage drops to 6 FPS with choppy video.

Downlink 600ms Jitter

When downlink jitter of 600ms was added, Agora held steady at 22 FPS with Vonage dropping to 7 FPS with choppy video.

Frame Rate Recovery, 500 Kbps Limit

This test starts without any bandwidth limitations, and then after 60 seconds the network is throttled to 500 Kbps in the upstream direction, and then in the downstream direction for a total of 30 seconds. Under these challenging conditions, Agora performed better than Vonage while the limitation was applied, maintaining consistent 24 FPS while Vonage dropped below 10 FPS at the start of the limitation and remained variable until the limitation was removed. After the limitation was removed, both Vonage and Agora recovered immediately to their initial FPS, with Agora at 24 and Vonage at 20.

Figure 6: Frame rate recovery in limited500K test case. — **Figure 6:** Frame rate recovery in limited500K test case.

Frame Rate Recovery, 500 Kbps Limit and 25% Packet Loss

This test starts without any bandwidth limitations or simulated packet loss. After 60 seconds, the network is throttled to 500 Kbps in the upstream direction along with 25% simulated packet loss. These conditions are then applied in the downstream direction for a total of 30 seconds. While the limitation was applied, Agora performed better than Vonage, fluctuating between 19 FPS and 24 FPS while the limitation was applied while Vonage dropped down to 0 FPS and only recovered to 5 FPS for a short time, resulting in frozen video. Once the limitation was removed, Agora had already recovered to an average of 24 FPS while Vonage recovered quickly, but only back to around 20 FPS.

Figure 7: Frame rate recovery in limited500K25%PL test case. — **Figure 7:** Frame rate recovery in limited500K25%PL test case.

Agora has several ways to combat network disruptions and costly packet loss. Most notably, these include leveraging our SD-RTN™ network as an overlay to the public internet, which routes traffic around impairments on the internet using AI algorithms and optimally routes real-time traffic for the best performance.

Agora also implements technologies to smooth out the effects of packet loss to optimize the end-user experience.

CPU Utilization

Agora consumed less CPU resources than Vonage across scenarios, with Agora consuming an average of 41.5% vs. Vonage’s average of 57.2% across all scenarios.

Processor: CPU Hexa-core (2×2.5 GHz Vortex + 4×1.6 GHz Tempest)

Test Case ID	Agora	Vonage
Unlimited	40.9%	64.2%
Limited500K	41.2%	65.3%
UL25%PL	41.5%	58.1%
DL25%PL	42.2%	45.6%
Limited500K25%PL	41.1%	65.4%
UL600msJitter	41.4%	49.6%
DL600msJitter	42.2%	52.5%

RAM Utilization

Agora consumed more RAM than Vonage in all scenarios, between 115 MB and 117MB compared to of Vonage’s average of 59 MB, respectively.

Test Case ID	Agora	Vonage
Unlimited	115 MB	60 MB
Limited500K	117 MB	61 MB
UL25%PL	116 MB	58 MB
DL25%PL	116 MB	59 MB
Limited500K25%PL	117 MB	60 MB
UL600msJitter	116 MB	57 MB
DL600msJitter	117 MB	58 MB

Conclusion

While Agora showed a 4 FPS advantage over Vonage under normal network conditions, Agora’s ability to maintain consistent performance and frame rate in common scenarios involving network limitations, jitter, and packet loss conditions dramatically outperformed Vonage. In many situations where limitations were applied, Vonage’s FPS dropped to a level that resulted in choppy, stuttering, and at times, completely frozen video. In the same scenarios, Agora’s video remained smooth, without noticeable changes in performance. Vonage consumed less RAM than Agora, while Agora consumed less CPU resources than Vonage across scenarios.

When it comes to providing real-time communication tools, the ability to deliver a high-quality and consistent end user experience under constantly changing network conditions is critical.

At Agora, our customers—ranging from some of the largest social media companies in the world to bootstrapped startups—continue to leverage our SDKs to easily integrate video, voice, and interactive live streaming experiences into any app. We empower developers to deliver best-in-class real-time experiences, with global delivery and scale.