Testing Agora vs. Vonage for 1:1 Web Video Calls
This blog summarizes the testing results of Agora versus Vonage for 1:1 Web video calls. The testing covers the most important considerations when assessing video SDKs, from analyzing performance under constrained network conditions to maximizing throughput given available bandwidth while keeping central processing unit (CPU) and random-access memory (RAM) usage as low as possible.
Test Configuration and Scenarios
Before we take a deep dive into the results, let’s review the test setup, configuration and scenarios covered in the testing.
Test Setup
In this test we used laptops with the following configuration for the sender and receiver:
- Sender: MacBook Pro
Processor 2.6 GHz 6-Core Intel Core i7
Memory 32 GB 2400 MHz DDR4
- Receiver: HP ProBook 630 G8 Notebook PC
Processor Intel Core i7-1185G7 @ 3.00 GHz
All video feeds are set to 640×480 @ 30 FPS with the laptops connected to a stable Wi-Fi network.
For Agora, we tested using the Agora App Builder 2.0 and for Vonage we tested leveraging the most recent version of the Vonage WebRTC Demo, to ensure that Vonage best practices were properly implemented for comparison.
Test Case Scenarios
| Test Case | Description |
|---|---|
| Unlimited | Normal network conditions, with no bandwidth constraints, packet loss or jitter. |
| Limited500K | The test starts with no bandwidth limitations, then a low bandwidth of 500 Kbps is applied, first on the send side, then applied on the receiver side. |
| UL25%PL | Uplink network with impairment of 25% packet loss |
| DL25%PL | Downlink network with impairment of 25% packet loss |
| Limited500K25%PL | The test starts with no bandwidth limitations, then a low bandwidth 500 Kbps with 25% packet loss is applied, first on the send side, then applied on the receiver side. |
| UL600msJitter | Uplink network with impairment of 600ms jitter |
| DL600msJitter | Downlink network with impairment of 600ms jitter |
The results are in – an overview:
- Under normal network conditions: The frame rate (FPS) for each was steady at 28 FPS for Agora and 27 FPS for Vonage.
- Packet loss and jitter: Agora consistently achieved a better frame rate than Vonage under numerous packet loss and jitter conditions, including when the network had uplink/downlink packet loss of 25%, as well as in cases where the uplink/downlink experienced jitter of 600ms.
- Limited bandwidth: When throttling to 500 Kbps on the send side, and then on the receive side, Agora and Vonage performed in an analogous way.
- 25% packet loss with 500 Kbps network bandwidth limitation: Vonage experienced choppy or frozen video while Agora recovered quickly to fluid video performance. After the limitation was removed, Agora recovered to an average of 28 FPS within ~2 seconds while Vonage struggled for ~50 seconds to get back to 26 FPS on average.
- CPU and RAM: Agora and Vonage had comparable results for CPU and RAM consumption.
Frame Rate Testing
In these tests, we again focused primarily on the effect of packet loss and limiting network conditions on the send and receive frame rates.
Normal Network Conditions
As you can see in the graph below, with the network functioning under normal conditions, the frame rate was steady at 28 FPS for Agora and 27 FPS for Vonage. The overall experience for end users was similar under normal network conditions.
25% Uplink Packet Loss
When introducing 25% uplink packet loss, Agora remains steady at 15 FPS while Vonage begins lagging at 12 FPS.
25% Downlink Packet Loss
With 25% downlink packet loss, Agora continues to maintain 16 FPS while the performance of Vonage is impacted significantly, down to 1 FPS, resulting in choppy video.
Uplink 600ms Jitter
A significant variation in latency can occur when sending traffic over an IP network. When simulating conditions of a highly congested network having an uplink jitter of 600ms, Agora outperforms Vonage, holding steady at 9 FPS. In comparison Vonage drops to 6 FPS.
Downlink 600ms Jitter
When downlink jitter of 600ms was added, Agora held steady at 13 FPS with Vonage dropping to 5 FPS.
Frame Rate Recovery, 500 Kbps Limit
This test starts without bandwidth limitations, and after 60 seconds the network is throttled to 500 Kbps in the upstream direction, and then downstream for 30 seconds. Under these challenging conditions, Agora performed better than Vonage, rapidly recovering to an average of 27 FPS once the limitations were removed. It took Vonage 10 seconds to recover.
Frame Rate Recovery, 500 Kbps Limit and 25% Packet Loss
This test starts without any bandwidth limitations or simulated packet loss. After 60 seconds, the network is throttled to 500 Kbps in the upstream direction along with 25% simulated packet loss. These conditions are then applied downstream for 30 seconds. While the limitations were applied, Vonage users experienced consistent frozen video for up to 10 seconds at a time.
Once the bandwidth limitation and simulated packet loss conditions were removed, Agora rapidly recovered within about 2 seconds to 28 FPS on average while Vonage struggled for about 50 seconds before recovering to 26 FPS on average.
Agora has several ways to combat network disruptions and costly packet loss. Most notably, these include leveraging our SD-RTN (Software Defined Real Time Network)™ network as an overlay to the public internet, which routes traffic around impairments on the internet using AI algorithms and optimally shapes real-time traffic for the best performance.
Agora also implements technologies to smooth out the effects of packet loss to optimize the end-user experience.
CPU Utilization
Processor: Intel Core i7-1185G7 @ 3.00 GHz
| Test Case ID | Agora | Vonage |
|---|---|---|
| Unlimited | 4.61% | 4.82% |
| Limited500K | 4.59% | 4.78% |
| UL25%PL | 4.72% | 4.98% |
| DL25%PL | 4.65% | 4.81% |
| Limited500K25%PL | 4.63% | 4.73% |
| UL600msJitter | 4.58% | 4.87% |
| DL600msJitter | 4.63% | 4.92% |
Agora consumed slightly less CPU resources vs. Vonage in all scenarios, with an average of 4.63% utilization vs. Vonage at an average of 4.84%.
RAM Utilization
| Test Case ID | Agora | Vonage |
|---|---|---|
| Unlimited | 149 MB | 129 MB |
| Limited500K | 156 MB | 134 MB |
| UL25%PL | 149 MB | 131 MB |
| DL25%PL | 146 MB | 125 MB |
| Limited500K25%PL | 163 MB | 131 MB |
| UL600msJitter | 145 MB | 123 MB |
| DL600msJitter | 152 MB | 125 MB |
Vonage consumed slightly less RAM than Agora, with an average of 128 MB compared to Agora’s 151 MB.
Conclusion
This blog summarizes the test results covering various real-world scenarios and conditions experienced when implementing Video SDKs for point-to-point Web use cases. When it comes to real-time communication, the ability to deliver a high-quality and consistent end user experience is critical. Our testing shows that Agora consistently outperforms Vonage in common RTC scenarios involving jitter and packet loss. Agora’s advantage in these scenarios comes from our global SD-RTN (Software Defined Real Time Network)™ which uses AI to intelligently route traffic for optimal RTC performance.
At Agora, our customers—ranging from some of the largest social media companies in the world to bootstrapped startups—continue to leverage our SDKs to easily integrate video, voice, and interactive live streaming experiences into any app. We empower developers to deliver best-in-class real-time experiences, with global delivery and scale.
Sign up for free today to start building or contact us for a free consultation.
