Part 4: Multi-Party Web Video Calls
Summary: In Part 1, Part 2, and Part 3 of this blog series, Agora reviewed results from a recent Zoom study that suggested its Video SDK outperformed Agora’s Video SDK on several fronts and then detailed the results of our own comprehensive testing. Agora topped Zoom’s Video SDK throughout our testing in numerous vital scenarios.
This testing series highlights important considerations when assessing Video SDKs, from analyzing performance under constrained network conditions to maximizing throughput given available bandwidth and keeping central processing unit (CPU) and random-access memory (RAM) usage low, regardless of device type.
Each blog in the series focused on testing in predefined real-world scenarios applicable to Agora and Zoom.
A quick look back: in our testing and analysis, Agora demonstrated the performance advantages of its Video SDK in 1:1 PC calling scenarios, multi-host video calls (Desktop PCs), and
In each of those three scenarios, Agora’s Video SDK outperformed Zoom’s under varying real-world conditions.
This fourth and final installment of the series focuses on Agora and Zoom Video SDKs, examining how each performed during multi-party web video calls with 32 participants.
Read on to see the results!
Test Configuration and Scenarios
Before we take a deep dive into the results, let’s review the test setup, configuration and scenarios covered in the testing.
Table 1 – Test Setup
|Product||Agora: https://videocall.agora.io/ based on Web SDK version 4.17.0|
Sender: MacBook Pro
Processor: 2.6 GHz 6-Core Intel Core i7
Memory: 32 GB 2400 MHz DDR4
Receiver: HP ProBook 630 G8 Notebook PC
Processor: Intel Core i7-1185G7 @ 3.00 GHz
|Product Configuration||Agora: 1280×720 @ 24fps|
Zoom: 1280×720 @ 24fps
|Network Configuration||Sender: WIFI, ATTWrt5g|
Receiver: WIFI, ATTWrt5g
Table 2 – Test Case Scenarios
|Limited1M||The test starts with no bandwidth limitation, then a low bandwidth 1 Mbps is applied, first on the send side, then applied on the receiver side.|
|UL25%PL||Uplink network has 25% packet loss|
|DL25%PL||Downlink network has 25% packet loss|
|Limited1M25%PL||The test starts with no bandwidth limitation, then a low bandwidth 1 Mbps with 25% packet loss is applied, first on the send side, then applied on the receiver side.|
|UL600msJitter||Uplink network has 600ms jitter|
|DL600msJitter||Downlink network has 600ms jitter|
For Agora, we tested using our Web SDK version: 4.17.0. For Zoom, we tested using https://pwa.zoom.us/wc/ [Version: 220.127.116.110 (0322) to ensure Zoom’s best practices were properly implemented for comparison.
After nearly a decade of providing best-in-class video SDKs to thousands of customers and hundreds of use cases, the Agora team was not surprised to come out on top in every test case scenario against a relative newcomer to the Video SDK game.
The results are in – an overview:
- Under normal network conditions: The frame rate (FPS) for each was steady, 23 FPS for Agora and 21 FPS for Zoom.
- Packet loss and jitter: Agora consistently achieved a better frame rate than Zoom under numerous conditions, including when the network had uplink/downlink packet loss of 25%, as well as in cases where the uplink/downlink experienced jitter of 600ms.
- Limited bandwidth: When throttling to 1Mbps on the send side and then on the receive side, Agora performed significantly better than Zoom, rapidly recovering to 23 FPS once the throttle was removed. It took Zoom 30 seconds to recover to 13 FPS.
- 25% packet loss with 1Mbps network bandwidth limitation: Agora rapidly recovered to a frame rate near 23 FPS, while Zoom struggled to recover to a much lower frame rate near 15 FPS. Zoom had a frozen video for 18 seconds while the limitations were present.
- CPU and RAM: Agora consistently consumed fewer CPU resources vs. Zoom and had lower RAM usage than Zoom.
The Agora Video SDK test results were robust even in poor network scenarios. A reliable Video SDK that can function while others struggle to deliver a high-quality user experience in adverse conditions can mean the difference between keeping customers or losing existing business.
Frame Rate Testing
In these tests, we again focused primarily on the effect of packet loss and limiting network conditions on the send and receive frame rates.
Normal Network Conditions
As you can see in the graph below, with the network functioning under normal conditions, the frame rate is steady at 23 FPS for Agora and 21 FPS for Zoom, for calls with 32 participants.
25% Uplink Packet Loss
When introducing 25% uplink packet loss, Agora remains steady at 21 FPS while Zoom begins lagging at 17 FPS.
25% Downlink Packet Loss
Much the same with 25% downlink packet loss, Agora is an even 20 FPS while Zoom shows 15 FPS.
Uplink 600ms Jitter
A significant variation in latency can occur when sending traffic over an IP network. When simulating conditions of a highly congested network having uplink jitter of 600ms, Agora outperforms Zoom, holding steady at 18 FPS. In comparison, Zoom drops to 7 FPS, with a noticeable difference in perceived video quality by end users.
Downlink 600ms Jitter
Nearly identical results occurred when conditions were simulated for downlink jitter of 600ms, with Agora holding at 12 FPS and Zoom bottoming out at 3 FPS. Under these conditions, participants in the Zoom call experienced choppy video.
Frame Rate Recovery, 1Mbps Limit
This test starts without any bandwidth limitations. After 60 seconds, the network is throttled to 1Mbps in the upstream direction and then in the downstream direction for a total of 30 seconds. Under these challenging conditions, Agora performs significantly better than Zoom, rapidly recovering to 23 FPS once the limitations are removed. It takes Zoom 30 seconds to recover to 13 FPS.
Frame Rate Recovery, 1Mbps Limit and 25% Packet Loss
This test starts without any bandwidth limitations or simulated packet loss. After 60 seconds, the network is throttled to 1Mbps in the upstream direction along with 25% simulated packet loss. These conditions are then applied in the downstream direction for a total of 30 seconds.
Once the bandwidth limitation and simulated packet loss conditions are removed, Agora rapidly recovers to a frame rate near 23 FPS, while Zoom struggles to recover to a much lower frame rate near 15 FPS. End users on the Zoom call experience frozen video for approximately 18 seconds while the limitations are present.
As we discussed in previous blogs, Agora has several ways to combat network disruptions and costly packet loss. Most notably, these include leveraging our SD-RTN™ network as an overlay to the public internet, implementing technologies that optimize performance over last-mile connections, and device optimization.
Our SD-RTN™ routes traffic around impairments on the internet using AI algorithms and optimally shapes real-time traffic for the best performance.
Agora also implements technologies to smooth out the effects of packet loss to optimize the end-user experience.
Table 3 – CPU Comparison
Processor: Intel Core i7-1185G7 @ 3.00 GHz
|Test Case ID||Agora||Zoom|
Agora consumed fewer CPU resources vs. Zoom in all scenarios, with an average of 7.48% utilization vs. Zoom at an average of 9.05%.
Table 4 – RAM Comparison
|Test Case ID||Agora||Zoom|
|Unlimited||225 KB||606 KB|
|Limited1M||229 KB||689 KB|
|UL25%PL||239 KB||617 KB|
|DL25%PL||255 KB||627 KB|
|Limited1M25%PL||258 KB||708 KB|
|UL600msJitter||242 KB||639 KB|
|DL600msJitter||256 KB||652 KB|
Throughout testing, Agora consumed less RAM than Zoom in every scenario, with an average of 243 KB to Zoom’s 648 KB.
Each of these four comprehensive blogs comparing Agora and Zoom’s video SDKs has taken you inside varying scenarios and conditions experienced when implementing. When it comes to providing real-time communication tools, delivering a high-quality and consistent end-user experience is critical.
- The first test compared Agora and Zoom for 1:1 calling scenarios between PCs. Detailed testing demonstrated the performance advantages of Agora’s SDK in 1:1 PC calling in every scenario.
- In our second test, we stacked up the two video SDKs against one another in multi-host video calls on PCs with 8, 16, and 32 participants under varying real-world network conditions, Agora outperformed Zoom in all five scenarios.
- In our third test, a side-by-side during Multi-Party Mobile Video Calls with up to 32 participants, Agora again beat Zoom in all five key categories.
- And lastly here, with multi-party Web video calls, the results were much the same.
At Agora, our customers—ranging from some of the largest social media companies in the world to bootstrapped startups—continue to leverage our SDKs to easily integrate video, voice, and live interactive streaming experiences into any app. We empower developers to deliver best-in-class real-time in-app experiences, with global delivery and scale.
We can help you too. Talk to us for a free consultation.