Going Mobile: Agora vs. Zoom Testing for Multi-Party Mobile Video Calls
Part 3: Multi-Party Mobile Video Calls—A Video SDK Comparison
Summary: In Part 1 and Part 2 of this multi-part blog, Agora examined a recent study from Zoom that suggested the Zoom Video SDK outperformed Agora’s Video SDK on several fronts. However, in our follow-up analysis, Agora’s detailed testing demonstrated the performance advantages of its SDK in 1:1 PC calling scenarios and Multi-Host Video Calls (Desktop PCs).
In our third installment, we are looking at how the Agora and Zoom video SDKs performed side-by-side during Multi-Party Mobile Video Calls with up to 32 participants, under varying real-world network conditions.
First, a quick refresher on the genesis of this debate: In a series of benchmark tests commissioned by Zoom late last year, the video telephony company dipped its toes further into the video SDK space, comparing its offering to several high-profile competitors, including Agora.
Obviously, after nearly a decade of providing best-in-class video SDKs to thousands of customers and hundreds of use cases, the Agora team was more than confident to take up the challenge.
As you can read here and here, rounds 1 and 2 went to Agora. You’ll have to read on to see if Agora makes it three in a row!
Test Configuration and Scenarios
Before diving into the results let’s examine the conditions and configurations deployed in testing.
Table 1 – Test Setup
| Product | Agora Video Call V6.2.26.2533, SDK 3.5.2 Zoom for Android Version: 5.12.8 |
| Scenario | Video 32v32 Sender: Galaxy S20 Processor Octa-core, 1x 2.84GHz Kryo 585 + 3x 2.42GHz Kryo 585 + 4x 1.80GHz Kryo 585 12GB RAM Receiver: Galaxy S10+ Processor Octa-core, 2.80 GHz, Kryo 485 8GB RAM |
| Product Configuration | Agora 480×640 @ 30fps Zoom 480×640 @ 30fps |
| Network Configuration | Sender: WIFI, ATTWrt5gC Receiver: WIFI, ATTWrt5g |
Table 2 – Test Case Scenarios
| ID | Description |
|---|---|
| Unlimited | Normal network conditions |
| Limited500K | The test starts with no bandwidth limitation, then a low bandwidth 500Kbps is applied, first on the send side, then applied on the receiver side. |
| UL25%PL | Uplink network has 25% packet loss |
| DL25%PL | Downlink network has 25% packet loss |
| Limited500K25%PL | The test starts with no bandwidth limitation, then a low bandwidth 500Kbps with 25% packet loss is applied, first on the send side, then applied on the receiver side. |
| UL600msJitter | Uplink network has 600ms jitter |
| DL600msJitter | Downlink network has 600ms jitter |
And a note about pagination.
Although this is a 32-video host test on mobile, there is little to no user experience value for rendering 32 video tiles simultaneously on a mobile screen. That is why, for a realistic user experience, pagination was implemented to solve this problem.
For Agora, we tested using our Agora Video Call V6.2.26.2533 and SDK version 3.5.2. For Zoom, we tested the recent Android Version: 5.12.8, to ensure Zoom’s best practices were properly implemented for comparison.
* All results were observed and measured on a Galaxy S10+ receiving video from a 32-video conference meeting.
Drum Roll Please…Not All Video SDKs are Created Equal
Whether under optimal network conditions or during significant packet loss, Agora outperformed Zoom from top to bottom. If you’re a CEO in the money-making business, having a reliable video SDK that can function while others struggle to deliver content in adverse conditions can mean the difference between success for failure.
Consider these results:
- Under normal network conditions, both Agora and Zoom performed well. However, Agora outperformed Zoom by 2 Frames Per Second (FPS). With fluent video and audio, and an average received frame rate of 26 FPS, Agora outpaced Zoom’s 24 FPS.
- With 25% uplink packet loss Agora held steady maintaining an FPS advantage of 2 at 25 FPS. Zoom dropped to a frame rate of 23.
- With 25% downlink packet loss the Agora advantage stretched, outperforming Zoom with a 5 FPS advantage against Zoom on the received frame rates. Agora 25 FPS, Zoom 20 FPS.
- Creating conditions with the network having uplink 600ms jitter, furthering the divide between the Video SDKs’ performance. Agora holds steady at 23 FPS, while Zoom drops precipitously to 3 FPS, a result that is generating a very choppy video and participants possibly dropping the call.
- Nearly identical results occur when conditions were created for downlink 600ms jitter, with Agora again at 23 FPS, while Zoom bottoms out at 4 FPS. This resulted in a very choppy Zoom video.
- When the test starts with no bandwidth limitation and then a low bandwidth of 500Kbps is applied, first on the send side, then applied on the receiver side, Agora performs much better than Zoom. Zoom dropped to 0 FPS for over 10 seconds total with frozen video, then struggled for 20 seconds to recover to 25 FPS.
A Mobile Summary
Frame Rate Testing
In this test, we focused primarily on the effect of packet loss and limiting network conditions on the send and receive frame rates.
As we discussed in previous blogs, Agora has several ways to combat network disrupts that often result in costly packet loss by leveraging our SD-RTN™ network as an overlay to the public internet, implementing technologies that optimize performance over last-mile connections, and device optimizations.
By routing traffic around impairments on the internet using AI algorithms and optimally shapes real-time traffic for the optimal performance, Agora directly combats possible consumer churn.
Normal Network
Under normal network conditions, we see the average frame rate received during Multi-Party Mobile Video Calls Agora maintains fluent video and audio with an average received frame rate of 26 FPS. Zoom lags behind at 24 FPS.
25% Uplink Packet Loss
With 25% uplink packet loss, Agora held steady, maintaining an FPS advantage of 2 to 3 at 25 FPS. Zoom dropped to a frame rate of 23.
25% Downlink Packet Loss
When testing introduced downlink packet loss of 25%, Agora continued to put distance between it and the competition. While Zoom dropped to 19 FPS, Agora held steady with a healthy 25 FPS. This difference is significant enough for users to notice issues with the call.
Uplink 600ms jitter
This variation in latency occurs when sending data over a network. 600ms is considered high and will likely cause noticeable performance issues. If the jitter is consistently at 600ms or higher, it is likely due to an issue with the network connection, such as congestion or interference.
Once the network in our testing reached uplink 600ms jitter, Zoom’s FPS dropped precipitously to a low of 4 FPS.
Here Zoom’s frame rate dropped to 4 FPS generating choppy video. Once again, Agora held steady at 23 FPS. The testing again demonstrated the divide between the Video SDKs’ performance.
Downlink 600ms jitter
Much like the uplink jitter, the downlink 600ms jitter yielded similar results. Nearly identical results occur when conditions were created for downlink 600ms jitter, with Agora again at 23 FPS, while Zoom bottoms out at 4 to 5 FPS.
Again, when the frame rate is in the mid-20s, the users will experience no lag or disruption. A frame rate below 5 will begin to freeze the video.
Frame Rate Recovery
When the test starts with no bandwidth limitation and then a low bandwidth of 500Kbps is applied, first on the send side, then applied on the receiver side, Agora performs much better than Zoom. Zoom dropped to 0 FPS for over 10 seconds total, freezing the video, and then struggled for 20 seconds to recover to 25 FPS.
Agora dips to 6 FPS but quickly rebounds to the mid-20s in under 2 seconds.
Frame-Rate Recovery in Limited 500K, 25% Packet Loss
As you can see in the graph below, the initial frame rate drops to zero, rebounds briefly to 5 then struggles for 30 seconds.
CPU Utilization
For mobile multi-party video calls, we monitored CPU Octa-core, 2.80 GHz, Kryo 485.
Table 3 – CPU Comparison
| Test Case ID | Agora | Zoom |
|---|---|---|
| Unlimited | 10.61% | 8.15% |
| Limited500K | 10.52% | 7.27% |
| UL25%PL | 10.28% | 7.75% |
| DL25%PL | 10.75% | 7.88% |
| Limited500K25%PL | 10.46% | 7.71% |
| UL600msJitter | 10.88% | 6.57% |
| DL600msJitter | 10.92% | 7.01% |
For this use case, we would not expect a user experience impact for either Zoom or Agora, even with multiple applications running. Agora averaged 10.63%, while Zoom averaged 7.47%.
RAM Utilization
Agora consumed less RAM than Zoom, with an average of 254MB vs. Zoom’s average of 293MB.
Table 4 – RAM Comparison
| Test Case ID | Agora | Zoom |
|---|---|---|
| Unlimited | 251MB | 298MB |
| Limited500K | 262MB | 295MB |
| UL25%PL | 254MB | 300MB |
| DL25%PL | 255MB | 298MB |
| Limited1M25%PL | 258MB | 274MB |
| UL600msJitter | 242MB | 294MB |
| DL600msJitter | 256MB | 298MB |
Conclusion
Once again, Agora squared off with Zoom under varying real-world network conditions and came away the clear-cut winner. In every test, Agora’s video SDKs outperformed Zoom’s.
For anyone integrating video capabilities into an application, Agora provides tools that allow developers to quickly and easily add video features to any app, more reliably and efficiently than others, including Zoom. As the tests have shown, under any condition, especially in more challenging environments, Agora delivers fluent, natural audio and video, while Zoom freezes or delivers choppy video.
Next Up
Tune in for our next testing updates, covering a breakdown of how Agora compares to Zoom when leveraging Video SDKs for web.
In the meantime, sign up for free to test out Agora’s performance for yourself.
