Zoom Out and Look at the Big Picture When Evaluating Real-time Engagement Solutions
1:1 Desktop PC Video Calls
This blog looks at the specific scenario of 1:1 Desktop PC Video Calls. For a summary of all test cases, check out this summary blog: Agora vs. Zoom: A Comprehensive Comparison of Video SDKs
When evaluating real-time engagement solutions zoom out and look at the big picture. After nearly a decade of building real-time audio and video SDKs and APIs for app developers, this is advice we share with everyone who works with Agora.
A recent study commissioned by Zoom provides a case in point.
The testing released compares the performance of the Zoom video SDK to video SDKs from Agora and three other real-time engagement PaaS providers.
In the results, Zoom claims that (a) they maintain the highest frame rate on video calls compared to all vendors and (b) they have the most consistent video quality in all tested network conditions.
There is a lot to unpack in just those two statements. However, it’s imperative to point out, to do a comprehensive performance comparison, there are some missing pieces in Zoom’s test:
- Mobile and web testing results were not included (only PCs were involved in the testing).
- Specifications not detailed, such as the version of each vendor’s SDK used in the testing and whether SDK implementation best practices for each vendor were followed.
- Real-world testing between various geographic regions. It is important for customers to ensure that the vendor they select can deliver consistent quality and performance in and between regions where they plan to offer their services.
- Coverage of resource utilization for CPU and RAM on lower end devices vs. higher end devices where minor differences typically have no impact on user experience.
While this blog focuses on 1:1 Desktop PC Video Calls, the Agora team also tested the following scenarios here:
Agora Testing
In this blog, we will zoom out and look at the bigger picture of what is important for customers to consider when evaluating real-time engagement (RTE) solutions.
Today, however, we’ll start on highlighting some key results and takeaways from recent testing performed by Agora focusing on the 1:1 calling scenarios covered in Zoom’s study.
Test Setup
In this test we’re using two Windows laptops for the sender and receiver:
- Sender: OMEN HP
Processor Intel Core i7-9750H CPU @2.60GHz - Receiver: MSI
Processor Intel Core i7-10750H CPU @2.60GHz
All video feeds are set to 1280×720 @ 30 FPS (720p30) with the laptops connected to a stable Wi-Fi network.
For Agora, we tested using our Native SDK version 4.1.0 and for Zoom we tested leveraging the recent Zoom Application version 5.12.3, to ensure that all of Zoom’s best practices were properly implemented for comparison.
In Zoom’s study it is apparent that they were using an older version of the Agora Video SDK as they stated: “…Agora was unable to host a 32-grid gallery view”. Agora supports up to 128 hosts!
Frame Rate Test
In this test we focus primarily on the effect of packet loss and limiting network conditions on the received frame rate.
Agora minimizes the instances of packet loss and other poor network conditions by leveraging our SD-RTN network as an overlay to the public internet, implementing technologies which optimize performance over last mile connections, and device optimizations. Our SD-RTN routes traffic around impairments on the internet using AI algorithms and optimally shapes real-time traffic for the best performance. Agora also implements technologies to smooth out the effects of packet loss to optimize the end user experience.
Normal Network
Under normal network conditions, we see the frame rate averaged 29 Frames Per Second (FPS) for Zoom and Agora, which is an expected value as frame rate will seldom maintain 30 FPS.
25% Uplink Packet Loss
All frame rates remain above 26 FPS, but Agora is holding a bit above Zoom.
25% Downlink Packet Loss
Agora remains at the same level of approximately 28 FPS; however, Zoom is showing to be slightly weaker still, a touch above 25 FPS.
Frame Rate Recovery with 1Mbps Limit
Here the network is throttled to 1Mbps in the upstream direction and then in the downstream direction for a total of 30 seconds. This would be similar to someone using a fairly poor 3G network, or very busy 4G.
When the limit is applied, all providers suffer an immediate frame rate drop. It’s amazing to see how well Agora and Zoom both improve the frame rate given this throttle; however, it’s clear that Agora handles the recovery remarkably better.
Frame Rate Recovery, 1Mbps Limit and 25% Packet Loss
A brutal combination, not something we’d wish on anyone trying to have a video call, but let’s see how we fare against Zoom:
Well, that’s about as dramatic as you might expect. It’s clear that the recovery ability from Agora’s network in this test is hard to dispute. After approximately 5 seconds of the throttle being applied Agora rapidly adapts to this situation and gets back to above 25 FPS, and then of course maintains that level after the throttle is lifted. Zoom, however, appears to be struggling at 0 FPS for over 15 seconds and then at less than 5 FPS on average while the limitation is still applied.
With these conditions in an active call, you can expect to see choppy and frozen video – ultimately delivering a poor user experience.
Bitrate Test
Bitrate is also a key factor when communicating with others via a video call. To deliver the best possible experience and quality for end users, maintaining the highest possible throughput of video for the given network conditions is important.
Normal Network
Under normal network conditions, the sending bitrate for Agora and Zoom was the same, however, Agora’s receive bitrate remained higher than Zoom.
25% Uplink Packet Loss
With 25% uplink Packet Loss Agora’s send and receive bitrate remained higher than Zoom.
25% Downlink Packet Loss
With 25% downlink Packet Loss Agora’s send and receive bitrate remained higher than Zoom.
Throughout the bitrate focused testing, Agora was able to adapt to the changing network conditions, maximize throughput, and maintain a quality video experience more consistently than Zoom.
Overall, Agora maintained a higher bitrate through changing network conditions, providing users with a better-quality experience overall. As demonstrated, Agora’s bitrate adapts based on the network conditions to provide the best experience.
Developer Tooling
There are several features available to developers using the Agora SDK to help them handle bad network conditions. Features include a simple priority switch, so that the SDK will prioritize video resolution, video frame rate, or audio depending on the use-case.
Conclusion
While the study commissioned by Zoom covered some of the aspects of what customers should consider when evaluating real-time engagement solutions, as we mentioned, it did not tell the complete story.
In this blog we covered a comparison of Agora and Zoom for 1:1 calling scenarios between PCs, and Agora came out ahead overall.
Next Up
For a summary of all testing scenarios, check out this blog:
To see the detailed results for each test case, check out blogs covering the other scenarios here:
In the meantime, sign up for free to test out Agora’s performance for yourself.
