This blog summarizes the testing results of Agora versus Twilio for one-on-one mobile video calls. The testing covers the most important considerations when assessing video SDKs, from analyzing performance under constrained network conditions to maximizing throughput given available bandwidth while keeping central processing unit (CPU) and random-access memory (RAM) usage as low as possible.
For more Twilio comparisons, check out:
- Testing Agora vs. Twilio for 1:1 web video calls
- Testing Agora vs. Twilio for multi-party web video calls
Test Configuration and Scenarios
Before we take a deep dive into the results, let’s review the test setup, configuration and scenarios covered in the testing of Agora and Twilio for 1:1 mobile video calls.
Test Setup
For Agora, we tested using Agora Video SDK for iOS (version 4.2). For Twilio, we tested leveraging the most recent version of the Twilio Video SDK for iOS (version 5.2.1), to ensure that Twilio best practices were properly implemented for comparison.
Test Case Scenarios
Here’s a quick overview of the results for each testing scenario before we dive into the full details for each one:
- Under normal network conditions: Agora and Twilio performed similarly, with Agora at 24 FPS and Twilio at 23 FPS.
- Packet loss and jitter: Agora consistently achieved a better frame rate than Twilio under numerous packet loss and jitter conditions. With both 25% uplink packet loss and 25% downlink packet loss, Agora’s FPS was 23 in comparison with Twilio’s 4.
- Limited bandwidth: When the network was limited to low bandwidth, 500Kbps on the send side first, and then on the receiver side, Agora maintained an excellent 24 FPS during with no need for recovery. Twilio fell below 10 FPS and took 30 seconds to recover.
- 25% packet loss with 500 Kbps network bandwidth limitation: Agora maintained an excellent 24 FPS during the limitation with no need for recovery while Twilio fell below 5 FPS and took a full 45 seconds to recover.
- CPU and RAM: Agora used slightly less CPU resources than Twilio but used more RAM.
Frame Rate Testing
In these tests, we again focused primarily on the effect of packet loss and limiting network conditions on the send and receive frame rates.
Normal Network Conditions
With the network functioning under normal conditions, the frame rate was steady at 24 FPS for Agora and 23 FPS for Twilio. The overall experience for end users was similar under normal network conditions.
25% Uplink Packet Loss
When introducing 25% uplink packet loss, Agora remains steady at 23 FPS while Twilio lags dramatically at 4FPS.
25% Downlink Packet Loss
With 25% downlink packet loss the results are the same, with Agora maintaining 23 FPS while the performance of Twilio is impacted significantly, dropping down to 4 FPS.
Uplink 600ms Jitter
A significant variation in latency can occur when sending traffic over an IP network. When simulating conditions of a highly congested network having an uplink jitter of 600ms, Agora outperforms Twilio, holding steady at 23 FPS. In comparison Twilio drops to 4 FPS with choppy video.
Downlink 600ms Jitter
When downlink jitter of 600ms was added, Agora held steady at 22 FPS with Twilio dropping to 4 FPS with choppy video.
Frame Rate Recovery, 500 Kbps Limit
This test starts without any bandwidth limitations, and then after 60 seconds the network is throttled to 500 Kbps in the upstream direction, and then in the downstream direction for a total of 30 seconds. Under these challenging conditions, Agora performed better than Twilio while the limitation was applied, with an average of about 17 FPS before stabilizing at 24 FPS while Twilio averaged at about 10 FPS. After the limitation was removed Agora maintained 24 FPS immediately, while Twilio took a full 30 seconds to recover.
Frame Rate Recovery, 500 Kbps Limit and 25% Packet Loss
This test starts without any bandwidth limitations or simulated packet loss. After 60 seconds, the network is throttled to 500 Kbps in the upstream direction along with 25% simulated packet loss. These conditions are then applied in the downstream direction for a total of 30 seconds. While the limitation was applied, Agora performed better than Twilio, recovering within about 10 seconds to an average frame rate of 24 FPS while Twilio averaged 5 FPS with choppy video for the duration of the limitation. Once the limitation was removed, Agora had already recovered to an average of 24 FPS while Twilio took a full 45 seconds to recover to that level.
Agora has several ways to combat network disruptions and costly packet loss. Most notably, these include leveraging our SD-RTN™ network as an overlay to the public internet, which routes traffic around impairments on the internet using AI algorithms and optimally routes real-time traffic for the best performance.
Agora also implements technologies to smooth out the effects of packet loss to optimize the end-user experience.
CPU Utilization
Agora and Twilio consumed CPU resources similarly, with Agora consuming an average of 50% vs. Twilio’s average of 50.03% across all scenarios.
RAM Utilization
Agora consumed more RAM than Twilio in all scenarios, at an average of 132.8 MB vs. Twilio’s 83.5 MB, respectively.
Conclusion
This blog summarizes the test results comparing Agora and Twilio across various scenarios and conditions experienced when implementing video SDKs for one-on-one mobile use cases. Agora’s ability to maintain consistent performance and frame rate in common scenarios involving network limitations, jitter, and packet loss conditions dramatically outperformed Twilio. Twilio and Agora performed comparably under normal network conditions and regarding CPU consumption while Twilio consumed slightly less RAM than Agora.
When it comes to providing real-time communication tools, the ability to deliver a high-quality and consistent end user experience under constantly changing network conditions is critical.
At Agora, our customers—ranging from some of the largest social media companies in the world to bootstrapped startups—continue to leverage our SDKs to easily integrate video, voice, and interactive live streaming experiences into any app. We empower developers to deliver best-in-class real-time experiences, with global delivery and scale.
Sign up for free today to start building or contact us for a free consultation.
