Testing Agora vs. Twilio for 1:1 Mobile Video Calls

August 9, 2023 By Team Agora In Business, Developer

This blog summarizes the testing results of Agora versus Twilio for one-on-one mobile video calls. The testing covers the most important considerations when assessing video SDKs, from analyzing performance under constrained network conditions to maximizing throughput given available bandwidth while keeping central processing unit (CPU) and random-access memory (RAM) usage as low as possible.

For more Twilio comparisons, check out:

Test Configuration and Scenarios

Before we take a deep dive into the results, let’s review the test setup, configuration and scenarios covered in the testing of Agora and Twilio for 1:1 mobile video calls.

Test Setup

	Agora	Twilio
Network Connectivity	Wi-Fi, 5G cellular (ATT)	Wi-Fi, 5G cellular (ATT)
Participants in Sessions	1	1
Versions Tested	Agora Video SDK for iOS (version 4.2)	Twilio Video SDK for iOS (version 5.2.1)
Sender	iPhone 11 CPU Hexa-core (2×2.65 GHz Lightning + 4×1.8 GHz Thunder) Memory 4GB	iPhone 11 CPU Hexa-core (2×2.65 GHz Lightning + 4×1.8 GHz Thunder) Memory 4GB
Receiver	iPhone XS CPU Hexa-core (2×2.5 GHz Vortex + 4×1.6 GHz Tempest) Memory 4GB	iPhone XS CPU Hexa-core (2×2.5 GHz Vortex + 4×1.6 GHz Tempest) Memory 4GB
Video Configuration for Receive Test Device	Agora 480×544 @ 24fps	Twilio 480×544 @ 24fps

For Agora, we tested using Agora Video SDK for iOS (version 4.2). For Twilio, we tested leveraging the most recent version of the Twilio Video SDK for iOS (version 5.2.1), to ensure that Twilio best practices were properly implemented for comparison.

Test Case Scenarios

Test Case	Description
Unlimited	Normal network conditions
Limited500K	The test starts with no bandwidth limitations, then a low bandwidth 500Kbps is applied, first on the send side, then applied on the receiver side.
UL25%PL	Uplink network with impairment of 25% packet loss
DL25%PL	Downlink network with impairment of 25% packet loss
Limited500K25%PL	The test starts with no bandwidth limitation, then a low bandwidth 500 Kbps with 25% packet loss is applied, first on the send side, then applied on the receiver side.
UL600msJitter	Uplink network with impairment of 600ms jitter
DL600msJitter	Downlink network with impairment of 600ms jitter

Results

Here’s a quick overview of the results for each testing scenario before we dive into the full details for each one:

Under normal network conditions: Agora and Twilio performed similarly, with Agora at 24 FPS and Twilio at 23 FPS.
Packet loss and jitter: Agora consistently achieved a better frame rate than Twilio under numerous packet loss and jitter conditions. With both 25% uplink packet loss and 25% downlink packet loss, Agora’s FPS was 23 in comparison with Twilio’s 4.
Limited bandwidth: When the network was limited to low bandwidth, 500Kbps on the send side first, and then on the receiver side, Agora maintained an excellent 24 FPS during with no need for recovery. Twilio fell below 10 FPS and took 30 seconds to recover.
25% packet loss with 500 Kbps network bandwidth limitation: Agora maintained an excellent 24 FPS during the limitation with no need for recovery while Twilio fell below 5 FPS and took a full 45 seconds to recover.
CPU and RAM: Agora used slightly less CPU resources than Twilio but used more RAM.

Frame Rate Testing

In these tests, we again focused primarily on the effect of packet loss and limiting network conditions on the send and receive frame rates.

Normal Network Conditions

With the network functioning under normal conditions, the frame rate was steady at 24 FPS for Agora and 23 FPS for Twilio. The overall experience for end users was similar under normal network conditions.

Figure 1: FPS comparison for Agora and Twilio under a normal network condition. — **Figure 1:** FPS comparison for Agora and Twilio under a normal network condition.

25% Uplink Packet Loss

When introducing 25% uplink packet loss, Agora remains steady at 23 FPS while Twilio lags dramatically at 4FPS.

Figure 2: FPS comparison for Agora and Twilio with network having uplink packet loss of 25%. — **Figure 2:** FPS comparison for Agora and Twilio with network having uplink packet loss of 25%.

25% Downlink Packet Loss

With 25% downlink packet loss the results are the same, with Agora maintaining 23 FPS while the performance of Twilio is impacted significantly, dropping down to 4 FPS.

Figure 3: FPS comparison for Agora and Twilio with network having downlink packet loss of 25%. — **Figure 3:** FPS comparison for Agora and Twilio with network having downlink packet loss of 25%.

Uplink 600ms Jitter

A significant variation in latency can occur when sending traffic over an IP network. When simulating conditions of a highly congested network having an uplink jitter of 600ms, Agora outperforms Twilio, holding steady at 23 FPS. In comparison Twilio drops to 4 FPS with choppy video.

Downlink 600ms Jitter

When downlink jitter of 600ms was added, Agora held steady at 22 FPS with Twilio dropping to 4 FPS with choppy video.

Frame Rate Recovery, 500 Kbps Limit

This test starts without any bandwidth limitations, and then after 60 seconds the network is throttled to 500 Kbps in the upstream direction, and then in the downstream direction for a total of 30 seconds. Under these challenging conditions, Agora performed better than Twilio while the limitation was applied, with an average of about 17 FPS before stabilizing at 24 FPS while Twilio averaged at about 10 FPS. After the limitation was removed Agora maintained 24 FPS immediately, while Twilio took a full 30 seconds to recover.

Figure 6: Frame rate recovery in limited500K test case. — **Figure 6:** Frame rate recovery in limited500K test case.

Frame Rate Recovery, 500 Kbps Limit and 25% Packet Loss

This test starts without any bandwidth limitations or simulated packet loss. After 60 seconds, the network is throttled to 500 Kbps in the upstream direction along with 25% simulated packet loss. These conditions are then applied in the downstream direction for a total of 30 seconds. While the limitation was applied, Agora performed better than Twilio, recovering within about 10 seconds to an average frame rate of 24 FPS while Twilio averaged 5 FPS with choppy video for the duration of the limitation. Once the limitation was removed, Agora had already recovered to an average of 24 FPS while Twilio took a full 45 seconds to recover to that level.

Figure 7: Frame rate recovery in limited500K25%PL test case. — **Figure 7:** Frame rate recovery in limited500K25%PL test case.

Agora has several ways to combat network disruptions and costly packet loss. Most notably, these include leveraging our SD-RTN™ network as an overlay to the public internet, which routes traffic around impairments on the internet using AI algorithms and optimally routes real-time traffic for the best performance.

Agora also implements technologies to smooth out the effects of packet loss to optimize the end-user experience.

CPU Utilization

Agora and Twilio consumed CPU resources similarly, with Agora consuming an average of 50% vs. Twilio’s average of 50.03% across all scenarios.

Processor: CPU Hexa-core (2×2.5 GHz Vortex + 4×1.6 GHz Tempest)

Test Case ID	Agora	Twilio
Unlimited	49.9%	51.8%
Limited500K	50.1%	51.9%
UL25%PL	49.1%	48.1%
DL25%PL	49.6%	48.7%
Limited500K25%PL	49.7%	52.2%
UL600msJitter	50.9%	48.6%
DL600msJitter	50.7%	48.9%

RAM Utilization

Agora consumed more RAM than Twilio in all scenarios, at an average of 132.8 MB vs. Twilio’s 83.5 MB, respectively.

Test Case ID	Agora	Twilio
Unlimited	131 MB	86 MB
Limited500K	132 MB	86 MB
UL25%PL	132 MB	82 MB
DL25%PL	133 MB	82 MB
Limited500K25%PL	135 MB	85 MB
UL600msJitter	133 MB	82 MB
DL600msJitter	134 MB	82 MB

Conclusion

This blog summarizes the test results comparing Agora and Twilio across various scenarios and conditions experienced when implementing video SDKs for one-on-one mobile use cases. Agora’s ability to maintain consistent performance and frame rate in common scenarios involving network limitations, jitter, and packet loss conditions dramatically outperformed Twilio. Twilio and Agora performed comparably under normal network conditions and regarding CPU consumption while Twilio consumed slightly less RAM than Agora.

When it comes to providing real-time communication tools, the ability to deliver a high-quality and consistent end user experience under constantly changing network conditions is critical.

At Agora, our customers—ranging from some of the largest social media companies in the world to bootstrapped startups—continue to leverage our SDKs to easily integrate video, voice, and interactive live streaming experiences into any app. We empower developers to deliver best-in-class real-time experiences, with global delivery and scale.