Testing Agora vs. Twilio for Multi-Party Web Video Calls

June 20, 2023 By Team Agora In Business, Developer

This blog summarizes the testing results of Agora versus Twilio for multi-party web video calls with 32 participants. The testing covers the most important considerations when assessing video SDKs, from analyzing performance under constrained network conditions to maximizing throughput given available bandwidth while keeping central processing unit (CPU) and random-access memory (RAM) usage as low as possible.

For more comparisons, check out the results of our Agora vs. Twilio testing for 1:1 web video calls.

Test Configuration and Scenarios

Before we take a deep dive into the results, let’s review the test setup, configuration and scenarios covered in the testing of Agora and Twilio for multi-party web video calls.

Test Setup

	Agora	Twilio
Network Connectivity	Wi-Fi	Wi-Fi
Participants in Sessions	32	32
Versions Tested	Agora Web Demo (SDK 4.17.0)	Twilio Video React App
Receive Test PC Specification	Receiver: HP ProBook 630 G8 Notebook PC Processor: Intel Core i7-1185G7 @ 3.00 GHz Memory: 16 GB 2400 MHz DDR4	Receiver: HP ProBook 630 G8 Notebook PC Processor: Intel Core i7-1185G7 @ 3.00 GHz Memory: 16 GB 2400 MHz DDR4
Video Configuration for Receive Test PC	Receive 1280×720 @ 24 FPS	Receive 1280×720 @ 24 FPS
Participant Devices	Mac or Windows PCs	Mac or Windows PCs

For Agora, we tested using Agora Web Demo (SDK 4.17.0). For Twilio, we tested leveraging the most recent version of the Twilio WebRTC Demo , to ensure that Twilio best practices were properly implemented for comparison.

Test Case Scenarios

Test Case	Description
Unlimited	Normal network conditions
Limited1M	The test starts with no bandwidth limitations, then a low bandwidth 1 Mbps is applied, first on the send side, then applied on the receiver side.
UL25%PL	Uplink network with impairment of 25% packet loss
DL25%PL	Downlink network with impairment of 25% packet loss
Limited1M25%PL	The test starts with no bandwidth limitations, then a low bandwidth 1 Mbps with 25% packet loss is applied, first on the send side, then applied on the receiver side.
UL600msJitter	Uplink network with impairment of 600ms jitter
DL600msJitter	Downlink network with impairment of 600ms jitter

Results Overview

Here’s a quick overview of the results for each testing scenario before we dive into the full details for each one:

Under normal network conditions: The frame rate (FPS) for each was steady at 23 FPS for Agora and 22 FPS for Twilio.
Packet loss and jitter: Agora consistently achieved a better frame rate than Twilio under numerous conditions, including when the network had uplink/downlink packet loss of 25%, as well as in cases where the uplink/downlink experienced jitter of 600ms.
Limited bandwidth: When throttling to 1 Mbps on the send side, and then on the receive side, Agora performed better than Twilio while the limitation was applied, with an average of about 21 FPS vs. Twilio at an average of about 13 FPS. After the limitation was removed Agora had a more stable frame rate with an average of 23 FPS, while Twilio fluctuated more, at times dipping to 13 FPS.
25% packet loss with 1 Mbps network bandwidth limitation: While the limitation was applied Agora performed better than Twilio, recovering to an average frame rate of approximately 13 FPS while Twilio remained near an average of 5 FPS with choppy video. Once the limitation was removed, Agora recovered to an average of 23 FPS while Twilio recovered to an average near 13 FPS.
CPU and RAM: Agora consumed slightly less CPU and RAM resources than Twilio.

Frame Rate Testing

In these tests, we again focused primarily on the effect of packet loss and limiting network conditions on the send and receive frame rates.

Normal Network Conditions

As you can see in the graph below, with the network functioning under normal conditions, the frame rate was steady at 23 FPS for Agora and 22 FPS for Twilio. The overall experience for end users was similar under normal network conditions.

25% Uplink Packet Loss

When introducing 25% uplink packet loss, Agora remains steady at 15 FPS while Twilio begins lagging at 12 FPS.

25% Downlink Packet Loss

With 25% downlink packet loss, Agora delivers 14 FPS while the performance of Twilio is impacted significantly, down to 6 FPS.

Uplink 600ms Jitter

A significant variation in latency can occur when sending traffic over an IP network. When simulating conditions of a highly congested network having an uplink jitter of 600ms, Agora outperforms Twilio, holding steady at 8 FPS. In comparison, Twilio drops to 3 FPS with choppy video.

Downlink 600ms Jitter

When downlink jitter of 600ms was added, Agora held steady at 8 FPS with Twilio dropping to 3 FPS with choppy video.

Frame Rate Recovery, 1 Mbps Limit

This test starts without any bandwidth limitations, and then after 60 seconds the network is throttled to 1 Mbps in the upstream direction, and then in the downstream direction for a total of 30 seconds. Under these challenging conditions, Agora performed better than Twilio while the limitation was applied, with an average of about 21 FPS vs. Twilio’s average of about 13 FPS. After the limitation was removed, Agora had a more stable frame rate with an average of 23 FPS, while Twilio fluctuated more, at times dipping to 13 FPS.

This test starts without any bandwidth limitations or simulated packet loss. After 60 seconds, the network is throttled to 1 Mbps in the upstream direction along with 25% simulated packet loss. These conditions are then applied in the downstream direction for a total of 30 seconds. While the limitation was applied Agora performed better than Twilio, recovering to an average frame rate of approximately 13 FPS while Twilio remained near an average of 5 FPS with choppy video. Once the limitation was removed, Agora recovered to an average of 23 FPS while Twilio recovered to an average near 13 FPS.

Frame Rate Recovery, 1 Mbps Limit and 25% Packet Loss

Agora has several ways to combat network disruptions and costly packet loss. Most notably, these include leveraging our SD-RTN™ network as an overlay to the public internet, which routes traffic around impairments on the internet using AI algorithms and optimally shapes real-time traffic for the best performance.

Agora also implements technologies to smooth out the effects of packet loss to optimize the end-user experience.

CPU Utilization

Processor: Intel Core i7-1185G7 @ 3.00 GHz

Test Case ID	Agora	Twilio
Unlimited	6.25%	7.62%
Limited1M	6.18%	7.76%
UL25%PL	6.21%	7.52%
DL25%PL	6.19%	7.28%
Limited1M25%PL	6.35%	7.28%
UL600msJitter	6.48%	7.78%
DL600msJitter	6.27%	7.15%

Agora consumed slightly less CPU resources vs. Twilio in all scenarios, with an average of 6.28% utilization vs. Twilio at an average of 7.48%.

RAM Utilization

Test Case ID	Agora	Twilio
Unlimited	261 MB	282 MB
Limited1M	258 MB	281 MB
UL25%PL	264 MB	287 MB
DL25%PL	262 MB	278 MB
Limited1M25%PL	245 MB	292 MB
UL600msJitter	259 MB	283 MB
DL600msJitter	257 MB	286 MB

Agora consumed slightly less RAM than Twilio in all scenarios, at an average of 258 MB vs. 284 MB, respectively.

Conclusion

This blog summarizes the test results comparing Agora and Twilio across various scenarios and conditions experienced when implementing video SDKs for multi-party web use cases. When it comes to providing real-time communication tools, the ability to deliver a high-quality and consistent end user experience is critical.

At Agora, our customers—ranging from some of the largest social media companies in the world to bootstrapped startups—continue to leverage our SDKs to easily integrate video, voice, and interactive live streaming experiences into any app. We empower developers to deliver best-in-class real-time experiences, with global delivery and scale.