Back to Blog

Testing Agora vs. Twilio for Multi-Party Web Video Calls

Testing Agora vs. Twilio for Multi-Party Web Video Calls

This blog summarizes the testing results of Agora versus Twilio for multi-party web video calls with 32 participants. The testing covers the most important considerations when assessing video SDKs, from analyzing performance under constrained network conditions to maximizing throughput given available bandwidth while keeping central processing unit (CPU) and random-access memory (RAM) usage as low as possible.

For more comparisons, check out the results of our Agora vs. Twilio testing for 1:1 web video calls.

Test Configuration and Scenarios

Before we take a deep dive into the results, let’s review the test setup, configuration and scenarios covered in the testing of Agora and Twilio for multi-party web video calls.

 AgoraTwilio
Network ConnectivityWi-FiWi-Fi
Participants in Sessions3232
Versions TestedAgora Web Demo (SDK 4.17.0)Twilio Video React App
Receive Test PC SpecificationReceiver: HP ProBook 630 G8 Notebook PC
Processor: Intel Core i7-1185G7 @ 3.00 GHz
Memory: 16 GB 2400 MHz DDR4
Receiver: HP ProBook 630 G8 Notebook PC Processor: Intel Core i7-1185G7 @ 3.00 GHz
Memory: 16 GB 2400 MHz DDR4
Video Configuration for Receive Test PCReceive 1280x720 @ 24 FPSReceive 1280x720 @ 24 FPS
Participant DevicesMac or Windows PCsMac or Windows PCs

For Agora, we tested using Agora Web Demo (SDK 4.17.0). For Twilio, we tested leveraging the most recent version of the Twilio WebRTC Demo , to ensure that Twilio best practices were properly implemented for comparison.

Test Case Scenarios

Test CaseDescription
UnlimitedNormal network conditions
Limited1MThe test starts with no bandwidth limitations, then a low bandwidth 1 Mbps is applied, first on the send side, then applied on the receiver side.
UL25%PLUplink network with impairment of 25% packet loss
DL25%PLDownlink network with impairment of 25% packet loss
Limited1M25%PLThe test starts with no bandwidth limitations, then a low bandwidth 1 Mbps with 25% packet loss is applied, first on the send side, then applied on the receiver side.
UL600msJitterUplink network with impairment of 600ms jitter
DL600msJitter Downlink network with impairment of 600ms jitter

Results Overview

Here’s a quick overview of the results for each testing scenario before we dive into the full details for each one:

  • Under normal network conditions: The frame rate (FPS) for each was steady at 23 FPS for Agora and 22 FPS for Twilio.
  • Packet loss and jitter: Agora consistently achieved a better frame rate than Twilio under numerous conditions, including when the network had uplink/downlink packet loss of 25%, as well as in cases where the uplink/downlink experienced jitter of 600ms.
  • Limited bandwidth: When throttling to 1 Mbps on the send side, and then on the receive side, Agora performed better than Twilio while the limitation was applied, with an average of about 21 FPS vs. Twilio at an average of about 13 FPS. After the limitation was removed Agora had a more stable frame rate with an average of 23 FPS, while Twilio fluctuated more, at times dipping to 13 FPS.
  • 25% packet loss with 1 Mbps network bandwidth limitation:  While the limitation was applied Agora performed better than Twilio, recovering to an average frame rate of approximately 13 FPS while Twilio remained near an average of 5 FPS with choppy video. Once the limitation was removed, Agora recovered to an average of 23 FPS while Twilio recovered to an average near 13 FPS.
  • CPU and RAM: Agora consumed slightly less CPU and RAM resources than Twilio.

Frame Rate Testing

In these tests, we again focused primarily on the effect of packet loss and limiting network conditions on the send and receive frame rates.

Normal Network Conditions

As you can see in the graph below, with the network functioning under normal conditions, the frame rate was steady at 23 FPS for Agora and 22 FPS for Twilio. The overall experience for end users was similar under normal network conditions.

Figure 1: Normal Network Conditions
Figure 1: Normal Network Conditions

25% Uplink Packet Loss

When introducing 25% uplink packet loss, Agora remains steady at 15 FPS while Twilio begins lagging at 12 FPS.

Figure 2: 25% Uplink Packet Loss
Figure 2: 25% Uplink Packet Loss

25% Downlink Packet Loss

With 25% downlink packet loss, Agora delivers 14 FPS while the performance of Twilio is impacted significantly, down to 6 FPS.

Figure 3: 25% Downlink Packet Loss
Figure 3: 25% Downlink Packet Loss

Uplink 600ms Jitter

A significant variation in latency can occur when sending traffic over an IP network. When simulating conditions of a highly congested network having an uplink jitter of 600ms, Agora outperforms Twilio, holding steady at 8 FPS. In comparison, Twilio drops to 3 FPS with choppy video.

Figure 4: Uplink 600ms Jitter
Figure 4: Uplink 600ms Jitter

Downlink 600ms Jitter

When downlink jitter of 600ms was added, Agora held steady at 8 FPS with Twilio dropping to 3 FPS with choppy video.

Figure 5: Downlink 600ms Jitter
Figure 5: Downlink 600ms Jitter

Frame Rate Recovery, 1 Mbps Limit

This test starts without any bandwidth limitations, and then after 60 seconds the network is throttled to 1 Mbps in the upstream direction, and then in the downstream direction for a total of 30 seconds. Under these challenging conditions, Agora performed better than Twilio while the limitation was applied, with an average of about 21 FPS vs. Twilio’s average of about 13 FPS. After the limitation was removed, Agora had a more stable frame rate with an average of 23 FPS, while Twilio fluctuated more, at times dipping to 13 FPS.

Figure 6: Frame Rate Recovery, 1 Mbps Limit
Figure 6: Frame Rate Recovery, 1 Mbps Limit

This test starts without any bandwidth limitations or simulated packet loss. After 60 seconds, the network is throttled to 1 Mbps in the upstream direction along with 25% simulated packet loss. These conditions are then applied in the downstream direction for a total of 30 seconds. While the limitation was applied Agora performed better than Twilio, recovering to an average frame rate of approximately 13 FPS while Twilio remained near an average of 5 FPS with choppy video. Once the limitation was removed, Agora recovered to an average of 23 FPS while Twilio recovered to an average near 13 FPS.

Frame Rate Recovery, 1 Mbps Limit and 25% Packet Loss

Figure 7: Frame Rate Recovery, 1 Mbps Limit and 25% Packet Loss
Figure 7: Frame Rate Recovery, 1 Mbps Limit and 25% Packet Loss

Agora has several ways to combat network disruptions and costly packet loss. Most notably, these include leveraging our SD-RTN™ network as an overlay to the public internet, which routes traffic around impairments on the internet using AI algorithms and optimally shapes real-time traffic for the best performance.

Agora also implements technologies to smooth out the effects of packet loss to optimize the end-user experience.

CPU Utilization

Processor: Intel Core i7-1185G7 @ 3.00 GHz

Test Case ID AgoraTwilio
Unlimited6.25%7.62%
Limited1M6.18%7.76%
UL25%PL6.21%7.52%
DL25%PL6.19%7.28%
Limited1M25%PL6.35%7.28%
UL600msJitter6.48%7.78%
DL600msJitter6.27%7.15%

Agora consumed slightly less CPU resources vs. Twilio in all scenarios, with an average of 6.28% utilization vs. Twilio at an average of 7.48%.

RAM Utilization

Test Case ID AgoraTwilio
Unlimited261 MB282 MB
Limited1M258 MB281 MB
UL25%PL264 MB287 MB
DL25%PL262 MB278 MB
Limited1M25%PL245 MB292 MB
UL600msJitter259 MB283 MB
DL600msJitter257 MB286 MB

Agora consumed slightly less RAM than Twilio in all scenarios, at an average of 258 MB vs. 284 MB, respectively.

Conclusion

This blog summarizes the test results comparing Agora and Twilio across various scenarios and conditions experienced when implementing video SDKs for multi-party web use cases.  When it comes to providing real-time communication tools, the ability to deliver a high-quality and consistent end user experience is critical.

At Agora, our customers—ranging from some of the largest social media companies in the world to bootstrapped startups—continue to leverage our SDKs to easily integrate video, voice, and interactive live streaming experiences into any app. We empower developers to deliver best-in-class real-time experiences, with global delivery and scale.

Sign up for free today to start building or contact us for a free consultation.

RTE Telehealth 2023
Join us for RTE Telehealth - a virtual webinar where we’ll explore how AI and AR/VR technologies are shaping the future of healthcare delivery.

Try Agora for Free

Sign up and start building! You don’t pay until you scale.
Try for Free
Get Started with Agora thumbnail