Skip to content
Testing Agora vs. Twilio for 1:1 Web Video Calls featured

Testing Agora vs. Twilio for 1:1 Web Video Calls

By Author: Team Agora In Business, Developer

This blog summarizes the testing results of Agora versus Twilio for 1:1 Web video calls. The testing covers the most important considerations when assessing video SDKs, from analyzing performance under constrained network conditions to maximizing throughput given available bandwidth while keeping central processing unit (CPU) and random-access memory (RAM) usage as low as possible.

Test Configuration and Scenarios

Before we take a deep dive into the results, let’s review the test setup, configuration and scenarios covered in the testing.

Test Setup

In this test we used laptops with the following configuration for the sender and receiver:

  • Sender: MacBook Pro
    Processor 2.6 GHz 6-Core Intel Core i7
    Memory 32 GB 2400 MHz DDR4
  • Receiver: HP ProBook 630 G8 Notebook PC
    Processor Intel Core i7-1185G7 @ 3.00 GHz

All video feeds are set to 640×360 @ 24FPS with the laptops connected to a stable Wi-Fi network.

For Agora, we tested using the Agora App Builder 2.0 and for Twilio we tested leveraging the most recent version of the Twilio Video React App, to ensure that Twilio best practices were properly implemented for comparison.

Test Case Scenarios

Test CaseDescription
UnlimitedNormal network conditions
Limited500KThe test starts with no bandwidth limitations, then a low bandwidth 500 Kbps is applied, first on the send side, then applied on the receiver side.
UL25%PLUplink network with impairment of 25% packet loss
DL25%PLDownlink network with impairment of 25% packet loss
Limited500K25%PLThe test starts with no bandwidth limitations, then a low bandwidth 500 Kbps with 25% packet loss is applied, first on the send side, then applied on the receiver side.
UL600msJitterUplink network with impairment of 600ms jitter
DL600msJitter Downlink network with impairment of 600ms jitter

Results

Here’s a quick overview of the results for reach testing scenario before we dive into the full details for each one:

  • Under normal network conditions: The frame rate (FPS) for each was steady at an average of 23 FPS for Agora and 20 FPS for Twilio.
  • Packet loss and jitter: Agora consistently achieved a better frame rate than Twilio under numerous conditions, including when the network had uplink/downlink packet loss of 25%, as well as in cases where the uplink/downlink experienced jitter of 600ms.
  • Limited bandwidth: When throttling to 500 Kbps on the send side, and then on the receive side, Agora and Twilio performed in a similar way, though Twilio’s FPS fluctuated more than Agora after the bandwidth limitation was removed.
  • 25% packet loss with 500 Kbps network bandwidth limitation: Agora’s average frame rate while the impairments were applied was approximately 13 FPS while Twilio’s average was approximately 7 FPS.  Once the impairments were removed, Agora rapidly recovered to an average of 22 FPS while Twilio struggled for approximately 50 seconds to recover.
  • CPU and RAM: Twilio consumed slightly less CPU resources at an average of 4.28% vs. Agora at 4.58%. For RAM utilization. Twilio consumed more RAM than Agora at an average of 250 MB vs. Agora at 146 MB.

Frame Rate Testing

In these tests, we again focused primarily on the effect of packet loss and limiting network conditions on the send and receive frame rates.

Normal Network Conditions

As you can see in the graph below, with the network functioning under normal conditions, the frame rate was steady at 23 FPS for Agora and 20 FPS for Twilio. The overall experience for end users was similar under normal network conditions.

Figure 1: Normal Network Conditions
Figure 1: Normal Network Conditions

When introducing 25% uplink packet loss, Agora remains steady at 18 FPS while Twilio begins lagging at 12 FPS.

Figure 2: 25% Uplink Packet Loss
Figure 2: 25% Uplink Packet Loss

With 25% downlink packet loss, Agora continues to maintain 17 FPS while the performance of Twilio is impacted significantly, down to 7 FPS.

Figure 3: 25% Downlink Packet Loss
Figure 3: 25% Downlink Packet Loss

A significant variation in latency can occur when sending traffic over an IP network. When simulating conditions of a highly congested network having an uplink jitter of 600ms, Agora outperforms Twilio, holding steady at 9 FPS. In comparison Twilio drops to 4 FPS.

Figure 4: Uplink 600ms Jitter
Figure 4: Uplink 600ms Jitter

When downlink jitter of 600ms was added, Agora held steady at 13 FPS with Twilio dropping to 5 FPS. 

Figure 5: Downlink 600ms Jitter
Figure 5: Downlink 600ms Jitter

Frame Rate Recovery, 500 Kbps Limit

This test starts without any bandwidth limitations, and then after 60 seconds the network is throttled to 500 Kbps in the upstream direction, and then in the downstream direction for a total of 30 seconds. Under these challenging conditions, Agora performed slightly better than Twilio while the limitation was applied. When the bandwidth limitation was removed, Agora maintained a more stable frame rate near 23 FPS, while Twilio’s FPS experienced greater fluctuations.

Figure 6: Frame Rate Recovery, 500 Kbps Limit
Figure 6: Frame Rate Recovery, 500 Kbps Limit

Frame Rate Recovery, 500 Kbps Limit and 25% Packet Loss

This test starts without any bandwidth limitations or simulated packet loss. After 60 seconds, the network is throttled to 500 Kbps in the upstream direction along with 25% simulated packet loss. These conditions are then applied in the downstream direction for a total of 30 seconds. Agora’s average frame rate while the impairments were applied was approximately 13 FPS while Twilio’s average was approximately 7 FPS.  Once the impairments were removed, Agora rapidly recovered to an average of 22 FPS while Twilio struggled for approximately 50 seconds to recover.

Figure 7: Frame Rate Recovery, 500 Kbps Limit and 25% Packet Loss
Figure 7: Frame Rate Recovery, 500 Kbps Limit and 25% Packet Loss

Agora has several ways to combat network disruptions and costly packet loss. Most notably, these include leveraging our SD-RTN™ network as an overlay to the public internet, which routes traffic around impairments on the internet using AI algorithms and optimally shapes real-time traffic for the best performance.

Agora also implements technologies to smooth out the effects of packet loss to optimize the end-user experience.

CPU Utilization

Processor: Intel Core i7-1185G7 @ 3.00 GHz

Test Case ID AgoraTwilio
Unlimited4.58%4.32%
Limited500K4.53%4.28%
UL25%PL4.61%4.18%
DL25%PL4.59%4.21%
Limited500K25%PL4.61%4.35%
UL600msJitter4.56%4.27%
DL600msJitter4.59%4.36%

Twilio consumed slightly less CPU resources at an average of 4.28% vs. Agora at 4.58%.

RAM Utilization

Test Case ID AgoraTwilio
Unlimited145 MB261 MB
Limited500K151 MB255 MB
UL25%PL145 MB260 MB
DL25%PL142 MB245 MB
Limited500K25%PL157 MB235 MB
UL600msJitter142 MB246 MB
DL600msJitter141 MB248 MB

Twilio consumed more RAM than Agora at an average of 250 MB vs. Agora at 146 MB.

Conclusion

 
This blog summarizes the test results covering various real-world scenarios and conditions experienced when implementing Video SDKs for point-to-point Web use cases.  When it comes to real-time communication, the ability to deliver a high-quality and consistent end user experience is critical.  Our testing shows that Agora consistently outperforms Twilio in common RTC scenarios involving jitter and packet loss. Agora’s advantage in these scenarios comes from our global SD-RTN (Software Defined Real Time Network)™ which uses AI to intelligently route traffic for optimal RTC performance.

At Agora, our customers—ranging from some of the largest social media companies in the world to bootstrapped startups—continue to leverage our SDKs to easily integrate video, voice, and interactive live streaming experiences into any app. We empower developers to deliver best-in-class real-time experiences, with global delivery and scale.

Sign up for free today to start building or contact us for a free consultation.