API Performance Testing: Tutorial & Examples

APIs play a crucial role in modern systems by enabling seamless interactions among various software components and facilitating access to third-party services. For this reason, ensuring API performance and reliability under a variety of conditions is tantamount to ensuring the performance of the system as a whole.

A key element in this process is API performance testing, which evaluates an API’s ability to function efficiently under a reasonable or expected workload. In today’s interconnected world, where user experience dictates success, robust API performance testing is not just desirable—it is essential.

This article delves into API performance testing, highlighting its significance in the development process and providing best practices to help you better integrate API performance testing into software projects.

Summary of API performance testing best practices

Adherence to best practices helps optimize the API performance testing process. The table below summarizes the practices covered in this article.

Define realistic test cases based on user behavior

Developers must meticulously analyze and replicate user interactions to define test cases that accurately reflect real-world user behavior. This involves mimicking diverse data input variations—such as different data sizes and formats—and evaluating transactional processes like user authentication, data retrieval, and data updates.

Crafting realistic test scenarios is crucial for accurate performance evaluation, as is ensuring seamless data sharing between APIs, including tokens, IDs, and other essential metadata. Be sure to develop user personas to represent diverse user groups, simulate varied user interactions and behaviors, and enhance test accuracy and reliability.

For example, the code block below simulates a user logging into a system, obtaining a unique JTW, and making a subsequent API call to a chat API.

// faker for generating synthetic data
import { faker } from '@faker-js/faker';

class LotsOfChatMxTestSpec {
  npmDeps = {
    '@faker-js/faker': '7.6.0',
  };
  async vuInit(ctx) {
    // Set the base url of Multiple's built-in axios instance
    // The built-in axios instance automatically captures metrics
    ctx.axios.defaults.baseURL = process.env.API_BASE_URL;

    // Log in as a user and get a JWT
    const res = await ctx.axios.post('login', {
      email: `user+${ctx.info.vuId}@multiple.dev`,
      password: 'testpassword#1234',
    });
    const jwt = res.data.token;

    // Debug Run log to check we are getting the JWT correctly
    console.debug('JWT Token: ', jwt);

    // Set the authorization header
    ctx.axios.defaults.headers.common['Authorization'] = `Bearer ${jwt}`;
  }

  async vuLoop(ctx) {
    // Send a POST request to the chat endpoint with a random message
    await ctx.axios.post('chat', {
      // Generate synthetic data with faker
      message: faker.lorem.paragraph(),
    });

    // Send a GET request to the chat endpoint
    await ctx.axios.get('chat');
  }
}

As you can see, the test script uses parameterized input for the user’s email address (based on ctx.info.vuId). This allows each virtual user to log in with a unique email address, obtain a unique JWT, and make unique, traceable API calls associated with those credentials. Using the Multiple platform, the test script can be executed by a configurable number of virtual users to simulate real users interacting with the API simultaneously.

Leverage virtual users and concurrency

Virtual users (VUs) play an important role in simulating real-world user interactions and load scenarios in API performance testing. Also known as simulated users or virtual clients, VUs are generated by performance testing tools to mimic the behavior of actual users interacting with an API. Leveraging virtual users enables developers to assess how an API behaves under various conditions, including different user loads, concurrency levels, and usage patterns.

In the context of API performance, concurrency refers to the ability of the API to handle multiple requests simultaneously. It is a critical factor in performance testing because it measures how well an API can scale to meet the demands of its users. During testing, developers can employ virtual users to evaluate an API’s performance under different levels of concurrency, ranging from a few simultaneous users to thousands or even more.

Within testing platforms like Multiple, virtual users are autonomous, concurrent entities that execute test scripts in tandem with one another. They possess individual memory capabilities, enabling them to store and recall information—such as authorization tokens, database connections, or other values—as required during testing.

Another essential advantage of virtual users is that, given a scalable testing infrastructure, thousands of virtual users can be launched to get realistically diverse test data with minimal code. This allows for greater modularity and reusability in test scripts.

Utilize a variety of load patterns

When evaluating API resilience, employing diverse load patterns provides performance insights under different load scenarios. Here are some load patterns to consider:

• Spike testing: Simulate abrupt increases in user loads, such as those experienced during promotional events or product launches, to evaluate the API’s ability to handle peak traffic efficiently.
• Soak testing: Assess the API’s performance under sustained high loads to ensure consistent responsiveness and reliability over extended periods.
• Off-peak load analysis: Evaluate API performance during non-peak hours when CPU usage and resource consumption are expected to be lower. This analysis helps ensure that resources scale down appropriately while still performing within acceptable tolerances.
• Error recovery simulation: Include scenarios in performance testing that deliberately introduce errors, evaluating how the API handles them. This step ensures that the API can recover gracefully from unexpected issues.

By incorporating these varied load patterns into your testing strategy, you gain a more comprehensive understanding of your API’s performance and resilience under a more comprehensive range of conditions.

{{banner-1="/design/banners"}}

Test all the protocols in your stack

Each communication protocol integrated into a product, such as HTTP/HTTPS, GraphQL, WebSockets, or RPC, plays a vital role in ensuring seamless communication and functionality among system components. Even if each system component performs efficiently in isolation, any failure or inefficiency at the protocol layer can cascade and cause issues throughout the system.

Some common problems that could arise at the protocol level include:

• Performance bottlenecks
• Scalability issues
• Security vulnerabilities
• Inefficient resource utilization (CPU, memory, network bandwidth, etc.)

Choosing a performance testing tool that supports a wide variety of communication protocols enables developers to conduct more comprehensive protocol testing and streamlines the protocol testing process considerably. Here is an example of a GraphQL API performance test using Multiple:

import { gql, GraphQLClient, request } from 'graphql-request';

class GraphQLTestSpec {
  npmDeps = {
    graphql: '16.6.0',
    'graphql-request': '6.0.0',
  };

  async vuInit(ctx) {
    // Create and pass down a GraphQL client
    const graphQLClient = new GraphQLClient(https://countries.trevorblades.com/graphql);
    return { graphQLClient };
  }

  async vuLoop(ctx) {
    // Get the GraphQL client from vuInit()
    const { graphQLClient } = ctx.vuInitData;

    // Define the query
    const query = gql`
      {
        country(code: "US") {
          name
          native
          capital
          emoji
          currency
          languages {
            code
            name
          }
        }
      }
    `;

    const startTime = Date.now();
    // Execute the query and capture the time taken
    const response = await graphQLClient.request(query);
    ctx.metric('US Country Query', Date.now() - startTime, 'ms');

    // Log the output during a Debug Run to verify the result
    console.debug(JSON.stringify(response, null, 2));
  }
}

The script above uses the graphql-request NPM package to query a GraphQL API. The testing platform’s support for NPM packages allows developers to write test scripts for different protocols in a way similar to how they create application code, which lowers the overhead of creating test scripts.

Choose an appropriate test environment

Selecting and provisioning a suitable environment for performance testing is a crucial step in ensuring the accuracy and reliability of test results and the efficiency of the testing process. In ideal scenarios, performance testing environments should be isolated from environments used for functional testing and should mirror the production environment as closely as possible. This approach ensures that performance testing does not interfere with functional tests and provides data that realistically represents how the application behaves in production.

Here are some specific practices to consider:

• Isolate the performance testing environment: Use a separate environment for performance testing to prevent interruptions from other applications, tests, or system components. Mimic the production environment as much as is feasible given time and resource constraints. If your organization conducts performance testing within shared environments or tests production directly, ensure that performance tests are scheduled during off hours when they will not interfere with the production application or other development activities.
• Manage test data: Replicate a subset of production data to the testing server or generate mock data with the same format and schema as production to simulate real-world scenarios without impacting the production environment. For a more in-depth discussion on using synthetic test data, see our free guide.
• Isolate network requests: To prevent side effects, replace actual network calls with “dummy calls” using HTTP interceptors like Polly.js or Nock. For complex systems, consider mocking entire APIs using tools like Mock Service Worker.
• Beware of potential risks: Keep in mind the risks associated with performance testing, such as overloading servers, hitting cloud usage quota limits, creating mock data in production environments, or inadvertently copying sensitive or regulation-protected data from production into testing environments.

Measure key API performance metrics

Performance metrics help developers optimize the API for better scalability and efficiency. Here are some important metrics to help measure API performance.

Example: Capturing CPU utilization metrics from AWS CloudWatch

The code example below shows how to use the Multiple platform to capture custom metrics using the built-in ctx.metric function. In this case, we capture the CPU utilization metric from Amazon CloudWatch:

/* The CloudWatch client and instanceId were previously defined */

// Specify the parameters for your query
const cwCmd = new GetMetricStatisticsCommand({
  Namespace: 'AWS/EC2',
  MetricName: 'CPUUtilization',
  Dimensions: [
    {
      Name: 'InstanceId',
      Value: instanceId,
    },
  ],
  StartTime: new Date(Date.now() - 5 * 60 * 1000), // 5 minutes
  EndTime: new Date(),
  Period: 1, // seconds
  Statistics: ['Maximum'],
});

// Get the CPU Utilization from CloudWatch
const cwResult = await cloudwatch.getMetricStatistics(cwCmd).promise();
if (cwResult.Datapoints && data.Datapoints.length > 0) {
  const cpuUtilization = data.Datapoints[0].Maximum;
  // Capture the metric
  ctx.metric('CPU Max', cpuUtilization, '%');
}

In this script, we capture the maximum amount of CPU utilized from an AWS EC2 instance per second over the course of five minutes using the Amazon SDK for JavaScript. After a test run, the CPU utilization metric will be present on the results page and can be used to determine whether any adjustments to cloud resources or configurations are needed to meet the performance requirements of the application under test.

Store and compare test results over time

Although a single test run does provide valuable performance data to development teams, the greatest insights from performance testing are gained over time. Each individual test run provides a snapshot of system performance at a given moment, but those insights become outdated once engineers upgrade packages or make changes to the application infrastructure or environment configuration.

The exact frequency of test runs will vary between applications depending on several factors, such as the scale and scope of the test being run, the organization’s dedicated QA resources, the importance of application performance for business goals, and the presence and expectations of SLAs. Determining which tests to run and how often therefore involves weighing tradeoffs between the costs of running performance tests and the potential business consequences that could come from running tests too infrequently.

Storing historical test results in a central, easily accessible location supports this process by providing relevant stakeholders with a historical record of API performance. This historical data serves as a valuable reference point, both for determining whether performance trends are heading in a positive direction and for evaluating whether current testing practices adequately achieve testing goals.

{{banner-2="/design/banners"}}

Conclusion

API performance testing is crucial for evaluating the functionality, reliability, and user satisfaction of systems in today’s interconnected digital environment. Developers can effectively assess, optimize, and maintain API performance under different conditions by following detailed best practices, methodologies, and strategies. Continuous monitoring, specialized tools, and proactive testing strategies are critical components of a robust API performance testing approach.

As technology evolves, prioritizing API performance remains critical for successful software development, deployment, and business continuity. Embracing the comprehensive best practices and methodologies presented in this article allows developers to navigate the complexities of API performance testing, ensuring smooth user experiences, high system reliability, and overall business success.