Load tests as code

An introduction to k6

Nicole Mendoza van der Hoeven

🇵🇭 🇺🇸 🇦🇺 🇳🇱

🇪🇸 🇫🇷 🇩🇪 🇸🇪 🇷🇺 🇷🇸 🇵🇱 🇮🇹 🇹🇭 🇨🇳 🇬🇷 🇭🇺 🇸🇮 🇻🇺 🇻🇳

Just a little bit about me - I’m a Developer Advocate at k6.io, which you’ll hear more about. I’ve been in testing, and especially performance testing, for about ten years now. What I love most about testing is the amount of choices we have. It’s a good thing, too, because I really like choices. I love travelling (when the world isn’t on fire).

And I love languages! You’re probably wondering what this has to do with load testing. Well, I think people think about languages in the same way that they think about testing tools– that there’s no point learning them if they don’t become experts.

Here are some languages I have learned at one point to different degrees. Some of them I can still hold a conversation in, but honestly, most of them I’ve forgotten. And I’m okay with that. I learned each of them for a specific purpose, and if they served that purpose, I think that’s time well spent.

I think we should think about tools this way, too. I think we should always be aware of what else is out there and try out different tools even if we don’t become experts in them, because that’s the only way we’ll be sure that the tool we’re using is the best for the job.

Here are the testing tools that I’ve used. These aren’t even all primarily load testing tools, but they are tools I’ve used for load testing to varying degrees. Some of them I’ve only downloaded and run some dummy tests on, and many of them I’ve used in actual load testing projects.

I wanted to go through all of this because I’m going to be telling you all about ONE tool out of these, k6– but I want to be clear that I’m not saying it’s the best tool for the job. I don’t believe there is a “best” tool, and anyone who tells you there is one, is probably just looking to sell it to you.

By the way, k6 is open source and completely free. Full disclosure: there is a SaaS version, but that just lets you run k6 tests on OUR cloud. If you don’t need to use our infrastructure, you don’t need to pay! The paid service and the free tool both use the same version of k6 and we’re pretty committed to feature parity between the two because we believe strongly that open source is key to innovation.

I think the better approach is to look at all these apps as tools in your Tester’s Toolbox. If you know what each tool’s good at, then you can choose the right one for the job.

So this presentation is about what k6 is good at. But first, let’s look at those tools again, but arranged by age.

I’ve arranged this by order of when the tool was first released. The left column is the oldest, and it has tools that are 16 years old or more. After that, each column to the right is in 5-year increments.

Then, in each column, I’ve also sorted tools with the oldest on top, and youngest at the bottom.

So LoadRunner, for example, was released 28 years ago, in 1993. JMeter was released in 1998 - it’s 23 years old. By the way, I should say that I’m not including load testing platforms in this, such as BlazeMeter or Tricentis Flood, because those platforms are still just using these load testing tools under the hood.

If we look at the history of the tool landscape, we might notice something interesting. The traditional way of writing test scripts involves some sort of GUI-driven tool.

The tools I marked with a red box are those that are predominantly GUI-driven. This doesn’t mean they can’t take code - LoadRunner scripts use C, for example. But it’s still C within VuGen. So these tools are ones where there’s an interface that needs to be loaded, before you can write a load testing script.

What we see here is that load testing tools as a whole are moving away from GUI-driven testing. All the other tools, the newer tools, that aren’t in red, including k6, are using load tests as code. But why? Here are a few reasons.

Scripting

It can take longer to set up and get started.

It’s difficult to collaborate.

It requires very specific knowledge that isn’t transferrable.

Learning a new UI means acquiring very specific knowledge that isn’t transferrable. A GUI can seem easy, but that’s actually misleading. Every UI you learn is for that tool only; that’s why adherence to design standards is so important - it’s a way to make some of the UI transferrable. Even then, learning a UI is mostly not a transferrable skill. Learning the LoadRunner UI won’t help you with JMeter, because they have completely different menus. They even have different terminology! A virtual user in LoadRunner is a thread in JMeter.

As testers, we know that tools come and go. When we have a choice (and sometimes we don’t), it makes sense to me to choose to learn something that we can use even if (when) the tool is deprecated. If you learn a programming language that is useful in other contexts, then it won’t be wasted when one of the tools you’re using it for goes away.

(NAVIGATE RIGHT TO EXECUTION)

This JMeter error might be very familiar to you if you’ve ever been on a team with multiple people working on the same script. This means that you’ve tried to load a script that uses plugins you don’t have installed. When you work on JMeter, everyone needs to have the same version of Java, JMeter, AND the same plugins. Inevitably there’s someone who forgets to tell everyone else that they’re using a new plugin.

You might be thinking, but wait, why are you picking on JMeter? JMeter scripts are saved in a .jmx file in XML format. Well, let’s have a look at that. I scripted a very simple test in JMeter. It makes one GET request, names that transaction “01_Home”, checks that a certain string is found in the response to make sure we’ve hit the right page, and then it does a variable think time. This is just a one-user, one iteration script.

(NAVIGATE DOWN TO JMETER SCRIPT)


<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0" jmeter="5.3">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
      <stringProp name="TestPlan.comments"></stringProp>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp name="TestPlan.user_define_classpath"></stringProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <stringProp name="LoopController.loops">1</stringProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">1</stringProp>
        <stringProp name="ThreadGroup.ramp_time">1</stringProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
        <boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
      </ThreadGroup>
      <hashTree>
        <TransactionController guiclass="TransactionControllerGui" testclass="TransactionController" testname="01_Home" enabled="true">
          <boolProp name="TransactionController.includeTimers">false</boolProp>
          <boolProp name="TransactionController.parent">true</boolProp>
        </TransactionController>
        <hashTree>
          <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
              <collectionProp name="Arguments.arguments"/>
            </elementProp>
            <stringProp name="HTTPSampler.domain">test.k6.io</stringProp>
            <stringProp name="HTTPSampler.port"></stringProp>
            <stringProp name="HTTPSampler.protocol">https</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path"></stringProp>
            <stringProp name="HTTPSampler.method">GET</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
            <stringProp name="HTTPSampler.connect_timeout"></stringProp>
            <stringProp name="HTTPSampler.response_timeout"></stringProp>
          </HTTPSamplerProxy>
          <hashTree>
            <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
              <collectionProp name="Asserion.test_strings">
                <stringProp name="-745788246">Collection of simple web-pages suitable for load testing</stringProp>
              </collectionProp>
              <stringProp name="Assertion.custom_message"></stringProp>
              <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
              <boolProp name="Assertion.assume_success">false</boolProp>
              <intProp name="Assertion.test_type">16</intProp>
            </ResponseAssertion>
            <hashTree/>
          </hashTree>
        </hashTree>
        <UniformRandomTimer guiclass="UniformRandomTimerGui" testclass="UniformRandomTimer" testname="Uniform Random Timer" enabled="true">
          <stringProp name="ConstantTimer.delay">1000</stringProp>
          <stringProp name="RandomTimer.range">4000</stringProp>
        </UniformRandomTimer>
        <hashTree/>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

Here’s what the resulting script looks like in a .jmx file. It’s 73 lines of code. If your teammate checks this into the repo and you need to modify it to add your changes, it is a pain to do that in sheer code! For example, let’s look at how we’d change the assertion. It looks like you only need to change this one string, but actually this entire code block refers to the one assertion element. And it’s not too easy to understand. What is this strong prop name? What is this assertion test type “16”? If it wasn’t you that wrote this script, it’s a little daunting to understand and modify it.

In reality, you’re not going to try to script in XML. You’re most likely going to download this script, save it locally, and open it up in JMeter. That’s why I consider JMeter mainly a GUI-driven tool.

(NAVIGATE DOWN TO K6 VERSION)


import http from 'k6/http';
import { sleep, check } from 'k6';

export default function () {
    let res = http.get('https://test.k6.io', {tags: { name: '01_Home' }});
    check(res, {
      'is status 200': (r) => r.status === 200,
      'text verification': (r) => r.body.includes("Collection of simple web-pages suitable for load testing")
    });
    sleep(Math.random() * 5);
}

Execution

It’s slower.

It costs more to run tests.

It matters because the most expensive part of running a load test is not the tool itself, because there are open-source tools for that– it’s the load generation. Regardless of the tool you’re using, you need to pay for machine to run load from, whether that’s by provisioning your own on-premise load generators, by paying a cloud service like AWS, or by paying for a full tool package where they have the cost of the machines built in.

How resource-hungry a tool is, how much CPU or memory it utilizes, affects how many users you can run from it. Let’s say you need to run a test with 10,000 users. On one end of the tool landscape, you might have tool A that runs 5 users per load generator. This might sound extreme, but this is probably what you’ll get with something like Tosca. You’d need 2,000 load generators for that amount of load. On the other end of the spectrum, you can run all 10,000 with a single load generator with something like k6.

2,000x is an extreme case, but even a 10x increase in cost isn’t something we can take lightly. Is that GUI worth that?

(NAVIGATE RIGHT TO CI)

https://k6.io/blog/comparing-best-open-source-load-testing-tools

Here’s a graph comparing the maximum throughput (requests per second) and memory usage of many open-source load testing tools. This IS on the k6 blog, but the source code (all the test scripts) and the details of it are on the blog for you to try yourself, so you can confirm these results.

The blue bar shows how many requests per second each tool can generate on a single load generator, and the orange bar is the memory usage for the same test. So a big blue bar and a small orange bar is good. You’ll see that in this group, the only GUI-driven tool, JMeter, has the highest memory usage and a middling maximum rps.

So why does memory utilization on the load generator matter? Well, it’s always a good idea to monitor the resources on your load generator to make sure that it’s still healthy. Otherwise, your tool IS the bottleneck. You’re relying on something slow to measure your application, so if you get bad results, you won’t know if it’s your application or your tool - or both!

By the way, I want to point out that the next most memory-hungry tool here is Gatling, which is also a Java-based tool even though it’s code-driven.

(NAVIGATE DOWN TO JMETER HEAP MEMORY ERROR)

ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError: 
Java heap space.

https://jmeter.apache.org/usermanual/get-started.html

Continuous testing

It’s hard to hook up to to CI/CD pipelines.

It encourages costly, one-off load tests at the expense of continuous ones.

Not being able to automate a performance test in a pipeline means favoring huge, several thousand-user tests that are manually triggered. These types of tests are expensive and require a test environment.

But that’s like saying there’s no point running a single functional test case unless you can run the entire test suite. The big load tests are really flashy, but not enough is said about the importance of smaller tests, not even necessarily LOAD tests, of, maybe tens of users instead of thousands, that run regularly throughout the day. You don’t need an environment, you don’t need special preparations, and you still get a good idea of the performance of your application if you run it consistently.

But only if you can tie your performance tests to a CI/CD pipeline.

(NAVIGATE RIGHT)

The alternative

Load tests as code

written in Go, scripted in js

browser extension recorder, Test Builder, HAR-to-k6, Postman-to-k6, OpenAPI specs

IDEs: VSCode, IntelliJ, TypeScript Type Definition

Results: Kafka, CloudWatch, Datadog, InfluxDB + Grafana, New Relic, statsd

CI/CD: Azure Pipelines, CircleCI, GitHub Actions, GitLab, Jenkins, TeamCity

k6 is written in Go, which has huge performance benefits as a programming language at a fundamental level over something like Java. Instead of scripting in Go, though, JavaScript is what you’ll actually be scripting in with k6, and we chose that because it’s nearly ubiquitous for web apps as well as some test tools you might already be familiar with.

We have recorders and converters that let you create a k6 script faster, including just being able to download a Chrome or Firefox browser extension, hit record, navigate to your website, and generate a k6 script from that.

If you want to just write a script in js from scratch, we have IDE extensions that help with that.

You can also pipe your results out to a variety of visualization and APM tools as well as CI/CD tools.

And all of this is available with the free, open-source version.

Demo

Presentation slides as code: show what this presentation looks like in Markdown. It’s an alternative to Powerpoint.
I publish the slides to my site, so it’s a very basic app. What if I wanted to load test this site?
Create a simple k6 script that hits this presentation. Save as slides-test.js within public.
Run test locally.

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
    vus: 10,
    duration: "5m",
};
export default function () {
  let response = http.get('https://slides.nicolevanderhoeven.com/2021-load-tests-as-code');
  check(response, {
      "text verification": (r) => r.body.includes("An introduction to k6"),
  });
  sleep(1);
}

Run on the cloud. k6 cloud slides-test.js
Open up the test run on k6 Cloud.
These other runs are from CI. I am using GitHub Actions.
Push repo to git and deploy.
Move to GitHub Actions and show what the other ones look like.
Show the script in GitHub and how easy that would be to change.
Now think of doing all this with Powerpoint. The same things that would make it clunky, if not impossible with Powerpoint, make this sort of flow clunky with a GUI-driven tool.

Sapir-Whorf hypothesis

Source: eaglenews.ph

Sapir-Whorf-van der Hoeven hypothesis

Phasic vs continuous?

Siloed vs collaborative?

Clunky vs streamlined?

Questions?

Site: k6.io / nicolevdh.com

Twitter: @n_vanderhoeven / @k6_io

Slides: slides.nicolevdh.com

YouTube: https://youtube.com/c/k6test

Email: nicole@k6.io