Bringing Fuzz Testing to Kotlin with kotlinx.fuzz

Identifying edge cases for testing can be a challenging task, often leaving hidden bugs and vulnerabilities undetected. Fuzz testing (fuzzing) addresses this challenge by automatically generating tests with random or invalid inputs, enabling developers to uncover unexpected exceptions and failure scenarios. Thanks to its effectiveness in detecting critical software flaws, fuzzing has gained widespread use across various programming languages. However, it remains relatively underutilized in Kotlin. This is why our Programming Languages and Program Analysis Lab at JetBrains Research has dedicated its efforts to developing kotlinx.fuzz, a fuzzer for Kotlin libraries. Since Kotlin compiles to JVM bytecode, fuzz testing systems for Kotlin can be built using existing Java fuzzers. This concept is the foundation of our current prototype.

Why fuzz testing matters

Fuzz testing is a powerful technique capable of exposing undetected errors and vulnerabilities. Research has shown it to be highly effective in detecting critical software flaws. Google’s OSS-Fuzz has discovered over 40,000 bugs in open-source projects, while studies like these highlight how coverage-guided fuzzing significantly enhances bug detection in large-scale systems.

Despite its success in other ecosystems (like C/C++ and Rust), fuzzing has not seen wide adoption in Kotlin development. Our goal with kotlinx.fuzz is to bridge this gap by introducing an efficient and scalable fuzzing framework specifically designed for Kotlin projects.

In this blog post, we start with an illustration of fuzz testing with kotlinx.fuzz, followed by an explanation of how it works and a few examples of what we were able to achieve using it. Then we’ll cover our plans for the future and ways you can contribute.

Fuzz testing with kotlinx.fuzz

Let’s start with an example from the Kotlin standard library to showcase what fuzz testing your code with kotlinx.fuzz might look like and walk through all the steps it involves.

1. Analyze code 

Before you begin fuzz testing, you should decide what it is that you want to fuzz. Fuzzing every single function in your program is not very efficient because it requires a lot of time and effort when analyzing the bugs. You’re better off selecting a few target functions that will be the main entry points for the fuzzer and starting with the functions that you expect to be used most frequently. Let’s say we want to fuzz test the Duration class from the Kotlin standard library, focusing on the parseIsoStringOrNull method. It is designed to parse a time duration from a string in the ISO-8601 format. If the string represents a valid time, it returns a Duration instance. Otherwise, it returns null. Let’s write a fuzz test that checks its correctness!

2. Design a fuzz test

After you select a target that you want to test, the next step is to decide how you are going to fuzz it. You need to decide what scenarios you want to test and how to generate all of the necessary data using a fuzzer. In our example, parseIsoStringOrNull just takes a string in an ISO format. Let’s start from the simplest case and write a fuzz test that generates a random string and passes it to the target:

@KFuzzTest
fun testDuration(f: KFuzzer) {
    val isoString = f.asciiString(10)
    val duration = Duration.parseIsoStringOrNull(isoString)
    println("$isoString -> $duration")
}

As you can see, our fuzz test looks very similar to a usual unit test. However, instead of working with the fixed set of input data that the developer manually encodes, we take a Fuzzer as an argument and generate input data with its help. We can then use the fuzzer to re-run this test many times, each time slightly tweaking the value returned by the asciiString call.

3. Choose an oracle

After creating this test, you can run it for some time and manually analyze the results. However, these results take a lot of effort to read through and make sense of. To fully leverage the power of fuzzing, you need to come up with an oracle – a way to automatically check if the execution result is correct. An oracle can be as simple or as complicated as you want, however, the quality of the oracle decides what types of bugs you will be able to find. The simplest oracle is just an exception – you can test that your program does not throw any unexpected exceptions. In some cases, you can also perform more complicated checks. For example, if you are fuzzing a JSON parsing library, you can do an inverse check: toJSON(fromJSON(string)) == string. In our example, we can use the Java standard library, which has a method with exactly the same functionality (at least, according to the documentation). This is what our final fuzz test looks like:

@KFuzzTest
fun testDuration(f: KFuzzer) {
    val isoString = f.asciiString(10)
    val duration = Duration.parseIsoStringOrNull(isoString)

    val javaDuration = try {
        java.time.Duration.parse(isoString)
    } catch (_: Throwable) { null }

    assertEquals(javaDuration?.toKotlinDuration(), duration)
}

This will help us ensure that the behavior of the Kotlin and the Java function are the same, find all of the mismatches, and analyze why they occur.

4. Configure and run the tests

Before you run the test, there are some things that you may want to configure. For example, you can indicate what parts of your project the fuzzer should target, how long you want to run fuzz tests, and where and how it should store the results. Here is an example of a simple configuration:

fuzzConfig {
    instrument = listOf("kotlin.time.**")
    maxFuzzTimePerTarget = 10.minutes
    coverage {
        reportTypes = setOf(CoverageReportType.HTML, CoverageReportType.CSV)
    }
}

Check out our documentation for more details on the possible configurations.

Now you can finally start the fuzz test:

~/example » ./gradlew fuzz

5. Analyze the results

After the fuzzer finishes, you can focus on analyzing the results and see what bugs were detected. There are two main results that we recommend focusing on:

• Coverage. build/fuzz/jacoco-report will contain JaCoCo coverage reports in the configured formats. We recommend analyzing the coverage to understand if the fuzzer was able to cover all of the parts of your program that you wanted to test. 
• Bugs. build/fuzz/reproducers will contain all of the crashes found by the fuzzer. You can analyze them by running the fuzzer in regression mode (./gradlew regression). Additionally, you can debug each crash by running your fuzz test in an IDE.

In our example test, running it for just 30 minutes covers a large portion of the parseIsoStringOrNull method and gives us a few interesting results:

• Java correctly parses "Pt+0,s" into an empty duration, while Kotlin fails.
• Conversely, Kotlin correctly parses "PT+-2H" into negative 2 hours, while Java fails.

Naturally, we reported the bugs we’ve found so they could be fixed!

This example showcases that even if your library passes all unit and integration tests, fuzzing can reveal subtle parsing errors, unexpected crashes, or corner cases you never thought of, helping you ship more reliable software. Even this simple fuzz test was able to find two edge cases that highlighted issues in both the Kotlin and Java standard libraries.

What kotlinx.fuzz provides

We launched the kotlinx.fuzz project to explore the applicability of fuzzing for detecting bugs in Kotlin programs. By building our system on top of an existing Java fuzzer, Jazzer, we were able to leverage a tool already capable of fuzzing and finding bugs in JVM programs, and thus in Kotlin programs as well. However, as a standalone tool, it has some limitations that may impede its adoption for fuzz testing in Kotlin. 

Jazzer’s main limitations are related to the user experience. In our opinion, while it provides a good foundation, Jazzer is inconvenient to run, configure, and analyze results. It provides a basic command-line interface and a limited integration with JUnit. Additionally, it sometimes falls short when working with Kotlin. For example, it fails to correctly collect coverage for certain Kotlin-specific features and fails to consider nullability when working in auto-fuzz mode. Thus, we decided to focus on Jazzer’s strengths and develop a new infrastructure around it to overcome its weaknesses.

In the kotlinx.fuzz system, we addressed the key issues by providing additional features:

• An intuitive API for writing fuzz tests in Kotlin (showcased above).
• A Gradle plugin that allows configuring and running fuzz tests easily.
• A custom JUnit engine for easy IDE integration.

Bugs that kotlinx.fuzz helped us find

Using our system, we analyzed several Kotlinx libraries and discovered a number of interesting bugs and insights. Extensive fuzzing experiments on libraries such as kotlinx.serialization and kotlinx.collections.immutable allowed us to uncover several non-trivial errors, including: 

• Stack overflow in kotlinx.serialization.cbor, which is an error in the decoding of concise binary object representation that leads to infinite recursion, causing a stack overflow.
• Persistent ordered sets equality in kotlinx.collections.immutable, a bug in persistent set operations where removing elements leads to inconsistent results.
• Invalid serialization in kotlinx.serialization.json, in which setting classDiscriminator to an existing JSON name causes an undetected name conflict, resulting in incorrect serialization.

If you are interested in more details, check out all of our findings on our Trophy List page.

What’s next for kotlinx.fuzz

Having successfully addressed several key usability issues, we will continue improving kotlinx.fuzz and enhancing the overall user experience. Our goal is to provide greater user control over fuzz test reporting and configurability for individual tests.

Our development is now focusing on introducing major new features to extend the functionality of our fuzzer. First, we aim to develop a feature that generates regular tests based on crashes and failures detected by the fuzzer. Second, we plan to implement more powerful crash deduplication mechanisms that can group errors by their underlying causes. Finally, we intend to create a new, Kotlin-specific tool for automatic fuzz test generation, enabling users to harness the power of fuzzing with minimal manual input while also allowing for more user control.

How you can use fuzz testing in your Kotlin project

We believe that fuzz testing can be a powerful tool in detecting bugs and vulnerabilities in your codebase. We invite Kotlin developers and library maintainers to try out kotlinx.fuzz, provide feedback, and contribute to its development. Our GitHub repository can help you get started, and we will be happy to receive your comments and feedback. Your insights can help us make fuzz testing a staple in Kotlin development!