CRUST-Bench: New Benchmark for Safe C-to-Rust Transpilation with Realistic Code & Tests

This is a Plain English Papers summary of a research paper called CRUST-Bench: New Benchmark for Safe C-to-Rust Transpilation with Realistic Code & Tests. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

The Need for Safe C-to-Rust Code Translation Benchmarks

C-to-Rust transpilation is essential for modernizing legacy code while enhancing safety and interoperability with modern Rust ecosystems. Many critical infrastructure systems are written in C, creating an urgent need for reliable automated techniques that support not only C-to-Rust translation but, more importantly, C-to-safe-Rust migration. While Rust supports unsafe code for low-level operations, the core value of migration lies in producing code that executes within Rust's safe subset, allowing users to benefit from Rust's strong memory safety guarantees.

Despite rapid progress in large language models (LLMs) for code generation, fully automated C-to-safe-Rust transpilation remains challenging. Existing machine learning benchmarks largely focus on competitive programming problems rather than realistic, multi-file systems programming. Other efforts like SWE-bench target bug-fixing scenarios with localized edits, not whole-program translation or structural refactoring.

To address these limitations, the researchers introduce CRUST-Bench, a benchmark for evaluating automated C-to-Rust transpilation in realistic settings. CRUST-Bench comprises 100 C repositories, each paired with manually crafted Rust interfaces and test cases to assess the correctness and quality of transpilation. The Rust interfaces serve as formal specifications, defining function signatures, type annotations, and ownership constraints to guide the translation process. These interfaces enforce idiomatic and memory-safe Rust patterns, ensuring transpiled code adheres to Rust's safety guarantees. The accompanying test cases provide validation, verifying that the generated Rust code correctly implements the specified behavior.

Figure 1: Example of a CRUST-Bench task showing the challenge of translating pointer-based C code to safe, idiomatic Rust.

Why Current C-to-Rust Transpilation Benchmarks Fall Short

A number of benchmarks have been proposed to evaluate source code translation and transformation tasks, including transpilation, automated debugging, and safe code generation. While these benchmarks provide valuable insights, those that focus specifically on C-to-Rust transpilation tend to exhibit important gaps—particularly limited task scope, lack of robust correctness validation, and insufficient evaluation of memory safety. Many are restricted to small, synthetic examples or overlook whether the generated Rust code adheres to safe and idiomatic practices.

Table 1: Comparison of C-to-Rust transpilation benchmarks.

Compared to prior benchmarks, CRUST-Bench includes a substantial number of real-world projects while keeping them within reach for LLM-guided transpilation techniques. A key feature of CRUST-Bench is that it provides safe Rust interfaces and corresponding test files. This is important because existing transpilation tools, such as c2rust, often produce naive translations that rely heavily on unsafe constructs, sidestepping Rust's safety guarantees rather than properly adapting C code. LLM-based approaches also risk generating inconsistent interfaces across files, making it difficult to produce a coherent, working Rust codebase.

Inside the CRUST-Bench Framework: Structure and Validation

Each instance in CRUST-Bench is built on a C repository, formalized as a collection of source files. The goal of transpilation is to produce a corresponding Rust repository with a parallel file structure, ensuring that each Rust file is a direct translation of the corresponding C file.

Validation of the transpiled Rust code is based on three criteria:

1. It must conform to a well-defined interface that specifies a set of abstract datatypes and functions
2. The transpiled Rust code must compile without errors
3. A set of tests must pass, confirming functional correctness

Table 2: Properties of C code in the CRUST-Bench dataset.

The projects in the benchmark were sourced from open-source repositories on GitHub, covering a diverse range of software categories created between 2005 and 2025. The selection includes programming language infrastructure (e.g., compilers), algorithmic libraries (e.g., solvers), system utilities (e.g., shell implementations), and others.

To construct CRUST-Bench, the researchers applied a multi-stage preprocessing and selection pipeline. They filtered for projects entirely written in C that meet a minimum complexity threshold: containing at least one dynamic memory allocation keyword, including build scripts, and having associated test files. Following initial filtering, they manually reviewed the remaining projects to assess code quality and ensure test completeness.

Table 3: Statistics on Rust interfaces in CRUST-Bench showing the complexity of the required transpilation task.

The final benchmark includes 100 C projects annotated with corresponding Rust interfaces, capturing a wide spectrum of interface complexity and idiomatic Rust features. The dataset includes 3,085 interface functions across 299 interface files, with an average of 30.9 functions per project. This benchmark structure provides a comprehensive and realistic evaluation of C-to-Rust transpilation capabilities.

Experimental Setup: Models and Systems Evaluated

The researchers evaluated both closed-source and open-weight large language models (LLMs) on the CRUST-Bench benchmark. For closed-source models, they included GPT-4o, o1, and o1-mini from OpenAI; Claude-3.5-Sonnet and Claude-3.7-Sonnet from Anthropic; and Gemini-1.5-Pro from Google. On the open-weight side, they evaluated models with strong code generation capabilities: Virtuoso (a distilled variant of DeepSeek-V3), QwQ-2-32B, and LLaMA-3-8B.

For models that support temperature control, they used greedy decoding (T=0) to ensure deterministic outputs and consistent pass@1 evaluation. Empirically, they found that higher temperatures significantly degraded performance, likely due to the long-context requirements and stringent correctness constraints of the tasks. The best results were obtained using a general task description followed by concise, point-based instructions. Notably, explicit directives—such as "do not use libc" and "the code must compile"—proved more effective than vague statements like "generate safe Rust."

The researchers investigated two strategies for leveraging additional compute to improve model performance through iterative self-repair:

1. Compiler repair: Incorporates only compiler error messages into the prompt during each repair round, allowing the model to iteratively address syntactic and type-level issues flagged by the Rust compiler.

2. Test repair: Extends the first approach by also including information about failing test cases in the prompt, providing richer feedback about the correctness of the generated code.

For both strategies, they performed three rounds of self-repair per task, balancing the benefits of iterative refinement with practical runtime constraints.

The researchers also adapted SWE-agent into a two-stage workflow they refer to as pipelined SWE-agent. In this setup, an LLM first generates an initial Rust implementation from a C source file and its associated Rust interface. SWE-agent then attempts to repair any resulting compiler errors by iteratively editing the generated code.

Results and Analysis

The results reveal that even the best-performing models struggle with C-to-safe-Rust transpilation, with OpenAI's o1 achieving only a 15% success rate in a single-shot setting.