ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

The world of enterprise software development is constantly evolving, and a crucial, often complex, part of this evolution is migrating applications between different frameworks. This isn't just about updating a few lines of code; it involves a deep understanding of architectural patterns, dependencies, and ensuring the application behaves exactly as it should in its new environment. With the rapid advancements in Artificial Intelligence, especially in large language models (LLMs) and AI agents, the promise of automating these migrations is becoming a hot topic. But how effective are these AI agents really?

Enter ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration. This open benchmark aims to provide a clear, standardized way to measure how well AI agents can handle the intricate process of moving enterprise Java applications between major frameworks.

What is ScarfBench?

ScarfBench, short for "Self-Contained Application Refactoring Benchmark," is a comprehensive suite designed to evaluate the capabilities of AI agents in migrating enterprise Java applications across different frameworks. It was introduced by IBM Research, with authors including Raju Pavuluri, Rahul Krishna, Srikanth Govindaraj Tamilselvam, Bridget M., Ashita Saxena, George Safta, Advait Pavuluri, and Michele Merler.

Unlike traditional software engineering benchmarks that often focus on simpler tasks like bug fixing or code generation, ScarfBench tackles the much more complex challenge of cross-framework migration. Its core purpose is to determine if an AI-driven migration results in a truly working, reliable system, rather than just code that compiles.

The benchmark specifically targets three major Java ecosystems: Spring, Jakarta EE, and Quarkus. These frameworks represent a significant portion of enterprise Java development, making ScarfBench highly relevant to real-world modernization efforts.

Why Enterprise Java Migration is a Big Deal

Java remains a cornerstone of enterprise software, with countless applications having long lifespans, often outliving the original architectural assumptions they were built upon. Organizations undertake framework migrations to improve maintainability, enhance cloud readiness, boost developer productivity, and access modern capabilities.

However, these migrations are notoriously difficult. They involve more than just syntax changes; they require preserving crucial elements like:

• Business logic and domain behavior
• Transaction boundaries and consistency guarantees
• Dependency injection lifecycles and architectural structure
• Persistence mappings and relational integrity
• Security configurations and integration contracts

A small error in any of these areas can prevent an application from deploying successfully or even cause subtle runtime issues that are hard to detect. This complexity is precisely why a robust benchmark like ScarfBench is so important for evaluating AI's role in this domain.

How ScarfBench Works: A Deep Dive into Evaluation

ScarfBench provides a systematic and reproducible way to evaluate AI agents. It consists of a dataset of Java applications, an evaluation infrastructure, and a public leaderboard.

The Benchmark Dataset

The dataset is a critical component of ScarfBench. It includes:

• Developer-verified enterprise applications: These applications are implemented across Spring, Jakarta EE, and Quarkus. Each implementation has been manually converted and rigorously tested by experienced developers to ensure functional correctness, adherence to framework-specific idioms, and architectural integrity.
• Focused examples: These are narrow, layer-level applications designed to isolate specific enterprise technology concerns, such as dependency injection, persistence, presentation, infrastructure, or business domain logic.
• Whole applications: These are larger, end-to-end open-source systems that combine multiple architectural layers, reflecting realistic use cases and cross-layer coupling. Examples include CargoTracker, Coffee Shop, DayTrader, PetClinic, and RealWorld.

In total, ScarfBench comprises 34 application families, yielding 102 framework-specific variants (around 151,000 lines of Java code) and 204 directed refactoring tasks. Each task gives an AI agent a working source application and a target framework, challenging the agent to synthesize a new implementation that preserves the original behavior.

The Evaluation Process

Correctness in ScarfBench is not just about compiling code. It's evaluated by an application-specific executable oracle that checks three critical aspects:

1. Build Success: Does the migrated application compile without errors in the target framework?
2. Deploy Success: Can the migrated application be successfully deployed in a containerized target runtime environment?
3. Behavioral Validation: Does the deployed application pass a suite of behavioral tests, ensuring it preserves its original functionality and observable interface? This is measured by 1,331 expert-written tests.

A task is only considered successful if it passes all three criteria. This rigorous approach highlights that successful migration goes far beyond mere code translation; it demands an understanding of the entire application lifecycle and its environmental dependencies.

The ScarfBench project also provides an open-source evaluation harness, a runtime CLI companion, and comprehensive documentation to help researchers and practitioners get started.

What ScarfBench Reveals About Current AI Agents

The initial evaluations using ScarfBench have provided crucial insights into the current state of AI agents for Java framework migration. Several state-of-the-art coding agents, including those powered by models like Claude Code (Claude Opus 4.6), Gemini CLI (Gemini-3.1 Pro), Codex (GPT-5.4), Opencode (GLM-5.1), and Qwen CLI (Qwen3.5-397B-A17B), were tested.

The findings indicate that despite impressive performance on other software engineering benchmarks, framework migration remains a significant challenge for current AI agents.

• Low Behavioral Success: Even the strongest current agents achieve less than 10% behavioral success for whole-application migrations. For focused-layer migrations, the strongest agent achieved only 15.3% aggregate test pass, and for whole applications, it was 12.2%. Only one of the 204 tasks yielded a fully behaviorally equivalent target.
• Overestimation of Quality: Compile success consistently exceeds deploy success, which in turn exceeds behavioral success. Relying solely on build success significantly overestimates the quality of a migration. Agents can also be overconfident in their self-assessments. For example, one agent reported successful builds for 29 out of 30 whole applications, but only 22 actually built successfully.
• Difficulty Asymmetry: The difficulty of migration varies across framework pairs and architectural layers. Migrations between Spring and Quarkus appear to be the most tractable, while Jakarta-targeted migrations are often the hardest.
• Beyond Code Transformation: A major takeaway is that the biggest challenge in framework modernization isn't just translating Java code; it's managing the complex web of dependencies across configuration, infrastructure, and runtime environments. Agents frequently struggle with environmental issues, Docker cache inconsistencies, port connectivity, and build tooling.

These results clearly show a substantial gap between generating compilable code and actually preserving application behavior in a new framework.

Implications for Developers and AI Practitioners

ScarfBench offers valuable insights and tools for several communities:

• Software Developers and Enterprise Architects: For those on the front lines of modernization, ScarfBench underscores the current limitations of AI agents in complex migration scenarios. It highlights the need for careful human oversight and validation even when using AI-assisted tools. It also provides a public leaderboard to compare different solutions.
• AI Researchers: ScarfBench provides a challenging and realistic benchmark to drive further innovation in AI-assisted program transformation. It exposes specific failure categories (build, deploy, test stages) that researchers can target to improve agent capabilities. The benchmark dataset and evaluation infrastructure are open resources for further study.
• Tool Builders: Companies developing automated modernization systems can use ScarfBench to rigorously assess the effectiveness of their tools in a standardized, reproducible manner.

The project encourages researchers, practitioners, and framework communities to evaluate their agents, contribute new migration scenarios, and help advance the state of the art in AI-assisted application modernization.

Conclusion

ScarfBench represents a significant step forward in understanding the true potential and current limitations of AI agents for enterprise Java framework migration. By focusing on the critical aspects of build, deployment, and behavioral preservation, it provides a much-needed realistic evaluation of these advanced tools. While current AI agents show promise, the benchmark clearly indicates that achieving truly autonomous and reliable application modernization is still an unsolved problem, requiring continued research and development to bridge the gap between code generation and comprehensive architectural understanding. ScarfBench is an essential resource for anyone working to make AI a more reliable partner in complex software engineering tasks.

Frequently Asked Questions

What problem does ScarfBench aim to solve?

ScarfBench aims to solve the problem of reliably evaluating AI agents' ability to migrate complex enterprise Java applications between different frameworks like Spring, Jakarta EE, and Quarkus, ensuring that the migrated applications not only compile but also deploy and function correctly.

Who developed ScarfBench?

ScarfBench was developed by researchers at IBM Research.

Which Java frameworks does ScarfBench cover?

ScarfBench focuses on migrations across three major Java ecosystems: Spring, Jakarta EE, and Quarkus.

What are the key metrics used by ScarfBench to evaluate AI agents?

ScarfBench evaluates AI agents based on three critical metrics: whether the migrated application successfully builds, successfully deploys, and passes behavioral validation tests that ensure functional equivalence to the original application.