What is a benchmark?
In GenAI, benchmarks measure the performance of large language models, allowing users to compare accuracy, cost, and speed across different models. Public benchmarks often score model performance on tasks such as math problems and coding challenges.
Creating a benchmark requires curating dozens or even hundreds of test scenarios—each with its own prompts—and setting up code‑based evaluators or other LLMs used as judges to measure each model’s performance in response to the test cases.
Because general benchmarks focusing on math or coding problems are often not correlated with product needs within a specific industry, some researchers and companies have recently produced benchmarks for domain‑specific tasks, such as stock analysis based on SEC filings. However, until now, no benchmark has existed to measure LLM performance for payments industry use cases, such as chargeback workflows.
That’s where the Findustry AI Benchmark comes in.
No benchmark has measured LLM performance for payments industry use cases—until now.
Why create the Findustry AI Benchmark?
The Findustry AI Benchmark began with the questions we wanted to ask while building our own product.
Each time a new model is released by the labs, it is accompanied by academic benchmarks showing higher scores on instruction following, coding, math, and other tasks with mass AI adoption. But how reliable are those scores if the test problems themselves are in the training sets? And do the scores on horizontal tasks necessarily imply improvement for the needs of our vertical AI application, such as generating chargeback responses, matching evidence to network rules, and drafting rebuttal letters?
Put simply: how do we ensure we are choosing the right LLM for our customers?
Before the Findustry AI Benchmark, there were no rigorous, quantitative answers to these questions. The Benchmark is the result of building the evaluations we needed to measure model capability inside of our own software.
Are there benefits in addition to selecting the right model?
Yes. These evaluations give us the ability to systematically measure our AI application’s performance, which is especially important because GenAI systems are nondeterministic. Without evaluations, developers update prompts and simply hope they didn’t break anything else. In contrast, our evaluations ensure our changes are grounded in data rather than vibes, resulting in the enterprise‑grade quality our customers demand.
Also, because the frontier labs frequently update their models even between major releases, evaluations enable us to detect regressions when model behavior changes behind the scenes. This is similar to the muscle we built at our prior company, CardX, where we had to monitor downstream vendors such as gateways and acquirers.
What are the findings?
Larger, more expensive models may be outperformed by smaller, faster, and cheaper models for payments use cases.
A domain‑specific benchmark may not align with the general capabilities, such as long‑running reasoning, that make larger models superior on standard benchmarks. For focused, well-defined payments tasks, we find that the base Sonnet 4.6 model beats the larger Opus 4.6 while being faster and cheaper.
Small models can excel when specialized for industry context.
Smaller models can prove particularly effective when specialized for the relevant industry context: paired with Findustry AI’s domain intelligence, Gemini 3.1 Flash Lite surpasses the performance of the base (unspecialized) Gemini 3.1 Pro model, despite Flash having 76% lower costs and 92% lower latency.
The most recent model release from the same frontier lab is not necessarily best.
In our payments‑specific scenarios, the base Opus 4.6 model outperforms Opus 4.7 despite Opus 4.7 advertising an 11% higher score on the coding benchmark SWE‑Bench Pro. This means that a strategy of updating to the latest model release without systematically measuring its performance in the relevant industry context may be detrimental to product accuracy, emphasizing the importance of evals.
The base LLMs, which are largely trained on the open internet, do not match the scores of a vertical AI system that leverages domain expertise and proprietary data.
The Findustry AI harness—which includes our prompts, few‑shot examples, and tool outputs—increases the score of GPT‑5.5 by 41.7% compared to the same model tested without our harness.
Score Improvement: Base Model vs. With Findustry AI Harness
Performance on chargeback‑related tasks and corresponding cost to accomplish, by LLM model, with and without the Findustry AI harness. Updated May 28, 2026.
Below, side‑by‑side response comparisons demonstrate that one model, Opus 4.6, fails to understand the nuances of the chargeback process “out of the box” but, tested with our harness, has the domain depth to correctly respond.
Response Comparison: Procedurally Invalid Chargeback · Opus 4.6
Fraud chargeback (Visa 10.4) for a $47.36 online flower order. AVS failed (no street or ZIP match), and there is no prior transaction history or compelling evidence to prove the cardholder authorized this transaction. Recommending acceptance.
This dispute is procedurally invalid under card network rules. The transaction occurred on August 11, 2025, and the chargeback was filed more than 120 days after the transaction date, exceeding the permissible dispute timeframe. On this basis alone, the chargeback should be reversed in the merchant’s favor.
Without our harness, Opus 4.6 accepts (concedes) a chargeback despite the dispute being filed outside of the date range permitted by card network rules. With our harness, the model correctly fights the chargeback on the grounds that it is procedurally invalid.
Response Comparison: Prior Refund · Opus 4.6
We are writing in response to the above‑referenced chargeback dispute. After careful review of the case, the merchant, EchoWave Media, has elected to accept this chargeback.
Although we note that AVS returned a full 5-digit zip and address match and that the CVV was verified as correct at the time of authorization, these alone are insufficient.
Responding based on a full refund already issued before the dispute was filed, rendering this chargeback procedurally invalid. AVS/CVV exact matches and IP geolocation consistency further support the case.
A refund of $89.53 has already been sent to the cardholder on February 8, 2026 (ARN: 83981349912448002576570), prior to the filing of this dispute. Reversing this chargeback is necessary to prevent a double credit to the cardholder.
Without our harness, Opus 4.6 accepts (concedes) a chargeback even when there was a prior refund. With our harness, the model responds to the chargeback by citing the details of the prior refund, including the ARN, to prevent a double credit.
Even frontier models have meaningful error rates parsing PDF documents, which are foundational to many payments workflows.
In the real world, the success of AI systems often depends on extracting accurate information from documents with inconsistent layouts—such as terms of service, screenshots of payment gateway interfaces, and scanned receipts. If the model hallucinates critical information like an ARN, the entire workflow may fail, meaning AI agents are only as good as their document extraction capabilities.
PDF Document Parsing: Text Extraction Error Rate by Model
Error Rate (%)
Text extraction error rate based on word count; lower percentage scores are better.
Analyzing merchant processing statements is challenging for LLMs due to variation in formats across every acquirer.
To solve this problem, we designed our Findustry AI agent to learn from each processor format it sees. Without this specialized intelligence, no model achieves a score higher than 90% in our evaluations. Paired with the Findustry AI harness, however, both Gemini 3.1 Pro and GPT‑5.5 reach 97.9% accuracy.
Processing Statement Analysis: Accuracy Score by Model
Score
Accuracy score on processing statement analysis tasks. Variation in processor formats across acquirers makes this a uniquely challenging task for base LLMs. The Findustry AI harness enables the agent to learn from each processor format it encounters.
What comes next?
Our team will keep updating the Findustry AI Benchmark as new models are released. Our goal is for the Benchmark to be a valuable resource for our partners in the payments industry as the fast‑moving GenAI space evolves.
And, as our platform expands, we will extend the Findustry AI Benchmark from chargebacks and document extraction to additional task categories in payments. Based on our internal testing, we believe the scores will continue to demonstrate there is no single model that is best for all use cases; rather, different models perform better at different tasks, meaning the optimal AI system has the sophistication to know which LLM it should call at each step in its workflows.
No single model is best for all use cases—the optimal AI system knows which LLM it should call at each step in its workflows.
From the time we started Findustry AI, we knew we wanted to create a company that excelled at both product and research. Just like our first venture, CardX, was both a product and research organization in the surcharging field—authoring a U.S. Supreme Court brief and helping to change four state laws—Findustry AI will continue to make investments at the intersection of payment processing and AI research.
Especially in an environment where many companies are “AI washing,” we believe customers are looking for differentiated insights and trusted partners in AI transformation.
Ready to put this intelligence to work?
If we can help your organization, or if you have feedback or requests, contact our team.