Can you trust a self-reported resolution metric?

The question

Every complaint closed at a US financial company gets a label, chosen by that company, saying how it ended. Nobody checks the label.

The CFPB publishes every consumer complaint it forwards to a company, along with how the company said it closed: with monetary relief, with non-monetary relief, or with an explanation. The agency confirms that the consumer has a relationship with the company. It does not verify what the company says happened next.

That makes relief rate a tempting metric and a risky one. It is the only published, cross-company signal of whether complaints actually get resolved, and it is entirely self-reported. I wanted to know what it is really measuring before treating it as a quality benchmark.

Metrics

Relief rate

Complaints closed with monetary or non-monetary relief, divided by closed complaints. Self-reported by the company, verified by no one. Does not measure whether the consumer's problem was fixed, how long it took, or whether the complaint was valid.

Timely response rate

Complaints the company answered within the CFPB's window, divided by all complaints. Measures acknowledgement speed, not outcome. Sits near 99% for every company in the dataset, so it separates nothing.

Relief rate is the only one of the two that varies, which is why the rest of this page is about whether its variation means anything.

What the data shows

The headline gap is real and survives every control I could test. The company-level picture is what complicates it.

Across 344,428 closed complaints in three product families, banks reported relief on 26.5% and fintechs on 3.5%. A logistic regression controlling for product family and year puts fintechs at 0.20× the odds of relief that banks get (95% CI 0.19 to 0.21). Controlling for issue and sub-issue moves it to 0.21 and 0.20, so complaint mix does not explain the gap.

Five companies in the dataset — Wells Fargo, Capital One, SoFi, Chime and Betterment — are included in the group aggregates above but their company-level rates are not yet pulled into this writeup.

Banks are higher within every product family

And within the same issue

What I ruled out

Before concluding anything about service quality, I tested the alternatives the data could reach:

• Complaint mix. Controlling for issue and sub-issue leaves the odds ratio at 0.20–0.21.
• Fee complaints. 8.7% of bank complaints versus 2.0% of fintech complaints, and they lift the bank rate only from 26% to 31%.
• Customer mix. Older Americans file 10.4% of bank complaints and 3.2% of fintech complaints. The tag raises relief only slightly.
• One company driving it. Leaving each company out in turn, the gap holds in direction every time, ranging from 0.11× without Chime to 0.41× without Cash App. Cash App drives much of the size but none of the direction.

Four things the data cannot reach, and that I am not going to claim either way: partner-bank attribution, dollar amounts, whether a company fixed the problem before the complaint was filed, and how closely each company is supervised.

The pattern that changed the question

Relief rates do not behave like a service measure. In places they behave like a filing convention.

Three things in the company-level data are hard to explain as service quality:

• Exact zeroes. Robinhood reported relief on 0 of 4,269 in-scope complaints, 0% in every year and every product family. Wealthfront is also 0% in every year. Cash App is under 0.5% in every year. A company that occasionally refunds someone does not land on zero four years running.
• A step change. Coinbase moved from 9% in 2023 to 41% in 2026, and the shift holds within issue types rather than following a change in what people complained about.
• Banks are not uniform either. They span 16% at Chase to 44.3% at Citi, a spread about as wide as the fintech-to-bank gap itself.

None of this proves mislabelling. A firm with a strict no-refund policy would also report near-zero relief, and that is a real policy difference rather than a recording artifact. But both explanations produce the same number, which is the problem: the metric cannot tell them apart. Nothing in this dataset can. The only field describing what happened is the one the company wrote, so to get any further I had to stop reading the labels and start reading the complaints.

The audit

I read 240 complaints and coded one question: if this person is telling the truth, would fixing their problem involve giving money back or restoring access?

That question is deliberately narrow. I am not judging whether consumers are honest, and I am not guessing what the company did next. The narrative is written when the complaint is filed, before the company responds, so the outcome is not in the text and nothing in it can confirm or refute a label. What it can do is describe a situation, and a situation either calls for relief or it does not.

Four companies, 60 narratives each, all in money transfer and virtual currency, closed complaints between January 2025 and August 2026. Robinhood and Cash App both report 0.0% relief. Coinbase reports 29.9% on near-identical complaints and is the comparison. Bank of America is there as a bank anchor.

Everything was fixed before I read a single narrative: the codebook, the sample size, the comparison, the sampling seed. I wrote down what I expected to find and what would count as being wrong. That turned out to matter.

What the complaints describe, against what the companies report

Robinhood’s complaints have a shape. 73% of its sampled narratives are account lockouts — funds frozen behind a failed ID check, a review that never ends, a transfer that leaves one side and never arrives at the other. Of its 47 lockout and unauthorised-transaction complaints, 44 described a loss that a resolution would have to address. “Closed with explanation” is a hard thing to reconcile with a person who cannot reach their own money.

My main test failed

I predicted the two zero-relief companies would look like Coinbase. Pooled together, they came out significantly worse — the opposite of what I said would happen.

Pooling Robinhood and Cash App against Coinbase gave 50.8% against 75.0%, p=0.002, in the wrong direction. If I had not written the prediction down first, this is the point where it would have been easy to quietly report something else.

The cause turned out to be worth more than the test. 36 of my 60 Cash App narratives are the same copy-paste form letter. Identical wording, differing only in typos, all alleging inadequate customer service and a Regulation E violation in the abstract and describing no specific loss. One of them opens: “Anyone need the copy and paste here you go.” All 36 were coded unclear, correctly — there is nothing in them to code. Drop them and the pooled difference disappears: −2.4 points, p=0.749.

The design error was mine. I assumed two companies sitting at zero were the same phenomenon. They are not. Robinhood’s complaints are lockouts; Cash App’s sample is mostly boilerplate. Pooling them destroyed the comparison. The Robinhood-to-Coinbase contrast, which is the one the finding rests on, holds in every specification I tried — with the template in, with it out, with scams in, with scams out, counting unclear as warranted.

The form letter is its own finding

That template appeared 0 times in the 180 narratives from the other three companies. In the Cash App sample it is 60%, which puts the population share between 47% and 72% with 95% confidence.

Cash App has 50,413 money-transfer complaints in this window, far more than anyone else in the dataset. If half to three-quarters of its narratives are one circulating template, then complaint volume at Cash App is partly measuring template circulation rather than customer harm. That is a second self-reported metric failing from the other direction: the company grades its own resolutions, and the regulator counts complaints that may be largely copies of each other.

What else went wrong

Three things, all of which make the finding weaker and all of which belong here rather than in a footnote.

The blinding did not hold

I scrubbed company names from every narrative, then recorded which company I thought each one was. I was right 87.8% of the time against 33% by chance — Cash App 96.7%, Coinbase 85.0%, Robinhood 81.7%. Bank of America was unblinded by design and coded last, and I got 100% of those.

Scrubbing names blinds nothing. The form letter identifies itself. Crypto vocabulary separates an exchange from a brokerage. So these are one informed coder’s judgments, not a blind protocol, and that is how they should be read. What keeps it honest is that the codebook and the comparisons were locked before I read anything, and the deviations are listed below.

One of my own rules drifted

The codebook said scam complaints default to unclear, because the consumer authorised the payment and the company’s duty is contested. In practice I coded 38 of 39 as no. I applied it consistently, and it works against my own headline — scams are 18% of the Coinbase sample and 5% of Robinhood’s, so the stricter rule pushed Coinbase down and narrowed the gap I was looking for. The analysis set above drops scams entirely.

Agreement is moderate

A second rater coded all 240 independently against the locked codebook. Cohen’s κ = 0.549 overall and 0.497 on the analysis set, which is moderate rather than strong. Almost all of the disagreement is the scam rule: harmonise it and κ = 0.756. Agreement on what the complaint was about was much higher at κ = 0.838, so we read the same facts and disagreed about entitlement, not events.

The second rater was Claude, which had already seen the company-level results before coding. That pulls a second rater toward the first and inflates agreement. These figures are a floor on disagreement, not a clean reliability estimate. A person who has seen none of this re-coding 40 of them would be worth more than those 240.

And the size of it

Sixty narratives per company detects differences of roughly 25 percentage points and nothing finer. Narratives are published only when the consumer opts in, so this is not a random sample of complaints either. It is a spot check.

What I am claiming

Robinhood’s complaints look like Coinbase’s, and it reports relief on none of them. Every company I sampled shows a double-digit gap between what its complaints describe and what its label records. An outsider can measure that gap with 165 hand-coded narratives and a weekend.

What I am not claiming: that fintechs under-report relief as a class. Bank of America’s gap is 63 points and Coinbase’s is 62 — the same pattern, which is exactly why this is a story about self-reported metrics and not about fintechs. Nor that any particular label is false. A company might resolve problems without recording them as relief, or resolve them before a complaint ever reaches the CFPB. Both of those produce the same number as mislabelling, and this method cannot separate them. It can only show that the number has drifted from what the complaints say.

The checking stopped in August

The CFPB stopped publishing complaint narratives on 14 August 2026. The self-reported relief label is still published.

The “consumer disputed” field was removed in 2017. The narratives went this August. So the metric a company grades itself on remains public, and both fields that let an outsider check it are gone. The narratives I used survive in a FOIA Reading Room archive, which is where this sample came from. This audit is close to the last one anyone can run on this data.

That is the part that generalises. The failure mode here is not a company lying. It is a metric with no verification surface, published for years, that nobody can check — and that keeps being published after the checking stops. Nobody decided to make it unverifiable. It happened one field at a time, each removal defensible on its own.

How you find this from outside

I wrote these three down before the audit ran. All three earned their place, and the third one half failed, which is the useful part:

• Look for flatlines. A rate sitting at exactly 0% across every period and category is describing a policy or a recording convention, not a varying outcome. Robinhood’s four straight years of zero is what started this.
• Test step changes within category. When a metric jumps, check whether it moved inside every category or whether the mix underneath it changed. Coinbase went from 9% to 41% and the shift held within issue types, which points at recording rather than at customers.
• Audit the source blind — and check that the blinding worked. Code the underlying cases with the label and the company hidden, then compare. I did that and the blinding failed at 87.8% identification. Recording a blinding check is what turned a claim I could not support into a limitation I could state.

So what, beyond the CFPB

None of this depends on US complaint data. What transfers is the method, not a verdict: any team that assigns its own metrics can run the same checks.

1. Any metric the team sets on itself needs a sampling check

Robinhood’s label drifted from its complaints for four years, in public, because nobody read the complaints. That is not fraud. A resolution label gets set by whoever closes the ticket, and once a convention takes hold nothing in the pipeline compares the label to what the ticket said.

So: any metric where the team being measured also assigns the value — resolution rate, first-contact resolution, escalation reason, CSAT disposition — gets a monthly blind sample. Fifty closed tickets, coded by someone outside the closing team on what the client was asking for and whether it was addressed, compared to the recorded disposition. Track the divergence rate as its own number. Fifty tickets is about two hours. The check that would have caught Robinhood costs less than the meeting where someone asks why the number looks strange.

Caveat I should apply to my own recommendation: 60 per arm only caught a 25-point divergence. A monthly 50-ticket check is a smoke alarm, not a thermometer.

2. Count distinct problems, not tickets

Sixty per cent of my Cash App sample was one form letter. Volume drives staffing and prioritisation, and a number inflated by duplicates sends people to the wrong fire. It fails the other way too — a hundred clients hitting one broken flow and describing it a hundred different ways looks like a hundred small problems instead of one big one.

Before a volume figure is used for a decision, cluster the text and report two numbers: tickets received, and distinct problems described. When those diverge, that is the signal.

3. Write down what would let someone check the number

Two CFPB fields disappeared eleven years apart and the metric outlived both. The same thing happens internally without anyone intending it: a free-text field becomes a dropdown because dropdowns report cleanly, recordings age out on a retention schedule, a survey adds a required rating and drops the optional comment. Each change makes reporting easier and quietly removes the evidence that would let someone audit the number.

When a metric is defined, document what evidence would verify it independently, and treat that evidence as part of the definition. A schema change that removes it is a change to the metric, not a storage decision. Where that line is blank, the metric is unauditable and the scorecard should say so.

Everything is in the open

The full workbook has all 256 coded rows with the narratives, both raters’ codes, the codebook, the exclusions, the reliability figures and the method. The sampling seed is fixed, so anyone can redraw the same sample and check my coding against theirs. I would rather be corrected than cited.

Still open: adjudicating the 62 inter-rater disagreements, of which 37 are the single scam rule; and an exact template count across all Cash App narratives, which is currently a confidence interval.