From Guesswork to Hypothesis: Standing Up a CRO Program for a National Mortgage Lender

Client: National Home Lender
Role: CRO Strategist & UX Lead
Collaborators: SEO Strategist, Content Strategist, UX Designers (direct report)
Timeline: November 2021 — August 2024

At a glance

99.9% confidence, 54.74% visit-to-lead lift on a redesigned in-body lead form — the program’s strongest single result, later confirmed by a 0.25% → 0.48% year-over-year conversion improvement
95% confidence, 12.09% visit-to-lead lift on a guided-copy redesign of the refinance lead flow
Ran an A/A validation test before trusting live results — confirming no structural bias between control and variant buckets, a rigor step most young testing programs skip

The Problem

The client’s content team was sitting on strong organic traffic to their homeownership learning-center articles, content built to educate consumers on homeownership topics and grow the business through organic channels, but had never run structured A/B tests. Tests had been launched before, with little success improving leads. The core issue wasn’t a lack of ideas; it was that the team didn’t yet have a way to build a strong hypothesis before testing, or to learn anything from the tests that didn’t work.

I worked with the team to build a more mature experimentation program: how to plan tests, run them, and learn from the ones that failed.

My Approach

CRO workshop. In Q1 2022, I ran a workshop to bring the client’s team up to speed on conversion rate optimization, user experience, and A/B testing fundamentals. Including how to gather the right data before deciding to change the website, and how to extract a lesson from an inconclusive test rather than shelving it. It also helped align our own internal team on how UX and CRO fit into the client’s broader web strategy.

Aligning on the right metric. The business made money by originating mortgages, but asking someone reading “how to buy a house” to submit a lead was a big ask. The average lead could take 30 days to 3 months to turn into revenue. I mapped the full lead-submission journey to understand what we could and couldn’t control, and the team settled on visit-to-lead as the primary KPI: the point closest to the content experience we could actually influence.

Prioritization framework. I worked with the team to build a shared framework for discussing, planning, launching, and analyzing tests, including a shared process for identifying confounding variables so they’d get raised during analysis instead of after the fact.

User research insights. We drew on existing product user research insights to inform new test ideas rather than starting from scratch.

Developer partnership. All tests were built in Adobe Target, which meant close coordination with the internal development team. I worked directly with our developer to explain the test rationale and which URLs the new experience needed to target.

The Results

In-body lead form redesign — 99.9% confidence, 54.74% visit-to-lead lift Redesigned the in-body lead form component and tested it across 77 articles in the learning center. The lift held up under a slower-moving check too: year-over-year conversion rate for the same period improved from 0.25% to 0.48%.

Guided copy experience (refinance lead form) — 95% confidence, 12.09% visit-to-lead lift Added contextual microcopy to each step of the embedded refinance lead form to help prospective customers understand what was being asked of them at each step. Tested across 79 articles against 656,939 visits in the trailing 90 days. Variant win.

Brand trust CTA — 69% confidence, 4.2% visit-to-lead lift User research showed that visible value-adds increased trust and conversion, so we added a third-party customer-satisfaction trust badge under the in-body CTA. Tested across 66 articles.

A/A validation test Before trusting any live experiment results, we tested the same lead-form experience against itself (193 vs. 169 leads across 56,498 vs. 56,336 visits, 89 articles) to confirm no structural bias between control and variant buckets. A small, unglamorous test — but it’s the kind of methodological discipline that makes every result above worth trusting.

“Oz has elevated our CRO program, and helped spearhead a lot of new initiatives to show our team as a true partner with the client, not a vendor. He spent a lot of the second half of 2022 working alongside the leadership team to show our value, and by 2023 became more of a voice on client calls to keep that momentum going. He’s also gotten much better at explaining to the team what UX needs in order to effectively execute their role, and communicating recommendations back in a way the team can act on.” — Feedback from account leadership

Challenges

Lack of clean data attribution. Organic, algorithm-driven traffic to educational content makes clean randomization difficult, seasonality and search-ranking shifts move the numbers independently of any test. We leaned on 90-day before/after windows and year-over-year comparisons alongside the live test results, rather than trusting a single simultaneous split in isolation.
Heatmap data spread across multiple articles. Heatmap data works great when analyzing a single page that receives plenty of traffic, but most of our articles did not receive enough traffic to find anything meaningful off one heatmap view. Consolidating multiple heatmap reports from the articles takes time, but it gave us the insights we needed to provide data-informed recommendations.
Other department priorities affecting which tests could launch. Company-wide initiatives that changes the entire domain experience were confounding variables we had to account for. Our team documented all of those changes to monitor closely to understand how it impacted the tests we were running.
Federal interest rate swings. Mortgage demand is macro-sensitive in ways no test design controls for, a reminder to read results in the context of what the market was doing, not just what the variant did.

What I Learned

On staying aligned with my own team. I kept falling behind on analysis, new test design, and QA because I was in too many meetings. The fix was recording short videos walking my internal team through test ideas ahead of client presentations, so everyone was aligned without needing another live meeting.

On developing less experienced team members. I learned to do a better job looping my direct reports into larger roadmap initiatives and client updates — context they weren’t otherwise getting, since they weren’t part of the broader team-leads conversations, but needed in order to connect their day-to-day work to the bigger picture.