How to test a new mortgage lead source

Run the new source beside one you already know rather than in place of it, hold everything else still, the dialer, the script, the hours, the geography and product mix, the CRM path, and decide before the first lead arrives what result would make you stop. Then leave it alone long enough to read.

How long to run it, and how many leads

Length is a decision you make before you start, not one you make when the numbers look good. Fix a window in whole weeks so that day-of-week differences fall on both sides evenly, and fix the count of leads you will judge on at the same time. Stopping the moment a run of good ones arrives, or a run of bad ones, turns the test into a story about when you happened to look.

What to hold constant while the test runs

Hold constant everything that is not the source: who dials, how quickly they dial, the script they read, the calling hours covered, the states and loan products ordered, and the route the lead takes through the CRM. Change the dialing and the source together and the result measures neither one. Most buyers get this part wrong, because the urge to improve the follow-up peaks at exactly the moment a new source arrives.

Source: NIST/SEMATECH e-Handbook of Statistical Methods, NIST Handbook 151, National Institute of Standards and Technology, 2002, which defines nuisance factors as those that may affect the measured result but are not of primary interest, and describes blocking as creating groups "in which the nuisance factors are held constant and the factor of interest is allowed to vary". itl.nist.gov

A known source beside it, and a stopping rule written first

A known source kept running at its usual volume through the whole window is what gives the new one something to be read against, because both then meet the same market in the same weeks with the same team. Judging a new source against last quarter's memory compares two different periods as well as two different sources. The stopping rule belongs in writing before the first lead arrives: name the result that would end the test early, and the one that would end it in the new source's favor.

Source: ICH E10, Choice of Control Group and Related Issues in Clinical Trials, Step 4 version dated 20 July 2000, issued as guidance by the US Food and Drug Administration in May 2001, which notes that a control group external to a study can be dissimilar on a wide range of factors other than the treatment itself. On writing the rule down first, ICH E9, Statistical Principles for Clinical Trials, Step 4 version dated 5 February 1998, asks that interim analyses be planned in advance and that stopping guidelines be described in the protocol. Both are clinical-trials guidance, not lead-buying guidance; what carries across is the design principle, not the subject matter.

Reading the result once it arrives is a separate question, and how many bad leads it takes before the source rather than the desk is at fault, plus when a verdict is actually due, is covered in judging a paid lead source.

No primary source sets a minimum number of leads for a test like this one, and we looked. The NIST handbook above gives sample-size methods that depend on how large a difference you want to be able to detect, not a fixed count, and neither the federal statistical literature nor any regulator publishes a figure for buying leads specifically. Anyone quoting one is quoting themselves.