How proxies for lawyer success can mislead us in finding merit

It’s right there, on page 40 of Van Halen’s concert contract, under Food Requirements: “M & M’s (WARNING: ABSOLUTELY NO BROWN ONES).” Why did these rock prima donnas ban backstage brown M&Ms? In his 1997 biography, David Lee Roth claimed it was a way to see if the performance venue had actually read the entire contract:

We’ll probably never know for sure, but taking Roth at his word, “No Brown M&Ms” was a proxy. A proxy is an “If-Then” decision-making tool – in Van Halen’s case, if there are brown M&Ms, then better double check everything (and if there are no brown M&Ms, then we’re likely OK).

We use proxies all of the time to help us make efficient decisions. If this product or service provider is a familiar name brand, then we’ll be happy with choosing it (despite its higher price). And if this food container is beyond its sell-by date, then we shouldn’t buy it (even if it may not be perishable for a far longer time). Proxies make our decision-making faster and simpler. And they make sense with low cost, low risk decisions.

But for decisions with higher risk and higher cost, what if the proxy we use isn’t predictive? What if the If doesn’t reliably lead to the Then?

One of the most common proxies traditionally used by law firms when hiring new lawyers is the ranking of the recruit’s law school. Another is high class rank in their law school class. These proxies for future success make gut sense to us because, we assume, the “best of the best of the best” go to highly ranked law schools and are at the top of their class. So, in the If-Then logic of a proxy, surely highly ranked law school and high class rank are reliably predictors of future success as a lawyer, right?

In my experience, these are not useful proxies. High law school rank and high class rank do not reliably predict which new lawyers will be successful.

I should explain. At an earlier point in my career I was the Recruiting and Professional Development Partner at an Am Law 100 law firm. Periodically I pulled hiring-file resumes for all of our associates, totaling about 100 in early years and eventually, due to our firm’s growth, closer to 200 associates. I next identified which associates came from the top tier of US News-ranked law schools, and which were in the top 10% of class rank or Order of the Coif. Then I overlaid this mapping of all associates with their performance evaluation results. Notably, my old firm by then had abandoned lockstep pay and promotion for associates, replacing them with a competency-based level system. Associates were compensated and advanced toward partnership based not upon tenure but instead upon their demonstrated growth in competencies. So, our semiannual performance evaluations were truly important and carefully done.

The results were at first surprising to me, and in later years confirming:

• High law school ranking did not reliably predict high performance. Graduates of second tier law schools were more likely to have exemplary results in their associate performance evaluations than those from first tier law schools.
• High class rank did not reliably predict high performance. The percentage of associates who received the highest performance evaluations remained essentially the same regardless of whether they were in the top 10 percent of their law school class and regardless of their law school’s ranking.
• High law school ranking did not reliably predict high productivity. Associates who graduated from second tier law schools were more likely to have high billable hours than those from first tier law schools.

Don’t get me wrong – it’s a big deal for law students to graduate with strong performance from excellent law schools. But used in new lawyer recruiting as a determinative proxy, high law school rank and high class rank are no more reliable in predicting future success than only hiring new lawyers born on Mondays or Tuesdays (full disclosure, I’ve never checked the day-of-the-week thing).

When I chatted with Dr. Arin Reeves about her Written in Black & White study (see last post), I also asked about her experience with the reliability of such proxies. During her many years of working closely with U.S. and global law firms, Dr. Reeves similarly struggled (with obviously more expertise than me!) to identify any reliable, predictive proxies for lawyers’ future success.  The closest factors Dr. Reeves ever found correlating with high performance evaluations were (1) the person had completed a part-time or evening program for advanced skills, or (2) the person had a military background.  In her experience, traditional proxies, such as what law school the person attended and their class rank, were simply not reliable predictors of high performance as a lawyer.

Data is of course important input for decision-making, but we can become ensorcelled by numbers and rankings. Remember that old adage from W. Edwards Deming, “if you can’t measure it, you can’t manage it”? Turns out Dr. Deming instead said the opposite: “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.” Peter Drucker didn’t say it either. Drucker was a renowned advocate for measuring results and performance, considered by him as crucial to an organization’s effectiveness. But he also said:

But so what if high law school ranking and high class rank are unreliable proxies for future lawyer success. What’s the harm in continuing to use them in recruiting new lawyers?

• If we’re focused on these ineffective proxies, we’re less likely to be looking for the right things. When we lock-in to ineffective hiring criteria, we’re less likely to observe what is truly important about a candidate – whether they have the drive, talents, and competencies that actually do lead to success at our law firm. We also open the door to bias. There are obviously many reasons why talented individuals may not – or cannot – attend a top tier law school, just as there are many talents essential to lawyering success that are not well-measured by law school grades. Remember, it’s law school, not lawyer school, and only a small part of what it takes to be a successful lawyer is taught or measured there. We’re not hiring for a faculty lounge – we’re hiring for our law firm’s success.
• These ineffective proxies cause unhelpful scarcity, hampering hiring of the talent we need. There are only a few top tier law schools, with few law students at the top of their class. By unnecessarily limiting our candidate pool, it becomes harder to find talented candidates. It’s also more difficult and expensive to compete against other firms recruiting in the same, small pool. This is especially true for candidate diversity. How many times have we heard “we’ve tried, but we simply can’t find diverse candidates.” It might work better to stop artificially shrinking the candidate pool by imposing arbitrary high class rank cut-offs and limiting the law schools where our firms look for recruits.
• These proxies contort firm culture in counterproductive ways. Imagine a law firm with an ingrained preference for top tier law schools and top class rank. When that law firm tries to diversify its hiring, it “relaxes” its “standards” to consider more Black and Brown law students, and perhaps law students with a different socio-economic background. See the problem? This firm now has a two-tier hiring approach that casts a pall over these new lawyers once hired, tainting how they are viewed by others at the firm. “DEI Hire” is a slur with a long history, and it poisons more than just its target. And ironically, such a firm may do the kind of careful consideration of the talents and experiences of such “exception” candidates that it should uniformly apply to all candidates, regardless of law school and class rank. But the seed of doubt has been planted, encouraging bias in treatment of such lawyers going forward, and corroding their sense of belonging.

If we’re serious about searching for true merit, we should be clear-eyed about the proxies we use to predict lawyer success. Something to ponder over some M&Ms.