What Predicts Law School Grades? UPDATED.

A new, well done study analyzes the quantitative impact of LSAT scores, college grades, college major, college quality, work experience and other facts on law school grades.  (Substantially updated and expanded on October 21, 2016). Finer details are provided below the break.

The Main Relationship Discerned

The study makes a good case that if maximizing the law school grades of admitted students is the goal that undergraduate grades should get more emphasis relative to LSAT scores in the law admissions process than they do today. Typical admissions indexes don't give undergraduate GPA enough weight. In their regression model:

Each 0.06 rise in undergraduate GPA is akin to one extra LSAT point, but above 3.4, the effect doubles, so each 0.03 rise in undergraduate GPA is akin to one extra LSAT point. Thus, the difference between average and weak undergraduate GPA is material (e.g., 3.0 vs. 3.3 is akin to five LSAT points), but not as powerful as the difference between good and elite undergraduate GPA (e.g., 3.5 vs. 3.8 is akin to 10 LSAT points).

The strong admissions bias for students with high LSAT scores and low undergraduate GPAs over comparable students with low LSAT scores and high undergraduate GPAs (the former are roughly twice as likely to be admitted as the latter) has no support in subsequent law school performance. But, these biases may be a necessary response to U.S. News and World Reports law school ranking incentives.

STEM and to a slightly lesser extent economics, finance and accounting majors (3.5 to 4 LSAT point equivalent effect), as well as applicants with 4 to 9 years of non-military work experience (a 6.5 LSAT point effect) should be preferred in the admissions process, while criminal or disciplinary history should be a more strong negative in the process than it already is now (a 7 LSAT point effect). An adjustment based upon the selectivity of the applicant's undergraduate school, which is particularly strong for less selective schools is also appropriate.

Troublingly, after controlling for all other factors in the study it found that "any self-identification as a person of color—African American, Latino/a, Asian American, or Native American—was a statistically significant negative predictor of both LGPA and 1L GPA . . . each category of person of color self-identification was akin to 9 to 9.5 LSAT points." This is equivalent to a law school GPA difference of about 0.15 grade points and affects 18% of incoming students. 

This result does not appear to involve any affirmative action effect (except that it is significantly stronger for academically weak African Americans who are in the bottom quarter of admitted law students measured by LSAT scores and undergraduate GPA relative to African Americans who are in the top 75% academically, who actually do better than other non-white, non-Hispanic students after controlling for all other factors), mostly disfavoring the "mismatch hypothesis."

The study is also quite unusual for statistics of this kind for Asian Americans to cluster with other non-white students in terms of academic underperformance (although despite below average law school grades Asian Americans are significantly less likely to be in the bottom 25% of their class, something not true for any other group of minority students holding all other factors equal).

In contrast, there were no statistically significant effects or only marginal ones, associated with gender, other amounts of work experience (except for a slight negative effect associated with prior work in the tech jobs), other college majors, graduate school attendance, military veteran status, or college leadership roles.

The 2% of students who were military veterans, and 5% who had significant criminal or disciplinary records were the most heterogeneous groups in the study.

The Limits Of Explanatory Power

The tools available to admissions officers at law schools are still not wonderful at explaining variance in law school grades.

The full model in the study predicts 26% of the variance in law school GPA and 28% of the variance in first year law school grades, and would predict slightly less if statistically insignificant variables were removed. A regression model with only LSAT scores and undergraduate GPA to first year law school grades explained 22% of the variance in a large LSDAS study and 15% in this study.

LSAT is stronger at predicting first-year grades (the correlation between 1L GPA and LSAT, and 1L GPA and UGPA, are 0.36 and 0.27, respectively); UGPA is slightly better at predicting cumulative grades (the correlation between LGPA and LSAT, and LGPA and UGPA, are 0.28 and 0.29, respectively)." Note that the amount of variance explains is the square of correlation, not the correlation itself. "LSAT and UGPA had a positive and modest correlation of 0.187. . . . in a study of 152 law schools between 2011 and 2012, finding the average correlations between LSAT and UGPA are close to zero and range from -0.45 to 0.24[.]

The best predictor of law school grades is the not entirely independent factor of first year law school grades. First year law school grades explain 68.8% of variance in overall law school grades.

If this discrepancy between the predictive power of admissions data and the predictive power of first year grades is typical, perhaps a sink or swim approach, while wasteful, is worth serious consideration.

The Two Typical Mid-Range Law Schools Sample. It tracked the performance of 1400 admitted law school students (complete samples) at the University of Colorado Law School and Case Western Reserve Law School in Ohio. 

Incidentally, law school grade point averages are inflated by 0.209 points as Case Western relative to the University of Colorado. But, this is mostly a curiosity. Big law firm recruiters tend to care as more about law school prestige and class rank than raw law school GPA (despite the fact that law school GPA without regard to law school prestige is one of the best predictors available at the time of hiring of career success as a lawyer after graduation, even though it almost never considered in hiring or promotion decisions at any time after a first real legal job).

* LSATs and law school grades have a consistent relationship at all LSAT score levels. The relationship between LSAT scores and incremental law school GPA is uniform across the range of LSAT scores.  Six points on the LSAT increases law school GPA by about 0.1 points consistently across the scoring range. 

For this reason, the authors describe effect sizes in terms of LSAT point equivalents. All effect sizes are after controlling for all other factors, except as noted.

LSAT scores are twice as accurate at predicting first year law school grades as overall law school grades despite the fact that first year law school grades make a one-third contribution to overall law school grades. The authors of the test surmise that this may be due to the fact that first year law school grades are generally based almost entirely on timed, closed book final exams, while second and third year law school grades are often based in part on untimed take home tests and research papers. The LSAT is better than undergraduate GPA at capturing performance on timed tests.

The Study Reveals No Obvious LSAT Score Threshold Effects - Possibly Because It Didn't Look At Meaningful Numbers Of Low Performing Law Students.

There is no sharp "threshold" LSAT score or undergraduate GPA or combination of the two, notwithstanding bar passage rate data for law schools that seems to track such a median LSAT score threshold. The relationship between law school graduates and LSAT scores is instead roughly linear.

However, since bar passage, unlike law school grades, are an all or nothing affair, a linear relationship between LSAT scores and law school grades could still imply a pretty dramatic threshold effect in bar exam passage. The lack of bar exam passage data in this study, which is otherwise excellent, is one of its biggest defects, although the authors have promised a follow up study looking at post-graduation results.

The study also doesn't examine law school dropout rates, but those are very low at mid-grade law schools like the University of Colorado and Case Western Reserve, and at more selective law schools, and as in selective colleges and universities dropping out of law school at these schools is often due to non-academic personal or financial factors, so it is likely that drop out rate statistics would not have been statistically significant.

The study also acknowledges that while it is valid for roughly the middle 80% of admitted law students nationally (and validated against less comprehensive college board studies of all law schools), it has insufficient statistical power to be probative of the top 10% of law students academically, and the bottom 10% of law students academically. As the study explains:

This article’s findings, though, may not reliably generalize to schools outside that range—roughly, schools ranked in the top 15 or bottom 30. The findings also may not reliably generalize to students, at any school, with LSAT scores at the extreme tails. As the charts in Appendix Figure 1 show, LSAT had a wide distribution with a high peak at the mode, but the far upper and lower ends featured long tails, and outliers in the 140s and 170s. Even though LSAT and UGPA were not negatively correlated overall, conventional wisdom and anecdotal examples indicate a negative correlation at such extremes; many schools offer admission to those with a 140s LSAT only with substantial other plusses, or to those with a 170s LSAT despite substantial other minuses. Thus, this article’s findings
may not reliably generalize to students with extreme scores at any law school, nor to schools with substantial numbers of such unusually high- or low-scoring students.

Since apparent threshold effects in bar exam passage and drop out rates appear to be highly concentrated at schools which admit a large share of students in the bottom 10% of law students nationally, these samples may not be adequate to capture these effects.

The "threshold" that I have previously estimated for having a reasonable chance of graduating from law school and passing the bar exam is an LSAT score of about 145. Only 1.46% of students (21 out of 1,437 students at two law schools over four consecutive years) in the sample in this study had LSAT scores that low and as the study explains above, those 2-3 students per year per school are probably highly atypical of students with those LSAT scores. Of course, even if this study did draw conclusions about these students those results would probably not be statistically significant because the sample size is too small.

* Undergraduate law school grades matter a lot, especially for students with above average grades. The relationship between undergraduate GPA and law school grades is non-linear with increasing returns to higher scores. The benefit per GPA point from a higher undergraduate GPA is half as strong for undergraduate GPAs under 3.4 (0.06 GPA points equal 1 LSAT point) as it is for undergraduate GPAs over 3.4 (0.03 GPA points equal 1 LSAT point).

A positive grade trend during one's undergraduate years (i.e. significantly better upper division grades than lower division grades in college), which was found in 18% of law students, is equivalent to a 2 LSAT point equivalent benefit, but only in students with no prior work history. But this effect is only statistically significant in first year law school grades and then only at the p=0..1 level. A positive undergraduate grade trend for students with no prior work history does, however, make a law student 34% less likely to be in the bottom quarter of his law school class.

* Undergraduate institution quality matters at little between good schools and a lot for weak schools. College quality was measured in the study by the average LSAT score of LSAT test takers at the law student's undergraduate school (which closely tracks U.S. News Rankings of those schools in a way that is easily comparable across the entire sample since the law school data assembly service adjunct to the firm that administers the LSAT keeps track of it and provides it to law schools).

College quality also has a non-linear relationship to law school grades that demonstrates diminishing returns. For schools with an average LSAT score of 152 or more, each 0.2 points of average college LSAT score translates into 1 LSAT point worth of law school GPA performance. For schools with an average LSAT score below 152, 1 point of average college LSAT score translates into 1 LSAT point worth of law school GPA performance.

Take the state of Colorado, the source of many Colorado law students: the flagship state college, the University of Colorado at Boulder, typically has a 156 LCM (depending on the year), while the other prominent state college, Colorado State University, typically has a 153 LCM; both draw students from across and outside the state. In contrast, other public colleges in Colorado have a mainly local, commuter draw: the University of Colorado campuses in Denver and Colorado Springs typically have 151 LCMs; Metro State University in Denver has a 149. The four-point discontinuity between 151 and 152 plausibly reflects that the three-point difference between the top state schools (with LCMs of 153 and 156) matters less than the difference between those two and the weaker local public colleges (with LCMs of 149–151). . . . a difference between solid and strong colleges matters less than a difference between weak and solid colleges that is plausibly marked by having a sub-152 LCM.

I would note also that the difference between law school bound students at these institutions is much smaller than the differences in academic ability between the entering classes of these colleges. The stronger quality effect at the low end of the scale may be adjusting for lower law student application rates at "lower quality" colleges, since the entire student body and not just the law school bound part of the student body affects the quality of the college experience for a student at these colleges.

While college quality matters, with a single adjustment for college quality, the same amount of undergraduate GPA difference at a school is the same across the board.

Put another way, there is grade inflation in average GPA which is particularly great at lower quality schools, but the standard deviation and shape of the grade curve is pretty uniform at all colleges that send students to law school.

* College majors in STEM, Economics, Finance and Accounting Help. Undergraduates who had STEM majors (science, technology, engineering, and math) who make up about 16% of law students, over perform relative to all other factors in first year law school grades (a 2 LSAT point advantage), and over perform even more strongly in overall law school grades (a 4 LSAT point advantage). But, they are atypical of STEM undergrads who are 75% male, while STEM undergraduate law students are about 50% male.

Undergraduate majors in economics, finance and accounting, who make up about 12% of law students, also over perform, but less strongly. The advantage in first year law school grades is about 1 LSAT point, and about 3.5 LSAT points for overall law school grades.

STEM majors are 71 percent more likely to be the top quarter of their law school class compared to economics, finance and accounting majors, who are 30 percent more likely to be in the top quarter of their law school class compared to other law students.

The authors attribute some of these effects to more rigorous curriculums (on average) and reduced grade inflation (on average) in these majors relative to other college majors. In particular, they hypothesize that undergraduates seeking the route of least resistance and the easiest classes with easiest As are unlikely to choose these majors. But, they note that the benefits of these majors is much smaller than the effects of a strong undergraduate GPA, so undergraduates are still well advised to chose the majors that they are best suited to personally.

An alternative possibility is that quantitative reasoning ability is more helpful in law school than is commonly understood.

All other undergraduate majors are irrelevant. Traditional pre-law majors are not an advantage and atypical undergraduate majors for law students (e.g. dance) are not a disadvantage.

* Graduate School Is Mostly Irrelevant. The 13% of law students with graduate degrees (overwhelmingly master's degrees of many different types) do not have law school grades that are statistically significantly different from other law students. But, they are 32% less likely to be in the bottom quarter of their law school class.

* Work Experience Is Helpful In The Right Amounts.  Four to nine years of work experience (a range reflected in the data, not an arbitrary bin size) at a non-military job prior to law school is equivalent to a 6.5 LSAT point benefit. About 8% of law students have this amount of work experience.

There is no statistically significant benefit to having less than four years (about 28% of law students), or ten or more years of work experience (about 2% of law students).

There is an additional 5 LSAT point benefit to having worked as a teacher (for any length of time) prior to going to law school (about 5% of law students), but it is only statistically significant at the p=0.1 level. The authors attribute this benefit to comfort in an educational environment and greater than average maturity.

There is a 3 LSAT point negative effect on first year law school grades to having worked in science and technology who make up about 9% of law students (usually overbalanced by the advantage of having had a STEM major as an undergraduate), but this ceases to be statistically significant for overall law school grades. But, students with science/technology work experience are 83 percent more likely to be in the bottom quarter of their classes, although they are not less likely to be in the top quarter of their class (and indeed are often more likely to be in the top quarter of their classes if they had a STEM major).

Another explanation for the disparity between STEM majors who go straight to law school, and those who work in tech sector jobs (usually for 1-3 years) before going to law school, is that those who work in the economically thriving tech sector and then leave after just a few years to go to law school, may do so because they failed badly in their tech sector jobs and may be relatively aimless and not self-aware and are following what they see as a route of least resistance with the highest likely future income. In contrast, STEM majors who go directly to law school are sufficient self-aware to know when they graduate from law school that the tech sector is not for them and have not previously failed in the world of work.

Otherwise, the type of work experience a law student has is irrelevant. In particular, having worked in a law related job (e.g. as a paralegal) or in public service do not provide any particular benefit relative to other kinds of work.

* Military Experience Is Not Helpful.  There is no statistically significant benefit in terms of law school grades to being a military veteran (the substantial benefit for work experience is almost exactly cancelled by a deduction for being a military veteran), a group that accounted for 2% of law students. But, military veterans do take an average 4 LSAT point equivalent hit in first year law school grades (not of work experience benefits). 

It is also worth noting that military veterans are more heterogeneous in law school performance (after adjusting for all other factors) than almost any other group of law students (except those with criminal or disciplinary records who are comparable). Students with military work experience are 209 percent more likely to be in the bottom quarter of their law school class, but are no less likely than other law students to be in the top quarter of their class.

Interestingly, presumably because the 34 person sample size was too small to further subdivide, the study did not distinguish between military officers, who generally attend college before their military service, and veterans who were not military officers, who generally attend college after their military service.

The authors hypothesize that the fact the military veterans do not receive the same work experience benefits that other students with comparable work experience do may be partially a proxy for the fact that they are often have poor or working class socioeconomic backgrounds (I've read elsewhere that the average law student has a family income well in excess of $100,000 per year, if I recall correctly, around $175,000 a year.) and notably the magnitude of the effect is similar to that for racial or ethnic minority students.

They also suggest that the military may favor people with hands on "kinetic" learning styles and other attitudes that are ill matched to the practice of law, and that non-stereotypical military veterans (e.g. soldiers who had an intelligence, paralegal or administrative occupational speciality involving desk work and abstract ideas) might perform better in law school than more typical military veterans.

The authors hypothesize that poor first year performance relative to total law school GPA by military veterans, STEM majors, economics, finance and accounting majors, and former science and technology employees reflects an adjustment period between law school and quite different work or student experiences prior to law school.

* Criminal History Is A Serious Negative. Despite the fact that law schools only admit students with less serious criminal histories, a serious criminal or disciplinary history (defined as more serious than underaged alcohol consumption, or drug possession strictly for personal use, mostly marijuana) which was present in 5% of law students was equivalent to a 7 point LSAT score reduction in its effect on law school grades.

But, students with a serious criminal or disciplinary history, like military veterans, have very heterogeneous law school performance relative to their predicted performance.  

* Race All Students Who Are Not Non-Hispanic White Significantly Underperform. Law students who are not non-Hispanic white (even Asian Americans who usually cluster with whites on academic measures) across the board have much lower law school grades than their academic credentials, work experience and criminal history would suggest. The effect size is statistically significant and roughly the same across the board. Minority students as a whole make up about 18% of all law students.

This is not meaningfully an "affirmative action" effect because it controls for LSAT scores, undergraduate grades, undergraduate grade trends, college major, graduate school completion, work experience, military veteran status, criminal history, and college leadership roles. Some of this effect could in theory be due to excessive college quality effects due to affirmative action, but that would not explain why Asian Americans who generally experience reverse affirmative action that disfavors their admission experience the same effect.

The persistence of these effects even after controlling for these factors also disfavors a "mismatch hypothesis" that being admitted to schools above your abilities hurts your performance once you arrive due to stiff competition. Also, notably, African American, Latino/a and Native American students are not more likely after controlling for their pre-law school academics and life experiences to be in the bottom quarter of their law school classes, beyond across the board lower average law school grades with a magnitude of about 0.15 GPA points. 

Any self-identification as a person of color—African American, Latino/a, Asian American, or Native American—was a statistically significant negative predictor of both LGPA and 1L GPA. . . .all but Native American are significant at the 1 percent level, and Native American is significant at the 5 percent level. However, even with a combined data set from two schools, the number of observations in the categories—African American (59), Latino/a (45), Asian American (142), and Native American (15)—is relatively low. A group of 15 is too small from which to draw conclusions, and even 45 is relatively low. Still, the magnitude of the racial disparity was substantial and relatively consistent: each category of person of color self-identification was akin to 9 to 9.5 LSAT points. . . .

We found that among African Americans (but not other people of color), having an index [of LSAT scores and undergraduate GPA] not in the bottom quarter more than halved the disparity . . . . Thus, controlling as carefully as possible for academic credentials lessens the disparity, but does not eliminate it.

The only exception to the unusual result that Asian Americans cluster with other minority students, rather than as well as or better than non-Hispanic white as seen in most statistics of academic performs is that Asian Americans are 62 percent less likely to be in the bottom quarter of their law school classes than other law students.

The authors are at a loss to explain this effect and inconclusively suggest that this may be an effect on unconscious discrimination in the classroom.

* Gender Is Mostly Irrelevant. Gender has no statistically significant effect on average law school grades. This is a significant change compared to twenty years ago.

While average law school grades between men and women are insignificantly different from each other, however, women are 28% more likely to be in the bottom quarter of their class than men, and aren't materially more likely to be in the top quarter of their class.

Lani Guinier noted two decades ago, from survey and academic performance data, that women, then a minority of law students, found law school a source of “alienat[ion]” and “distress”—and performed worse in law school despite credentials on par with those of men: [W]e find strong academic differences between graduating men and women. Despite identical entry-level credentials, this performance differential [is] ... maintained over ... [all] three years. By the end of their first year ..., men are three times more likely than women to be in the top 10% of their law school class.114 

114 Lani Guinier et al., Becoming Gentlemen: Women’s Experiences at One Ivy League Law School, 143 U. Penn. L. Rev. 1, 2--3 (1994) (finding the female minority at the University of Pennsylvania Law School experienced “alienat[ion]” and “distress,” based on academic performance data from 981 students and self-reported survey data from 366 students).

In the early 1970s, the percentage of law students who were women was in the low single digits. This increased steadily almost every year until the number of women reached almost half of law school students. Women make up 45% of this study's sample of two medium sized mid-rated law schools.

* College Leadership Is Irrelevant. There is no statistically significant relationship between holding significant leadership roles in college (for example, top officer positions in major organizations or top officer positions in many small clubs) who made up 8% of the sample, and law school grades.

* Splitters With High Grades Overachieve, Those With High LSAT Scores Underachieve. The study examined the admissions chances and law school performance "splitters" who either have a high LSAT and low undergraduate GPA, or have a low LSAT score and a high undergraduate GPA.

Both types of splitters had mean law school GPAs statistically indistinguishable from the average. But, splitters with high LSAT scores and low undergraduate GPA are admitted at about twice the rate of those with with low LSAT scores and high undergraduate GPAs. High LSAT splitters underperform relative to the combined index of the LSAT scores and undergraduate GPAs, while low LSAT splitter over perform relative to this index. 

The mean law school GPA for the entire sample was 3.18 with a standard deviation of 0.34 (range 2.03 to 3.99). The mean undergraduate GPA for the entire sample was 3.48 (middle 50% 3.35 to 3.62; range 2.0 to 4.11) to and the mean LSAT score for the entire sample was 159 (middle 50% 156-163; range 133-178).

Students in the bottom 25% of undergraduate GPA and the top 25% of LSAT scores (average college GPA 3.01, average LSAT score 165, mean law school GPA of 3.18) made up 8% of the sample.  

Students in the top 25% of undergraduate GPA (average college GPA 3.81, average LSAT score unknown due to clerical error in study but close to 154 which includes the 12.5th percentile, average law school GPA 3.17) made up 5.6% of the sample.

This irrationally high weighting of LSAT scores to undergraduate GPA is driven by the fact that about 90% of U.S. News and World Report ranking variation can be explained by LSAT scores.

On the other hand, the relative underperformance of those with high LSAT scores may be mostly due to the fact that they are admitted more or less indiscriminately, despite other faults, while those with low LSAT scores and high undergraduate GPAs receive more discerning examination.