Does Test Prep Drive LSAT scores?

It’s been well-documented that the LSAT is a great indication of past performance, a solid indicator of law school performance, and a very poor judge of future legal success.

So what is the LSAT really testing anyway? We all know really smart people who didn’t do too well on the LSAT, and we all know incredibly dumb people who got a high score.

On the Huffington Post, Noah Baron argues that the LSAT is really testing one thing: whether or not you are wealthy enough to spend the time it takes to prepare for the exam.

From Above the Law.

The correlation between good test scores on standardized tests on one hand, and high socio-economic class on the other, is well established, and I won't repeat it.

Short test preparation classes don't work

The beneficial effect of test preparation courses, however, is dramatically oversold:

[T]he National Association for College Admission Counseling reported that test preparation courses have minimal effect on improving SAT scores, showing boosts of only about 10-20 points on average in mathematics and 5-10 points in critical reading. . . retaking the test in hopes of score improvement might not be the answer either. Kathleen Steinberg, a spokeswoman for the College Board, says students who take the SAT test a second time, on average, only "increase their scores by about 30 points."

The SAT is scored on a 200-800 scale, and the LSAT for law school admissions is very similar in what it measures and its design principles, although it is designed so that it is less discriminating at the low end of the range than the SAT as the success in several years of college of test takers and their aspirations to obtain three more years after higher education after their earn their B.A.'s that weeds out most of the low scoring part of the SAT test taking pool. Neither the SAT nor the LSAT measure much of the specific academic content taught in high school and college.

LSATs, SATs, IQ tests and other measures of academic success heavily overlap

One doesn't have to be a hard core adherent of Murray's controversial book, "The Bell Curve" that suggests that intelligence levels are purely genetic and strongly linked to race (the 19th century IQ theory that Above the Law snubs), to reach the much weaker conclusion that the SAT and LSAT measure very much the same thing that an IQ test does, and that the SAT, LSAT and IQ tests are all good measures of proficiency at the kind of academic tasks found in higher educational courses.

One also doesn't have to reach deep conclusions on the causes of differences in IQ test performance to acknowledge the overwhelming data that show that the children of the affluent generally score better on IQ, SAT and LSAT type tests than those of the less affluent, and that they before better on almost any measure of academic achievement.

Regardless of the "why" of the matter, the fact that the differences exist is uncontroversial.

Social class impacts on academic achievement are early and sustained

It is much harder to pin down precisely where these differences arise, but Noah Baron is simply wrong when he attributes the differences mostly to pricey test preparation courses.

Class based gaps in academic achievement and IQ type test performance are already immense by the third grade. The tendency of children to perform academically and on IQ type tests at levels similar to their parents and siblings actually gets stronger, rather than weaker, as one approaches the age at which the LSAT is taken.

There are many, repeatedly replicated studies that show a very significant hereditary component of IQ, which can safely be defined with a weak definition of tendency to perform well on traditional academic tasks, rather than broader and more nebulous measures.

Despite the evience for a strong hereditary (and probably genetic) component to IQ and because it measures the same thing, LSAT scores, I don't discount the benefits that class privileges may provide to those who are better off.

At the low end, affluence may mean the difference between being exposed to lead paint which clearly reduces IQ and not being so exposed. Similarly, an NPR report this month cited a study that showed that differences in day care quality were significantly corrolated to the disciplinary records of the children that experienced it when they were fifteen years old, a decade later. Breast feeding duration and exposure to pesticides during critical growth periods in life have also been linked to later childhood and adult mental functionality. In our economic system opportunties to reach the beneficial early childhood environments are linked to social class.

There is reason to think that our school system may impact the IQs at LSAT taking age of our youth along class lines. Our highly de facto segregated K-12 education system overwhelmingly breaks kids into schools with mostly poor kids, mostly middle kids and mostly affluent kids, in part via segregation by neighborhood, exacerbated by further segregation via school choice and private school options, and even in schools that are integrated at the school level, through segregation into different programs within schools (e.g. at Denver's East High School) notwithstanding good faith efforts to break down those barriers and to educate children from different socio-economic backgrounds together where practicable.

The lack of advanced academic content on the SAT and LSAT also systemically benefits those who had early privilege in life and then slacked off later in their educations.

Malcolm Gladwell, in his book "Outliers" (2008) (particularly in Chapters Eight and Nine) makes one of the strongest arguments in the literature that the income linked "achievement gap" in academic progress (which we observe after kids enter school almost always and almost everywhere in educational testing) is almost entirely attributable to that academic progress that high income kids make, but low and middle income kids do not, during summer vacations (see particularly page 257). He also makes the related case that academic success in Asian school systems and of children in programs like KIPP is almost entirely attributable to the fact that these kids spend more days and more hours in school than typical American kids do.

The example of Denver's acclaimed School of Science and Technology is an example that proves the rule. Unlike all other incoming middle schoolers in Denver, those doing to the Denver School of Science and Technology (a school that isn't attracting kids who hate school or has a high record of severe academic failure in elementary school) have a few weeks of summer school that they must attend.

The long and short of it is that while class privilege may very well lead to class based disparities in LSAT scores, that it has little or nothing to do with test preparation classes immediately before taking those tests. Instead, it flows from some combination of genetic edges in IQ at birth, and a lifetime of academic advantage over the two decades that follow.

Does The Mean The LSAT scores should be given a lot of weight?

The fact that the link between social class and LSAT performance (or any other statistically valid measure of academic achievement you'd like to devise) is well established does not imply that it is right to use highly correlated undergraduate grades and LSAT scores, both of which measure academic ability, to determine who should be admitted to law school.

While undergraduate college grades and LSAT scores are good at predicting academic performance in law school, and can even distinguish quite well between students who are likely to graduate from law school and pass the bar exam somewhere, and those who likely to drop out of law school or never successfully pass the bar exam, they are not terribly good predictors of success in the actual practice of law.

Academic ability is a talent with diminishing marginal returns in the practice of law. A certain amount of it is absolutely necessary to become a lawyer. More is helpful to success in the practice of law, particularly in more intellectually oriented specialties. But, a large share of the factors that predict success within a large law firm, and that predict economic success generally in the practice of law, have little or nothing to do with relative academic ability.

Moreover, as one reaches more elite academic institutions, where the floor for minimum academic ability is higher, the usefulness of academic ability as a predictor of post-graduation success in the practice of law becomes increasingly marginal. Graduate of Harvard or Michigan or Yale who are near the bottom of their law school classes academically do not do much worse in their careers than those who graduate which much higher class ranks, given the chance to do so. Law school graduates who had no hope of gaining admission to a "top ten" law school, but who manage to be hired by a big law firm anyway, tend to be more, rather than less successful than their better academically endowed peers.

The factors that distinguish those graduates of reasonably selective law schools who prosper in their subsequent careers (at least within big law firms) from those who do not, are not unknown, although they aren't widely known either. Studies have been done by management consulting firms, and those studies have shown, among other things, that people who have backgrounds of success as team players in non-academic aspects of their liives also perform better in the team player oriented context of large law firms.

Thus, there is a strong empirical basis to argue for a reform of the law school admissions process away from a very heavy emphasis on undergraduate grades and LSAT scores, which should be used instead to set a floor for admission and as a moderate plus factor beyond that floor. But, above a floor for admission sufficient to safely high enough ensure graduation from law school barring personal emergency and admission to the practice of law, most of the rest of the law school admission process should focus on those attributes distinct from academic ability which have been empirically shown to have the greatest impact on success in the practice of law, whatever those factors may be.

Would reform change the class bias in law school admissions?

Reform in the admissions process of selective law schools away from a numbers driven process that exclusively measures academic ability would change the mix of students at the most selective law schools, although these reforms would probably not change the mix of students who graduate from law school and are admitted to the practice of law very much at all.

The law schools with the least selective admissions standards already admit significant numbers of students with only dim chances of passing the bar exam. Students who previously were admitted to very selective, numbers oriented law schools who are denied admission under a new system that takes into account non-academic "X factors" that have been shown to be empirically valid in predicting practice of law success are still going to be academically strong enough to be admitted to law school somewhere. If the process isn't announced publicly enough, some of these students may overestimate their chance of admission and not apply to a wide enough array of schools, but most of them will wise up the second time around applying to law school and be admitted somewhere then.

In other words, academically ability will continue to be, and should be, especially important at the least selective law schools, where academic ability has the most marginal utility, but should be increasingly less important at the most selective law schools, where additional academic ability has the least marginal utility.

So there first answer to the question, "would reform change the class bias in law school admissions?", is no. Reform in the law school admissions process alone is unlikely the change the overall class composition of people who become lawyers or attend law school.

Also, while reform is very likely to change the composition of the students who attend highly selective law schools in a way that is not indifferent to social class, my intuition is that it would tend to increase rather than decrease the social class bias of the current system. The non-academic factors associated with success in the practice of law can be largely boiled down to work ethic and social grace.

Work ethic tends to favor social class climbers. Their level of comfort in life will come almost exclusively from their own professional success and they are the most aware of the downsides of not having money. In contrast, this disfavors academically able bohemians who may be independently wealthy. Research from studies popularized in books like "The Millionaire Next Door" make a strong case that people who have received (or anticipate receiving) "economic life support" from their parents or grandparents have poor work ethics. Thus, a revised admissions process would hurt the class of people whom we used to call "gentlemen" or "playboys" until the Ivy League admissions reforms of the 1960s made admissions to elite institutions of higher education more meritocratic and damped (without eliminating, of course) the formal and direct relevance of social class in the admissions process at the undergraduate level.

Social graces, on the other hand, is a factor that tends to favor those to whom the accepted conventions of upper middle class and upper class social interactions come naturally, rather than having to be acquired. Children learn the social graces that give them an edge in the real world from parents who have those social graces. But, those social graces aren't restricted to the social elites. The non-academic social talents associated with professional success in the practice of law are those associated with leadership and being a team player. Under a reformed system, quarterbacks and fraternity presidents and communmity organizers and people who had a first career as preachers are going to do better, while nerds and geeks are going to do worse. Leadership and team player social traits do track social class (those who lack them completely fail, often profoundly) but less monotonically and precisely than academic ability.

My suspicion is that law school admission reforms designed to favor leadership and teamwork will end up mostly favoring the children of the middle and upper classes, to the detriment of children of working class or upper middle class families.

Working class children whose hard work and natural ability permits them to achieve levels of academic success more often seen in children with much more affluent backgrounds will often have had to spend so much time on academics to catch up, that they will have had fewer spare time opportunities in which to show leadership or demonstrate teamwork. Their scarce financial resources will also often rule out these activities for them, as a result of an inability to pay for the sometimes high costs of team sports and the large scale activities that provide an opportunity to demonstrate leadership. In college, many of these young adults will have to spend so much time working to pay their way through college that again, opportunities to show teamwork and leadership will be lost.

Middle class students, in contrast, will come from families where the opportunity to participate in teamwork and leadership demonstrating activities was much greater, and together with working class students have the most to gain from a decreased emphasis on straight academics, where they and their working class peers are at a disadvantage visa-a-vis their more affluent peers. And, middle class students far outnumber working class students in the pool of people who are academically solid enough to graduate from law school and gain admission to the practice of law.

Many upper middle class students will have parents who rose to their economic place largely on the strength of their own academic ability, without being particularly socially graceful, and thus, may not be in as good a position to transmit those social skills to their children as their socio-economic betters. Also, since the marginal benefits of affluence in securing greater academic performance decline among the best performers, members of the true upper class will tend to have less of an academic edge over upper middle class students, than members of either class have academically over middle class and working class students on average.

To put this analysis together in a whole, one would expect law school admissions reform to increase the share of highly selective law school student bodies who are working or middle class, and who are "non-spoiled" members of the upper class, at the expense of members of the upper middle class and "spoiled" members of the upper class. But, those students who gain entrance to highly selective law schools will be mostly drawn from those who otherwise would have been admitted to less selective law schools, and those who are displaced from elite law schools will still mostly gain admission to less selective law schools. At the less selective end of law school admissions, and in the profession as a whole, reform to de-emphasize undergraduate grades and LSATs in favor of other factors would have little impact.

Another effect that this reform would have would be to significantly increase the numerical dominance of elite law school graduates (which is already immense) in elite law firms, because law schools would be more precisely selecting students for what the law firms desire. These reforms would also probably tend to reduce the rate at which elite law firms high graduates of elite law schools who swiftly wash out and leave those firms rather than advancing to partnership, and this reduced failure rate of new associates might also strengthen the williness of firms to be loyal to associate attorneys and visa versa.

Is there a way to reduce social class bias in higher education?

There is a way to reduce social class bias in higher education, but it has nothing to do with reducing the importance of the SATs or the LSATs.

The dominant reason that academically talented but less affluent students do not go to college, or go to college but do not continue to law school or other graduate studies, is financial pressure. Sometimes this financial pressure is direct and obvious. Less affluent students often simply cannot afford to go to college. Due to the structure of the financial aid system, the way this plays out is a little odd. Truly poor students have a relatively easy time getting a free ride or something close to it, and their families have little to lose financially by encouraging their pursuit of higher education since they have few financial assets in the first place and can safely assume that their children will provide for them better if they prosper.

The average college student and law school student is upper middle class or upper class, and often has saved for decades for higher education, so their college attendance is rarely seriously impeded by financial barriers.

The students most often discouraged from attending college or law school under the current system are those who are working class or middle class. More often than not, they qualify for some financial aid. But, their financial aid typically carries work requirements that impair students who are often not the very most academically talented students and often do not have the very best work habits, to perform well and stand out in college. And, the parents of working class and middle class students are expected to make significant financial sacrifices in order to help pay for the costs of their children's educations, which prospective students often unselfishly consider when making decisions regarding whether they should attend. Less affluent students are also often more accutely aware of the risk that they may attend college or law school and then struggle afterwards to pay off the student loans for which they are responsible.

And, since potentially college bound working class and middle class students are the cream of the crop of their social class peers academically, they often have better options than their peers in the market for entry level positions in the job market. So, they may not see themselves as having nothing to lose by going to college and possibly not completing it (a particularly common fate for working and middle class students for whom college completition rates are lower than for more affluent students).

Addressing this barrier to class mobility in the higher educational process doesn't have to be particularly expensive. The status quo is that a large share of students in colleges and universities pursuing four year degrees are quite affluent. The median family income of students at many public colleges that offer four year degrees is in the low six figures. At private colleges, family incomes tend to be higher. Not all students seeking four year degrees are affluent, but a large share are affluent enough that they will attend college even with little financial assistance, and a fairly price insensitive to tuition levels.

Also, most of those students who attend four year colleges who are less affluent already do receive some financial assistance.

To break down barriers to social class advancement in the higher education system, what is necessary to it reduce the expected family contributions and loan proportions in financial add packages for working class and middle class families, and to buffer these students from tuition increases.

The cost of doing this, and the atmosphere of doubt about the likelihood that a less affluent student will succeed in college, can be addressed by setting academic standards for students who will obtain this more generous financial aid package that are more rigorous and thus, are a better match to what is realistically necessary to have the academic ability to complete a four year college degree.

Thus, scholarships should be more generous, but should also require more strict merit standards for eligibility. Students who solid grades, no need for remedial classwork and decent test scores, should routinely receive the benefits of need blind admissions and generous scholarships that require only modest family contributions relative to their means.

Students with marginal grades, who need remedial classwork, and have marginal test scores shouldn't be denied access to higher educational opportunities all together, but should be given strong financial aid incentives to prove their ability to do college level work in a far less expensive to provide community college setting, rather than in a much more expensive setting where four year academic degrees are available, and shouldn't receive limitless financial assistance to pursue an educational course unlikely to benefit them or anyone else if they fail academically in that setting, as a stunning high percentage of community college students already do.

While actual admissions standards may be lower for students who are paying their own way with family support, the public interest in financing the education of students who are academically marginal at the time that they apply is not great, and limited public funds would be better spent to provide more generous packages for students with a proven greater likelihood of success.

Another way to pay for more generous financial aid packages for academically solid working class and middle class students, is to eliminate or reduce the completely non-need based, non-merit based tuition breaks given to in state students at public colleges. While equity may dictate that every in state student gets enough assistance to allow that student to attend a community college at a modest cost, there is no good reason to provide a larger price break in absolute dollars per student terms to in state students at other public institutions of higher education, without regard to financial need or academic merit, and financial need, rather than academic merit beyond that necessary to be comfortable that the student is very likely to graduate, should be the primary concern.

If states want to fund basic research at public institutions by college professors, that is good and well, but doing this via tuition breaks for in state students matched offset by institutional support for higher educational institutions isn't an optimal way to achieve this end. The part of the state's higher educational financial assistance package that truly benefits students needs to be better aimed at those who most need that assistance.

Cuts in financial aid to students at four year colleges who are academically marginal in order to encourage them to attend less expensive to provide community colleges, and reductions in the magnitude of in-state tuition breaks for affluent students at public colleges could go a long way towards freeing up funds for more generous financial aid programs that have smaller work and loan components by and large for those families who are already eligible for some form of financial aid. Some additional public funding might be necessary to get really adequate financial aid for working class and middle class students, but this would be only a fraction of the funding shifted to those students more in need of it.

Simply by lowering financial barriers to college attendance for academically solid students from working class and middle class backgrounds, the pool of college graduates from working class and middle class backgrounds who are in a position to consider attending law school will be much greater, which will significantly change the class makeup of future lawyers. And, if even some grant based financial aid is also provided to students academically solid enough to be likely to complete law school and pass the bar is provided at the law school level to student from less affluent families (the status quo is that almost all law school financial aid is in the form of student loans), the likelihood of social class slippage from college to law school will be reduced.

Put more simply, the best way to deal with lack of affluence as a barrier to law school admissions is to deal directly with it by providing financial support, not to overhaul what constitutes academic merit. There may be good reason to rethink what constitutes academic merit in law school admissions, but that class dimensions of that rethink are modest.