Stephen Carter on the Bar Exam

July 26th, 2022 / By

Eminent Yale Professor Stephen Carter has penned a thoughtful critique of the bar exam. Professor Carter notes the exam’s similarities to the LSAT, which some law schools have abandoned as an admissions requirement. In addition to their shared affection for multiple choice questions, the LSAT and bar exam both constrain the diversity of our profession. Despite the bar exam’s disproportionate racial impact, Professor Carter notes, the exam has never been properly validated. Here, he cites a column I wrote in 2017 for the AALS Newsletter.

As I wrote then, state bar examiners and NCBE designed the bar exam around a definition of minimum competence that they “felt in their bones.” NCBE did not conduct a practice analysis of the knowledge and skills that new lawyers need until 2012. That analysis supported some of the doctrinal subjects that NCBE was testing, but not the depth of memorization required by the exam. The analysis also confirmed that skills like researching the law, fact gathering, negotiating, and interviewing were essential for law practice–all skills conspicuously absent from the bar exam.

NCBE conducted another practice analysis in 2019, which once again exposed numerous flaws in the exam. My own research, conducted with Logan Cornett and IAALS (the Institute for the Advancement of the American Legal System), reached a similar conclusion: the written bar exam tests both too much and too little. It restricts admission to the profession (especially of people of color) without adequately protecting the public.

NCBE is developing a new exam that will better serve the goals of licensing, but that exam won’t be ready until 2026. And it may still demand more memorization than new lawyers need while omitting critical skills like legal research. Lawyers don’t memorize the millions of state, local, and national rules that govern our society; they master threshold concepts and research techniques that allow them to find the rules they need. No matter how improved, NCBE’s bar exam is likely to remain an artificial barrier to entry into the legal profession.

How else can we license lawyers? Professor Carter suggests wider use of Wisconsin’s diploma privilege–licensing all graduates of ABA-accredited law schools. Here I part ways with him. If law schools taught law students all of the ways they need to think like a lawyer, I might agree. But most law schools persist in the illusion that 3 years of reading judicial opinions (or, for many students, 1-2 semesters of reading judicial opinions followed by 4-5 semesters of downloading case squibs and course outlines from Quimbee and other sources) teaches students to “think like lawyers.”

The traditional law school curriculum shies away from the more complex thinking required to gather facts related to legal principles, interview clients and witnesses, negotiate letter matters, and counsel clients. Law school classes teach students two-dimensional thinking, while law practice requires thinking in four dimensions.

Fortunately, it is possible to improve both legal education and licensing by adopting an experiential education path to licensing. New Hampshire adopted this approach through its Daniel Webster Scholars Program. Oregon’s Supreme Court has approved a similar path in principle, and a committee is fleshing out details. These pathways assure that future lawyers learn all of the knowledge and skills they need to protect clients; they also keep the final licensing decision in the hands of bar examiners, rather than law school professors.

How do these programs work? How do they achieve reliability and fairness in a feasible manner? I’ll address those issues in future posts. But for an overview, see this research guide that I coauthored with Logan Cornett.

, No Comments Yet

Public Comment on the Bar Exam

July 25th, 2022 / By

David Lat is hosting a “Notice and Comment on the Bar Exam” at Original Jurisdiction. Several commentators have offered thoughtful insights. Here’s mine:

One of the many problems with the bar exam is that it doesn’t test the knowledge and skills that new lawyers really need. The exam was designed based on “gut instincts” about that knowledge and skills but, as often is the case, gut instincts were wrong. The problem was compounded by well intentioned efforts to create a national exam, which has resulted in candidates memorizing a vast number of federal or “consensus” rules that they will never use in practice.

But now we have good evidence about the knowledge and skills that new lawyers actually use. NCBE’s recent practice analysis offers some insights, and the Building a Better Bar study (which I coauthored with Logan Cornett) offers more. NCBE is now building a better exam around those studies, but a written exam can’t capture many of the skills that are essential to lawyering–and it’s hard to capture the type of knowledge most new lawyers use in a uniform national exam.

States like Oregon, California, Minnesota, and Utah are considering much better alternatives, with Oregon in the lead. It is possible to assess lawyering knowledge and skills through either law school coursework (including clinics and other experiential work) or post-graduate supervised practice. For both of these pathways, bar examiners would make the final decision based on portfolios of work product and assessments from professors or others. And it is feasible to construct both of these pathways with sufficient reliability, fairness, and validity.

The benefits of change? Cheaper pathways to licensure for candidates, better protection of the public, and (most likely, given the stereotype threat that affects high stakes testing) a more diverse profession. What stands in the way? Outdated ideas about how lawyers think and work, legal education’s reluctance to embrace more experiential education, our profession’s reluctance to innovate, and good old fashioned protectionism (the bar exam may exclude more lawyers than these alternatives would).

It’s time to honor our avowed commitments to open the profession to all qualified candidates, protect the public, and increase diversity. The bar exam is not achieving those goals.

, No Comments Yet

Growth of the Law

January 1st, 2022 / By

How much has the body of legal rules grown over the last fifty years? Daniel Martin Katz, together with colleagues in the United States and Germany, offers some intriguing insights into that question. In a recent paper, Katz and colleagues estimate that the United States Code added about 322,600 new chapters, parts, or sections in just the 22 years stretching from 1998 to 2019. Those additions represent a 63% increase in the number of structural elements in the Code.

Growth in federal regulations was even more aggressive during that period. Chapters, parts, or sections of the Code of Federal Regulations increased from 1.4 million in 1998 to 2.7 million in 2019–an increase of 91%.

If federal law grew this much between 1998 and 2019, how much did it grow in the decades before 1998? It is unlikely that Congress and federal agencies were more active after 1998 than in the decades just before that time. And what about growth of statutes and regulations at the state and local level? Or the growth of legal rules generated by judicial decisions? How much more “law” is there today than there was 50 years ago?

I offer here a small complement to the research of Katz and others: a description of the growth in federal constitutional rules governing criminal procedure. At the end, I offer a few preliminary thoughts about why the substantial growth of legal rules matters.

Method

My method is embarrassingly humble compared to the sophisticated data analytics used by Katz and his colleagues. But measuring growth in the federal constitutional law of criminal procedure is much simpler than assessing growth in the United States Code or Code of Federal Regulations. Scholars need sophisticated methods to measure the latter growth; for my project, a much simpler method sufficed.

The language in the Fourth, Fifth, Sixth, Eighth, and Fourteenth Amendments to the United States Constitution has not changed during the last 50 years, so growth in this field consists of judicial interpretations. I limited my study to decisions rendered by the Supreme Court of the United States, which (in these fields) bind all state and federal actors. Decisions by lower courts may signal changes in the law, but those changes do not become nationally binding until accepted by the High Court.

Conversely, I considered every SCOTUS decision as a change of some magnitude in the law. The Supreme Court does not accept cases that merely apply existing law to new facts. Instead, the Court grants certiorari only in cases that raise a new legal issue and/or represent a conflict among the lower courts. I decided, therefore, to count SCOTUS decisions explicating the constitutional law of criminal procedure through 1971–and then to contrast that number with the number of decisions in the same field rendered in 1972 or later.

As a source, I used a recent outline of the federal constitutional law of criminal procedure, Criminal Procedure, by Paul Marcus and Melanie D. Wilson (20th ed. 2021). I chose that text because it offers a concise, yet complete, overview of the field; includes significant historical discussion; and uses a format that made counting SCOTUS decisions relatively easy.

My method, like most estimates, is imprecise. Marcus and Wilson might have omitted some key decisions from their outline–although based on my knowledge of the field, the outline is quite complete. Some older decisions may have been superseded by more recent citations, which would lead to undercounting of decisions from before 1972. Marcus and Wilson’s frequent discussion of historical context, however, helped guard against this. Some decisions, finally, may have been double counted if they appeared in more than one discussion. One reason I chose the Marcus and Wilson text, however, was their diligent use of “supra” to cite cases that had been discussed previously. Overall, the estimates below offer a reasonable picture of how the federal constitutional law of criminal procedure has expanded during the last 50 years.

Results

I counted 160 pre-1972 SCOTUS decisions on the federal constitutional law of criminal procedure, and 692 of those decisions from 1972 through 2021. That represents an enormous expansion of the constitutional rules in this field: More than 430%. And, since most of the pre-1972 rules still bind the courts, the corpus of Supreme Court law in this field has grown from about 160 decisions to about 852 decisions. Lawyers studying or practicing criminal law today must master more than five times as many constitutional rules and nuances as their forebears did in 1971.

How did this growth come about? Some contemporary fields of constitutional jurisprudence emerged only after 1971. The Supreme Court, for example, first held a death penalty statute unconstitutional in Furman v. Georgia (1972). The Court’s increasingly detailed exposition of when and how the death penalty may be imposed all occurred after that year.

Similarly, much of the constitutional law governing collateral attacks on criminal convictions developed after 1971. The Warren Court suggested a broad role for these attacks in Fay v. Noia (1963), but subsequent Courts have cut back on the use of habeas corpus through dozens of decisions creating a complex set of rules limiting these challenges.

Explosive growth, however, has occurred even in fields that were well established by 1971. By that year, the Supreme Court had laid the foundation for the modern law of search and seizure, holding that the Fourth Amendment protects reasonable expectations of privacy (Katz v. United States 1967); that an arrest must be based on probable cause (Beck v. Ohio 1964); that the protections provided by the Fourth Amendment bind the states (Wolf v. Colorado 1949); and that the exclusionary rule likewise applies to the states (Mapp v. Ohio 1961). The Court had also recognized the various exceptions to the warrant requirement that it still permits today. In all, Marcus and Wilson cite 36 decisions delimiting the constitutional law of search and seizure through the end of 1971.

Since that time, however, the Court has added at least 198 new opinions–an increase of 550%. Those opinions have generated an extraordinarily detailed law of the Fourth Amendment. Before 1972, for example, the Court had done little to specify what expectations of privacy are reasonable enough to elicit Fourth Amendment protection. Now we know that individuals lack a reasonable expectation of privacy in their handwriting (United States v. Mara 1973); the sound of their voices (United States v. Dionisio 1973); telephone numbers they dial (Smith v. Maryland 1979); their bank records (Fisher v. United States 1976); the color and composition of the paint on their cars (Cardwell v. Lewis 1974); odors that a dog can detect from their luggage (United States v. Place 1983) or automobiles (Illinois v. Cabales 2005); the contents of their privately owned fields (Oliver v. United States 1984) and barns (United States v. Dunn 1987); objects on their private property that are visible from a low-flying helicopter (Florida v. Riley 1989) or through high-powered cameras (Dow Chemical Co. v. United States 1986); and any garbage they leave by the curb (California v. Greenwood 1988). The Court has approved warrantless searches in these and other areas.

On the other hand, the Court has recognized a reasonable expectation of privacy (or other Fourth Amendment protection) for the feel of an individual’s luggage (Bond. v. United States 2014); the movement of a car on public highways when tracked by a GPS tracking device (United States v. Jones 2012); the location of a cell phone (Carpenter v. United States 2018); odors a dog might detect from the front porch of a home (Florida v. Jardines 2013); the heat emitted from a home when detected by a thermal imager (Kyllo v. United States 2001); and data stored on a cell phone (Riley v. California 2014).

These post-1971 decisions do not represent obvious applications of Katz and other existing Fourth Amendment principles. Indeed, many of these opinions were issued by a divided Court. Instead, these and other decisions represent a detailed codification of the Fourth Amendment, with specific rules that criminal law practitioners must know or be able to find.

Similar growth has occurred in other areas of constitutional criminal procedure, including the legal principles governing the Sixth Amendment right to counsel; the contours of the Fifth Amendment right against self-incrimination; and the prosecution’s obligation to disclose information to the defense. The constitutional law of criminal procedure is much more complex today than it was fifty years ago.

Implications

Why does growth in the law matter, whether with repect to the constitutional rules of criminal procedure or in any other area? For the individuals and organizations seeking to comply with the law, growth may promote clarity. Police today know that they may not squeeze luggage to search for drugs, but may use a dog to sniff for that contraband. Similarly, homeowners know that the police may photograph their property from low-flying helicopters, but may not use thermal imaging devices to probe further.

On the other hand, this clarity increases complexity. Non-lawyers today have little hope of understanding the scope of their constitutional protections, and even lawyers struggle to keep up with all of the Court’s rulings. Criminal law practice has splintered into specialties, as lawyers strive to master these complex fields. A lawyer specializing in white-collar defense would have to bone up on the Fourth Amendment rules governing drunk driving cases to defend the latter type of case. And a lawyer who handles plea bargains and trials would have to master a new field before handling a collateral attack on a conviction.

As a law professor, I am particularly interested in how this growth in the law affects the ways in which we teach, learn, and test legal knowledge. In 1971, a professor of criminal procedure could lead students through all of the Supreme Court’s major opinions. The class could study those opinions in depth, studying the evolution of legal principles and their possible application to novel situations.

Today, we try to do the same, but there is so much more law to cover. How does one trace the evolution of principles through so many cases? And, although there are always novel situations to discuss, there is so much law to learn as a predicate for those discussions.

Equally important, our traditional ways of teaching omit some of the most valuable tools we can impart to today’s students. No one can remember the details of a criminal procedure class for very long, so how do practicing attorneys organize, remember, access, or find that information? Doctrinal professors have started teaching students about reference books and research tools in their field, but we need to do more of that. Similarly, we should help students develop the type of “cheat sheets” that practicing lawyers use to remember key points and assess cases. Today, those intellectual skills are as important as case analysis and synthesis.

And what about testing? Should exams require students to remember fine details? Or to discuss fundamental principles? Should exams be open or closed book? Most of my law school exams were open book during the 1970’s, a practice that seems even more appropriate today given the growth of the law since that time. Yet the trend in law school exams seems to have moved in the opposite direction.

How, finally, has growth of law affected the bar exam? The Multistate Bar Exam debuted in February of 1972, almost exactly 50 years ago. At that time, the criminal procedure portion of the exam would have been limited to testing the fundamental principles outlined in 162 or so Supreme Court opinions. That’s a healthy swath of law, especially for a closed book test covering numerous other subjects, but the field today is more than five times larger.

Does lawyering competence require memorizing the detailed rules that characterize much of the law today? That is both impractical and counterproductive. Today’s lawyers, more than ever before, need to know how to find rules–not recall them. Our contemporary definition of lawyering competence should focus on knowledge of threshold concepts (much like the foundational principles that existed in 1971) and the skills needed to implement those concepts and serve clients.

NCBE is designing a “Next Generation” of the bar exam that it suggests will test doctrinal rules “less broadly and deeply within the subjects covered.” At the same time, the new exam will place “greater emphasis . . . on assessment of lawyering skills.” If implemented, those commitments will serve our profession well: They recognize the enormous growth of legal rules over the last fifty years, as well as the importance of lawyering skills in managing that growth and serving clients effectively.

, No Comments Yet

Peter Lederer: A Modest Proposal

August 11th, 2021 / By

Peter Lederer brought unflagging inspiration and insight to the legal profession. On Sunday evening he sent me a copy of his latest essay, asking if I would like to publish it as a guest post here. I responded, of course, with enthusiasm–but I’m not sure that Peter saw my response. We all learned on Monday that Peter died Sunday night. With great sadness for his death, but immense gratitude for his words, I offer here Peter’s guest post:

A Modest Proposal, by Peter Lederer

From Chief Justice Bridget Mary McCormack of the Michigan Supreme Court comes a wise concept: using the “moment of disruption” where the door to fixing intractable problems has suddenly opened. Such moments come once in a century if that often.

Astute observers of the legal landscape hold that the present system is broken. Legal education, licensure, the inability to produce “practice-ready” lawyers after seven full years of prohibitively expensive training, are all under attack.

It is true that laudable efforts to bring about reform are underway. There are brilliant studies and recommendations; noble experiments have started in several states; a few dozen law schools have nurtured (or at least permitted) the pursuit of innovative programs. But unfortunately, all this has not moved the needle much. Moreover, many who are most deeply involved in the reform efforts believe that it will be, at best, a gradual process. Were this not enough, there is an overarching problem. Despite the hundreds of billions spent annually on legal services, the vast majority of the world’s people do not have access to legal services.

(more…) , No Comments Yet

Ranking Academic Impact

February 17th, 2020 / By

Paul Heald and Ted Sichelman have published a new ranking of the top U.S. law schools by academic impact. Five distinguished scholars comment on their ranking in the same issue of Jurimetrics Journal in which the ranking appears. But neither the authors of this ranking nor their distinguished commentators notice a singular result: The Heald/Sichelman rankings include a law school that does not exist.

According to Heald and Sichelman, Oregon State ranks 53d among U.S. law schools for its SSRN downloads; 35th for its citations in the Hein database; and 46th in a combined metric. Oregon State, however, does not have a law school. The University of Oregon has a law school, but it appears separately in the Heald/Sichelman rankings. So Heald and Sichelman have not simply fumbled the name of Oregon’s only public law school.

Instead, it appears that my own law school (Ohio State) has been renamed Oregon State. I can’t be sure without seeing Heald and Sichelman’s underlying data; even the “open” database posted in Dropbox refers to the nonexistent Oregon State. But Ohio State, currently tied for 34th in the US News survey, seems conspicuously absent from the Heald/Sichelman ranking.

I’m sure that my deans will contact Heald and Sichelman to request a correction–assuming that Oregon State actually is Ohio State. Oregon State Law School’s administrators probably will not complain. They can’t celebrate either, of course, because they don’t exist. But apart from that correction, let’s ruminate on this error. What does it have to say about rankings?

Reliability

A mistake like this obviously raises doubts about the reliability of the Heald/Sichelman ranking. If an error of this magnitude exists, what other errors lurk in the data? Even if you like the Heald/Sichelman method, how do you know it was carried out faithfully?

Some errors plague any type of large quantitative study, but an error of this nature is unusual. One of the key rules of quantitative analysis is to step back from the data periodically to ask if the patterns make sense. Surprising results may represent genuine, novel insights–but they can also be signs of underlying errors.

Heald and Sichelman studied the correlations between their rankings and several other measures. Didn’t they notice that one school produced a missing value? And when discussing schools that had highly discrepant rankings, didn’t they notice that one school in their scheme did not appear at all in other ranking schemes?

It’s possible, of course, that Heald and Sichelman misnamed Ohio State throughout their database so that they compared Oregon State’s Heald/Sichelman rank with the same misnamed school’s US News rank. Or perhaps the error slipped in near the end when they or an assistant changed “Ohio” to “Oregon” in the article’s spreadsheets.

Quantitative researchers who have their hands deeply in the data, however, should catch errors like this. Even after Heald and Sichelman banish Oregon State from their ranking, I will retain doubts about the reliability of their data. And my doubts about the reliability of other rankings, no matter how “scientific,” have been aroused.

For What Purpose?

Heald and Sichelman’s error is troubling, but I am equally concerned about the failure of any of their readers to spot the mistake. How could five commentators, as well as numerous other readers and workshop participants, blithely skim over the nonexistent Oregon State? Even if they weren’t familiar with Oregon’s law schools, weren’t they surprised to see Oregon State ranked 35th for Hein citations? The three schools in that state currently appear as 83d, 104th, and in the unranked fourth tier of the US News survey. Shouldn’t someone have noticed the surprising strength of Oregon State’s faculty?

I suspect that no one noticed the presence of Oregon State because most faculty read rankings primarily to see where their own school ranks. That’s what I did: I was curious where my own faculty ranked and, when Ohio State was absent, I looked more closely. It was only then that I noticed a law school that doesn’t exist.

But if that is the use of these academic impact rankings, to give faculty comfort or angst about where their law school ranks, are these rankings worth producing? They require a great deal of work and number crunching, as Heald and Sichelman make clear. Even with their presumably careful work, a substantial mistake occurred. Is the pay-off (including mistakes) worth the effort?

More worrisome, I think these rankings will harm the legal profession and its clients. Legal educators are key stewards of the legal profession. We are the profession’s primary gatekeepers: Few people become lawyers without first earning our diplomas. We are also responsible for giving students the foundation they need to serve clients competently and ethically.

Rankings of academic impact almost certainly will incentivize schools to invest still more of their resources in faculty scholarship—which, in turn, will raise tuition, reduce student discounts, and/or divert money from preparing students for their essential professional roles.

Scholarship is part of our commitment to the profession, clients, and society, but only one part. Over the last 20 years, I have seen law schools shift increasing resources to scholarship, while reducing teaching loads and raising tuition rapaciously. We produced excellent scholarship before 2000–scholarship that created fields like critical race theory, law and economics, feminist theory, and social science analyses of law-related issues. There is much still to explore, but why does today’s scholarship demand so many more resources? And will rankings further accelerate that trend?

, View Comment (1)

Women Law Students: Still Not Equal

January 5th, 2020 / By

The Law School Survey of Student Engagement (LSSSE) released its annual report just before the holiday break. This year’s report, titled “The Cost of Women’s Success,” explores the gendered nature of law students’ experience.

I had the honor of contributing a Foreword to the LSSSE report. Summing up the report’s findings, I wrote:

Two law students, a woman and a man, sit side-by-side in class. From the podium, they look similar: both concentrate intently on the professor, take notes, and listen to classmates’ comments. But, as this LSSSE report reveals, their broader law school experiences likely diverge in meaningful ways.

The man is more likely to have a parent who was a lawyer; he is also more likely to have a parent who attended college. When the professor pauses for questions, the man is more likely to raise his hand. If the man and women are Latinx, the gender difference in classroom participation will be particularly stark.

After class, the man is more likely to exercise, read for pleasure, and pursue other leisure activities. The woman is more likely to attend a student organization meeting, email a professor, or speak to an advisor about her career plans.

At the end of the day, the woman is less likely than the man to get a full night’s sleep: half of LSSSE’s women respondents report that they average no more than five hours of sleep a night. And when the woman wakes to face another demanding day, she is less likely to find institutional support for her burdens.

Nor do the woman’s challenges end with graduation. She is more likely than the man to shoulder high debt as she enters the workplace. Those differences, like others noted in this report, sharpen at the intersection of gender and race. Sixteen percent of Latinas borrow more than $200,000 to attend law school, compared to 12% of Latinos and 4.3% of White men.

Despite these differences, women succeed in law school. Among LSSSE respondents, women’s reported grades exceeded those of men. That was true for women overall, as well as within each racial or ethnic group. As the report’s title suggests, however, women succeed at a cost: less sleep, fewer wellness activities, and more debt.

In this new year, law schools need to look more deeply at the gender differences that color our students’ experience. Those of us who stand at the podium (faculty and administrators) see the equal numbers of men and women sitting before us. We pride ourselves on that equity without probing below the surface. Feminist scholars, including some student authors, have continued to illuminate the gendered nature of legal education. Now LSSSE adds to that literature through the voices of more than 18,000 law students.

, View Comment (1)

About Law School Cafe

Cafe Manager & Co-Moderator
Deborah J. Merritt

Cafe Designer & Co-Moderator
Kyle McEntee

ABA Journal Blawg 100 HonoreeLaw School Cafe is a resource for anyone interested in changes in legal education and the legal profession.

Around the Cafe

Subscribe

Enter your email address to receive notifications of new posts by email.

Categories

Recent Comments

Recent Posts

Monthly Archives

Participate

Have something you think our audience would like to hear about? Interested in writing one or more guest posts? Send an email to the cafe manager at merritt52@gmail.com. We are interested in publishing posts from practitioners, students, faculty, and industry professionals.

Past and Present Guests