You are currently browsing the archives for Deborah J. Merritt.

Deborah Merritt is the John Deaver Drinko/Baker & Hostetler Chair in Law at The Ohio State University's Moritz College of Law. She has received multiple teaching awards for her work in both clinical and podium courses. With Ric Simmons, she developed an "uncasebook" for teaching the basic evidence course. West Academic has adopted their template to create a series of texts that reduce the traditional focus on appellate opinions. Deborah writes frequently about changes in legal education and the legal profession.

Stephen Carter on the Bar Exam

July 26th, 2022 / By

Eminent Yale Professor Stephen Carter has penned a thoughtful critique of the bar exam. Professor Carter notes the exam’s similarities to the LSAT, which some law schools have abandoned as an admissions requirement. In addition to their shared affection for multiple choice questions, the LSAT and bar exam both constrain the diversity of our profession. Despite the bar exam’s disproportionate racial impact, Professor Carter notes, the exam has never been properly validated. Here, he cites a column I wrote in 2017 for the AALS Newsletter.

As I wrote then, state bar examiners and NCBE designed the bar exam around a definition of minimum competence that they “felt in their bones.” NCBE did not conduct a practice analysis of the knowledge and skills that new lawyers need until 2012. That analysis supported some of the doctrinal subjects that NCBE was testing, but not the depth of memorization required by the exam. The analysis also confirmed that skills like researching the law, fact gathering, negotiating, and interviewing were essential for law practice–all skills conspicuously absent from the bar exam.

NCBE conducted another practice analysis in 2019, which once again exposed numerous flaws in the exam. My own research, conducted with Logan Cornett and IAALS (the Institute for the Advancement of the American Legal System), reached a similar conclusion: the written bar exam tests both too much and too little. It restricts admission to the profession (especially of people of color) without adequately protecting the public.

NCBE is developing a new exam that will better serve the goals of licensing, but that exam won’t be ready until 2026. And it may still demand more memorization than new lawyers need while omitting critical skills like legal research. Lawyers don’t memorize the millions of state, local, and national rules that govern our society; they master threshold concepts and research techniques that allow them to find the rules they need. No matter how improved, NCBE’s bar exam is likely to remain an artificial barrier to entry into the legal profession.

How else can we license lawyers? Professor Carter suggests wider use of Wisconsin’s diploma privilege–licensing all graduates of ABA-accredited law schools. Here I part ways with him. If law schools taught law students all of the ways they need to think like a lawyer, I might agree. But most law schools persist in the illusion that 3 years of reading judicial opinions (or, for many students, 1-2 semesters of reading judicial opinions followed by 4-5 semesters of downloading case squibs and course outlines from Quimbee and other sources) teaches students to “think like lawyers.”

The traditional law school curriculum shies away from the more complex thinking required to gather facts related to legal principles, interview clients and witnesses, negotiate letter matters, and counsel clients. Law school classes teach students two-dimensional thinking, while law practice requires thinking in four dimensions.

Fortunately, it is possible to improve both legal education and licensing by adopting an experiential education path to licensing. New Hampshire adopted this approach through its Daniel Webster Scholars Program. Oregon’s Supreme Court has approved a similar path in principle, and a committee is fleshing out details. These pathways assure that future lawyers learn all of the knowledge and skills they need to protect clients; they also keep the final licensing decision in the hands of bar examiners, rather than law school professors.

How do these programs work? How do they achieve reliability and fairness in a feasible manner? I’ll address those issues in future posts. But for an overview, see this research guide that I coauthored with Logan Cornett.

, No Comments Yet

Public Comment on the Bar Exam

July 25th, 2022 / By

David Lat is hosting a “Notice and Comment on the Bar Exam” at Original Jurisdiction. Several commentators have offered thoughtful insights. Here’s mine:

One of the many problems with the bar exam is that it doesn’t test the knowledge and skills that new lawyers really need. The exam was designed based on “gut instincts” about that knowledge and skills but, as often is the case, gut instincts were wrong. The problem was compounded by well intentioned efforts to create a national exam, which has resulted in candidates memorizing a vast number of federal or “consensus” rules that they will never use in practice.

But now we have good evidence about the knowledge and skills that new lawyers actually use. NCBE’s recent practice analysis offers some insights, and the Building a Better Bar study (which I coauthored with Logan Cornett) offers more. NCBE is now building a better exam around those studies, but a written exam can’t capture many of the skills that are essential to lawyering–and it’s hard to capture the type of knowledge most new lawyers use in a uniform national exam.

States like Oregon, California, Minnesota, and Utah are considering much better alternatives, with Oregon in the lead. It is possible to assess lawyering knowledge and skills through either law school coursework (including clinics and other experiential work) or post-graduate supervised practice. For both of these pathways, bar examiners would make the final decision based on portfolios of work product and assessments from professors or others. And it is feasible to construct both of these pathways with sufficient reliability, fairness, and validity.

The benefits of change? Cheaper pathways to licensure for candidates, better protection of the public, and (most likely, given the stereotype threat that affects high stakes testing) a more diverse profession. What stands in the way? Outdated ideas about how lawyers think and work, legal education’s reluctance to embrace more experiential education, our profession’s reluctance to innovate, and good old fashioned protectionism (the bar exam may exclude more lawyers than these alternatives would).

It’s time to honor our avowed commitments to open the profession to all qualified candidates, protect the public, and increase diversity. The bar exam is not achieving those goals.

, No Comments Yet

This Year Is Still Different: An Outdated Bar Exam in Troubled Times

July 19th, 2022 / By

** This post is coauthored with Sara J. Berman, Marsha Griggs, and Carol Chomsky. All four of us are members of the Collaboratory on Legal Education and Licensing for Practice, a group of 10 scholars who have studied and written about the bar exam, licensing, and legal education for many years.**

Applicants for the July 2022 bar exam are buckling down for their final days of bar study. After two years of delays, remote testing, and other COVID-related changes, states have returned to traditional bar examination practices. For most applicants, this means two days of testing in large convention centers or hotel ballrooms. Following tradition, applicants will again answer questions from memory about a dozen or more doctrinal areas.

But this year’s examinees are different from those who preceded them. The pandemic overshadowed the entire law school career of 2022 graduates. Classes abruptly went online during their first year. Many received only pass/fail grades for their spring semester. That was essential relief for an upended semester, but the remedy deprived students of more nuanced information about their progress.

The pandemic continued to dog the class of 2022, limiting both work and externship opportunities. Many lost the chance to meet mentors, work in law offices, and develop confidence in their lawyering abilities. Second-year classes remained mostly online, escalating zoom fatigue and isolation. Even during their third year, when restrictions eased, extra-curricular activities and meetings were limited. Peers and professors hurried out of the room after class, reluctant to expose themselves to the latest COVID variant. Informal exchanges about the law, lawyering, and career prospects were limited for this class of aspiring attorneys.

And that’s not all. The class of 2022 experienced George Floyd’s murder at the end of their first year, a bloody attack on democracy and the Capitol during their second year, and a leaked opinion reversing Roe v. Wade during their third year. Whatever their personal beliefs about abortion, the leaked Dobbs opinion raised alarming questions about the Constitution, constitutional interpretation, and the future of other rights guaranteed by previous Courts—just as these students started studying Constitutional Law for the bar exam.

And then there were continued police shootings of unarmed Black people, attacks on Asian American women, Russia’s invasion of Ukraine, heartbreaking gun violence, and our ongoing failure to address planetary destruction. These are unsettling times for anyone committed to the rule of law. For law students still exploring their future as lawyers, the times weren’t just unsettling—they have been devastating. They may doubt both the rule of law and their own ability to affect the world around them.

Now these graduates must prepare for a difficult exam that they know bears little relationship to their practice as fledgling lawyers. Research by NCBE and others has confirmed this mismatch. A new exam may address some of these flaws, but that exam won’t be ready until 2026. Meanwhile, today’s graduates must recall hundreds of detailed rules from memory. They must also prepare to answer essay questions on conflicts of law, family law, secured transactions, and trusts and estates—all subjects that NCBE has decided need not be tested. And they will not have a chance to show their competence at negotiation, client counseling, and other skills that NCBE now acknowledges should be assessed.

As a profession, we have a responsibility to help today’s bar takers. Pandemic graduates carry a heavy load of mental distress. More than a third show symptoms of depression, and 11% have seriously considered suicide during the last year. Those burdens may impair their preparation for the bar exam and their performance on it. If they do, we can’t blame the graduates for the world that surrounds them. Nor can we blame the academic support faculty who are working double-time to help this group of graduates succeed.

No, we need to look to the profession and what we can all do to help. Several states are considering non-exam pathways to licensure. If an experiential education path had existed for current graduates, they might have built a strong sense of their lawyering efficacy during law school—while learning skills and reinforcing the doctrinal knowledge they will use in practice and. If supervised practice pathways existed, recent graduates could be demonstrating their knowledge and skills by assisting real clients and learning from supervisors this month, rather than by grinding through daily doses of multiple-choice practice questions.

We have confidence in this year’s bar applicants: confidence in their abilities, their grit, and their determination. But even in the best of times, less than three-quarters of graduates pass the bar exam on their first try. And the failure rates fall disproportionately on graduates of color, the same individuals who suffered greater physical and financial burdens from COVID; emotional stress from police killings and other manifestations of racism; and loss of important mentoring opportunities during their law school years.

This, we know, is not the best of times. Offer as much encouragement and support as you can to bar-takers this week. And get involved with activities in your state to reform our licensing system. Do it for both our graduates and the clients they will serve.

, No Comments Yet

Growth of the Law

January 1st, 2022 / By

How much has the body of legal rules grown over the last fifty years? Daniel Martin Katz, together with colleagues in the United States and Germany, offers some intriguing insights into that question. In a recent paper, Katz and colleagues estimate that the United States Code added about 322,600 new chapters, parts, or sections in just the 22 years stretching from 1998 to 2019. Those additions represent a 63% increase in the number of structural elements in the Code.

Growth in federal regulations was even more aggressive during that period. Chapters, parts, or sections of the Code of Federal Regulations increased from 1.4 million in 1998 to 2.7 million in 2019–an increase of 91%.

If federal law grew this much between 1998 and 2019, how much did it grow in the decades before 1998? It is unlikely that Congress and federal agencies were more active after 1998 than in the decades just before that time. And what about growth of statutes and regulations at the state and local level? Or the growth of legal rules generated by judicial decisions? How much more “law” is there today than there was 50 years ago?

I offer here a small complement to the research of Katz and others: a description of the growth in federal constitutional rules governing criminal procedure. At the end, I offer a few preliminary thoughts about why the substantial growth of legal rules matters.

Method

My method is embarrassingly humble compared to the sophisticated data analytics used by Katz and his colleagues. But measuring growth in the federal constitutional law of criminal procedure is much simpler than assessing growth in the United States Code or Code of Federal Regulations. Scholars need sophisticated methods to measure the latter growth; for my project, a much simpler method sufficed.

The language in the Fourth, Fifth, Sixth, Eighth, and Fourteenth Amendments to the United States Constitution has not changed during the last 50 years, so growth in this field consists of judicial interpretations. I limited my study to decisions rendered by the Supreme Court of the United States, which (in these fields) bind all state and federal actors. Decisions by lower courts may signal changes in the law, but those changes do not become nationally binding until accepted by the High Court.

Conversely, I considered every SCOTUS decision as a change of some magnitude in the law. The Supreme Court does not accept cases that merely apply existing law to new facts. Instead, the Court grants certiorari only in cases that raise a new legal issue and/or represent a conflict among the lower courts. I decided, therefore, to count SCOTUS decisions explicating the constitutional law of criminal procedure through 1971–and then to contrast that number with the number of decisions in the same field rendered in 1972 or later.

As a source, I used a recent outline of the federal constitutional law of criminal procedure, Criminal Procedure, by Paul Marcus and Melanie D. Wilson (20th ed. 2021). I chose that text because it offers a concise, yet complete, overview of the field; includes significant historical discussion; and uses a format that made counting SCOTUS decisions relatively easy.

My method, like most estimates, is imprecise. Marcus and Wilson might have omitted some key decisions from their outline–although based on my knowledge of the field, the outline is quite complete. Some older decisions may have been superseded by more recent citations, which would lead to undercounting of decisions from before 1972. Marcus and Wilson’s frequent discussion of historical context, however, helped guard against this. Some decisions, finally, may have been double counted if they appeared in more than one discussion. One reason I chose the Marcus and Wilson text, however, was their diligent use of “supra” to cite cases that had been discussed previously. Overall, the estimates below offer a reasonable picture of how the federal constitutional law of criminal procedure has expanded during the last 50 years.

Results

I counted 160 pre-1972 SCOTUS decisions on the federal constitutional law of criminal procedure, and 692 of those decisions from 1972 through 2021. That represents an enormous expansion of the constitutional rules in this field: More than 430%. And, since most of the pre-1972 rules still bind the courts, the corpus of Supreme Court law in this field has grown from about 160 decisions to about 852 decisions. Lawyers studying or practicing criminal law today must master more than five times as many constitutional rules and nuances as their forebears did in 1971.

How did this growth come about? Some contemporary fields of constitutional jurisprudence emerged only after 1971. The Supreme Court, for example, first held a death penalty statute unconstitutional in Furman v. Georgia (1972). The Court’s increasingly detailed exposition of when and how the death penalty may be imposed all occurred after that year.

Similarly, much of the constitutional law governing collateral attacks on criminal convictions developed after 1971. The Warren Court suggested a broad role for these attacks in Fay v. Noia (1963), but subsequent Courts have cut back on the use of habeas corpus through dozens of decisions creating a complex set of rules limiting these challenges.

Explosive growth, however, has occurred even in fields that were well established by 1971. By that year, the Supreme Court had laid the foundation for the modern law of search and seizure, holding that the Fourth Amendment protects reasonable expectations of privacy (Katz v. United States 1967); that an arrest must be based on probable cause (Beck v. Ohio 1964); that the protections provided by the Fourth Amendment bind the states (Wolf v. Colorado 1949); and that the exclusionary rule likewise applies to the states (Mapp v. Ohio 1961). The Court had also recognized the various exceptions to the warrant requirement that it still permits today. In all, Marcus and Wilson cite 36 decisions delimiting the constitutional law of search and seizure through the end of 1971.

Since that time, however, the Court has added at least 198 new opinions–an increase of 550%. Those opinions have generated an extraordinarily detailed law of the Fourth Amendment. Before 1972, for example, the Court had done little to specify what expectations of privacy are reasonable enough to elicit Fourth Amendment protection. Now we know that individuals lack a reasonable expectation of privacy in their handwriting (United States v. Mara 1973); the sound of their voices (United States v. Dionisio 1973); telephone numbers they dial (Smith v. Maryland 1979); their bank records (Fisher v. United States 1976); the color and composition of the paint on their cars (Cardwell v. Lewis 1974); odors that a dog can detect from their luggage (United States v. Place 1983) or automobiles (Illinois v. Cabales 2005); the contents of their privately owned fields (Oliver v. United States 1984) and barns (United States v. Dunn 1987); objects on their private property that are visible from a low-flying helicopter (Florida v. Riley 1989) or through high-powered cameras (Dow Chemical Co. v. United States 1986); and any garbage they leave by the curb (California v. Greenwood 1988). The Court has approved warrantless searches in these and other areas.

On the other hand, the Court has recognized a reasonable expectation of privacy (or other Fourth Amendment protection) for the feel of an individual’s luggage (Bond. v. United States 2014); the movement of a car on public highways when tracked by a GPS tracking device (United States v. Jones 2012); the location of a cell phone (Carpenter v. United States 2018); odors a dog might detect from the front porch of a home (Florida v. Jardines 2013); the heat emitted from a home when detected by a thermal imager (Kyllo v. United States 2001); and data stored on a cell phone (Riley v. California 2014).

These post-1971 decisions do not represent obvious applications of Katz and other existing Fourth Amendment principles. Indeed, many of these opinions were issued by a divided Court. Instead, these and other decisions represent a detailed codification of the Fourth Amendment, with specific rules that criminal law practitioners must know or be able to find.

Similar growth has occurred in other areas of constitutional criminal procedure, including the legal principles governing the Sixth Amendment right to counsel; the contours of the Fifth Amendment right against self-incrimination; and the prosecution’s obligation to disclose information to the defense. The constitutional law of criminal procedure is much more complex today than it was fifty years ago.

Implications

Why does growth in the law matter, whether with repect to the constitutional rules of criminal procedure or in any other area? For the individuals and organizations seeking to comply with the law, growth may promote clarity. Police today know that they may not squeeze luggage to search for drugs, but may use a dog to sniff for that contraband. Similarly, homeowners know that the police may photograph their property from low-flying helicopters, but may not use thermal imaging devices to probe further.

On the other hand, this clarity increases complexity. Non-lawyers today have little hope of understanding the scope of their constitutional protections, and even lawyers struggle to keep up with all of the Court’s rulings. Criminal law practice has splintered into specialties, as lawyers strive to master these complex fields. A lawyer specializing in white-collar defense would have to bone up on the Fourth Amendment rules governing drunk driving cases to defend the latter type of case. And a lawyer who handles plea bargains and trials would have to master a new field before handling a collateral attack on a conviction.

As a law professor, I am particularly interested in how this growth in the law affects the ways in which we teach, learn, and test legal knowledge. In 1971, a professor of criminal procedure could lead students through all of the Supreme Court’s major opinions. The class could study those opinions in depth, studying the evolution of legal principles and their possible application to novel situations.

Today, we try to do the same, but there is so much more law to cover. How does one trace the evolution of principles through so many cases? And, although there are always novel situations to discuss, there is so much law to learn as a predicate for those discussions.

Equally important, our traditional ways of teaching omit some of the most valuable tools we can impart to today’s students. No one can remember the details of a criminal procedure class for very long, so how do practicing attorneys organize, remember, access, or find that information? Doctrinal professors have started teaching students about reference books and research tools in their field, but we need to do more of that. Similarly, we should help students develop the type of “cheat sheets” that practicing lawyers use to remember key points and assess cases. Today, those intellectual skills are as important as case analysis and synthesis.

And what about testing? Should exams require students to remember fine details? Or to discuss fundamental principles? Should exams be open or closed book? Most of my law school exams were open book during the 1970’s, a practice that seems even more appropriate today given the growth of the law since that time. Yet the trend in law school exams seems to have moved in the opposite direction.

How, finally, has growth of law affected the bar exam? The Multistate Bar Exam debuted in February of 1972, almost exactly 50 years ago. At that time, the criminal procedure portion of the exam would have been limited to testing the fundamental principles outlined in 162 or so Supreme Court opinions. That’s a healthy swath of law, especially for a closed book test covering numerous other subjects, but the field today is more than five times larger.

Does lawyering competence require memorizing the detailed rules that characterize much of the law today? That is both impractical and counterproductive. Today’s lawyers, more than ever before, need to know how to find rules–not recall them. Our contemporary definition of lawyering competence should focus on knowledge of threshold concepts (much like the foundational principles that existed in 1971) and the skills needed to implement those concepts and serve clients.

NCBE is designing a “Next Generation” of the bar exam that it suggests will test doctrinal rules “less broadly and deeply within the subjects covered.” At the same time, the new exam will place “greater emphasis . . . on assessment of lawyering skills.” If implemented, those commitments will serve our profession well: They recognize the enormous growth of legal rules over the last fifty years, as well as the importance of lawyering skills in managing that growth and serving clients effectively.

, No Comments Yet

Does Racial Diversity “Yield Educational Benefits”?

August 16th, 2021 / By

The Supreme Court has upheld the constitutionality of race-conscious admissions programs in higher education–but only on the ground that racial diversity improves the quality of education. Supporters and opponents of affirmative action have both criticized this rationale. Opponents deride diversity as a euphemism that masks racial quotas. Supporters protest that the concept sidesteps the original rationale for affirmative action: to recognize and remediate the discrimination that people of color have suffered–and continue to suffer–in our society. As Melissa Murray has written, rosy hued images of “diversity” insist that “changes must benefit everyone–even as we compensate for past offenses that were strictly visited upon a few.”

I share this dissatisfaction with the diversity rationale. It seems like yet another attempt to ignore the racial discrimination of our past and present. Yet, since the courts seem wedded to this rationale, it is worth asking whether it holds water. Does racial diversity “yield educational benefits,” as Justice O’Connor maintained in Grutter? The question has taken on urgency as the Supreme Court ponders a petition for certiorari in a case challenging Harvard’s admissions processes.

Spurred by this context, Adam Chilton, Justin Driver, Jonathan Masur, and Kyle Rozema designed a test of the proposition that diversity programs yield educational benefits. They focused on top law reviews that have adopted diversity programs over the last 50 years and asked: Did law reviews that adopted these programs enjoy a rise in scholarly impact (as measured by citation counts) after they adopted these programs?

The short answer is “yes,” providing an important boost to claims that diversity enhances education–as well as to advocates of diversity programs on law reviews. Now let’s look at the study in more detail.

(more…) , No Comments Yet

Caste Revisited

August 13th, 2021 / By

I’ve written several times about the caste system in legal education: a hierarchy that favors professors who teach torts, contracts, and other legal “doctrine” over those who teach legal writing, clinics, and other legal “skills.” This favoritism includes higher pay, more job security, and greater respect. Many schools maintain third and fourth classes that rank even lower than the second class citizens of clinics and legal writing. Academic support professors, teaching fellows, contract faculty, adjuncts, librarians, and other staff members often occupy those lowest rungs of the academic hierarchy.

California Western Steps Up

I’m returning to this topic because several related items recently hit my inbox. First, I received a press release from the California Western School of Law announcing that it had adopted a unitary tenure track that “creates opportunities for its clinical, Legal Skills, and other skills professors who were hired as full-time faculty to achieve tenure, with the same faculty governance and voting rights that come with an existing tenure-stream faculty position.” Kudos!

The press release, however, leaves several open questions. Will pay be equalized for professors on this unitary tenure track? Or will some professors still be more equal than others? How much research will be required for professors to join this unitary tenure track? Will the currently tenured professors turn their noses up at the scholarly focus of their new colleagues? And what about professors who choose not to join the unitary tenure track? Will the school recognize their ongoing contributions through higher pay and respect?

I’m not trying to rain on California Western’s parade: they have taken a hard step that many other schools are still resisting. I hope they will also find answers to these remaining questions, which schools face whether or not they embrace a unitary tenure track. What type of distinctions are appropriate among employees in a single organization? How do we value different types of contributions to the overall enterprise? Are the answers different for an academic institution and a manufacturing plant?

(more…) , No Comments Yet

Peter Lederer: A Modest Proposal

August 11th, 2021 / By

Peter Lederer brought unflagging inspiration and insight to the legal profession. On Sunday evening he sent me a copy of his latest essay, asking if I would like to publish it as a guest post here. I responded, of course, with enthusiasm–but I’m not sure that Peter saw my response. We all learned on Monday that Peter died Sunday night. With great sadness for his death, but immense gratitude for his words, I offer here Peter’s guest post:

A Modest Proposal, by Peter Lederer

From Chief Justice Bridget Mary McCormack of the Michigan Supreme Court comes a wise concept: using the “moment of disruption” where the door to fixing intractable problems has suddenly opened. Such moments come once in a century if that often.

Astute observers of the legal landscape hold that the present system is broken. Legal education, licensure, the inability to produce “practice-ready” lawyers after seven full years of prohibitively expensive training, are all under attack.

It is true that laudable efforts to bring about reform are underway. There are brilliant studies and recommendations; noble experiments have started in several states; a few dozen law schools have nurtured (or at least permitted) the pursuit of innovative programs. But unfortunately, all this has not moved the needle much. Moreover, many who are most deeply involved in the reform efforts believe that it will be, at best, a gradual process. Were this not enough, there is an overarching problem. Despite the hundreds of billions spent annually on legal services, the vast majority of the world’s people do not have access to legal services.

(more…) , No Comments Yet

Reopening

August 8th, 2021 / By

I retired from full-time teaching at the end of July and have decided to reopen the Law School Cafe. No promises: retirement holds lots of tantalizing possibilities and I may not maintain posting. But for now, the cafe is open again. No masks or social distancing required. Make your own brew and pull up a chair.

No Comments Yet

COVID-19 and the Bar Exam

March 23rd, 2020 / By

The news about COVID-19 gets worse by the hour. People are dying. The virus is spreading. Health care workers lack protective gear. Businesses are closed. We are sheltered at home. In the midst of these life-or-death matters, it seems mundane to worry about the July 2020 bar exam.

But we need to plan for the July bar, just as we need to think about every other part of life affected by the pandemic. Jurisdictions almost certainly will not be able to administer the exam in its usual format–to hundreds of test-takers gathered in large rooms or arenas. What should they do instead?

Over the weekend, I joined with a team of researchers to propose alternatives for licensing the Class of 2020. Our short paper is available at
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3559060. We don’t have all the answers, but we think it is critical to start the conversation. Decisions about the July bar must be made during the next few weeks.

If you have comments or additional suggestions, please let us know. As we state at the end of the paper: These are unprecedented times and we must work together to meet them. Our society must first work to secure its health, but intense legal needs will follow soon thereafter. We can’t afford to leave the talented members of the Class of 2020 sitting on the sidelines as those needs erupt. It is time to call all hands on deck.

View Comment (1)

Ranking Academic Impact

February 17th, 2020 / By

Paul Heald and Ted Sichelman have published a new ranking of the top U.S. law schools by academic impact. Five distinguished scholars comment on their ranking in the same issue of Jurimetrics Journal in which the ranking appears. But neither the authors of this ranking nor their distinguished commentators notice a singular result: The Heald/Sichelman rankings include a law school that does not exist.

According to Heald and Sichelman, Oregon State ranks 53d among U.S. law schools for its SSRN downloads; 35th for its citations in the Hein database; and 46th in a combined metric. Oregon State, however, does not have a law school. The University of Oregon has a law school, but it appears separately in the Heald/Sichelman rankings. So Heald and Sichelman have not simply fumbled the name of Oregon’s only public law school.

Instead, it appears that my own law school (Ohio State) has been renamed Oregon State. I can’t be sure without seeing Heald and Sichelman’s underlying data; even the “open” database posted in Dropbox refers to the nonexistent Oregon State. But Ohio State, currently tied for 34th in the US News survey, seems conspicuously absent from the Heald/Sichelman ranking.

I’m sure that my deans will contact Heald and Sichelman to request a correction–assuming that Oregon State actually is Ohio State. Oregon State Law School’s administrators probably will not complain. They can’t celebrate either, of course, because they don’t exist. But apart from that correction, let’s ruminate on this error. What does it have to say about rankings?

Reliability

A mistake like this obviously raises doubts about the reliability of the Heald/Sichelman ranking. If an error of this magnitude exists, what other errors lurk in the data? Even if you like the Heald/Sichelman method, how do you know it was carried out faithfully?

Some errors plague any type of large quantitative study, but an error of this nature is unusual. One of the key rules of quantitative analysis is to step back from the data periodically to ask if the patterns make sense. Surprising results may represent genuine, novel insights–but they can also be signs of underlying errors.

Heald and Sichelman studied the correlations between their rankings and several other measures. Didn’t they notice that one school produced a missing value? And when discussing schools that had highly discrepant rankings, didn’t they notice that one school in their scheme did not appear at all in other ranking schemes?

It’s possible, of course, that Heald and Sichelman misnamed Ohio State throughout their database so that they compared Oregon State’s Heald/Sichelman rank with the same misnamed school’s US News rank. Or perhaps the error slipped in near the end when they or an assistant changed “Ohio” to “Oregon” in the article’s spreadsheets.

Quantitative researchers who have their hands deeply in the data, however, should catch errors like this. Even after Heald and Sichelman banish Oregon State from their ranking, I will retain doubts about the reliability of their data. And my doubts about the reliability of other rankings, no matter how “scientific,” have been aroused.

For What Purpose?

Heald and Sichelman’s error is troubling, but I am equally concerned about the failure of any of their readers to spot the mistake. How could five commentators, as well as numerous other readers and workshop participants, blithely skim over the nonexistent Oregon State? Even if they weren’t familiar with Oregon’s law schools, weren’t they surprised to see Oregon State ranked 35th for Hein citations? The three schools in that state currently appear as 83d, 104th, and in the unranked fourth tier of the US News survey. Shouldn’t someone have noticed the surprising strength of Oregon State’s faculty?

I suspect that no one noticed the presence of Oregon State because most faculty read rankings primarily to see where their own school ranks. That’s what I did: I was curious where my own faculty ranked and, when Ohio State was absent, I looked more closely. It was only then that I noticed a law school that doesn’t exist.

But if that is the use of these academic impact rankings, to give faculty comfort or angst about where their law school ranks, are these rankings worth producing? They require a great deal of work and number crunching, as Heald and Sichelman make clear. Even with their presumably careful work, a substantial mistake occurred. Is the pay-off (including mistakes) worth the effort?

More worrisome, I think these rankings will harm the legal profession and its clients. Legal educators are key stewards of the legal profession. We are the profession’s primary gatekeepers: Few people become lawyers without first earning our diplomas. We are also responsible for giving students the foundation they need to serve clients competently and ethically.

Rankings of academic impact almost certainly will incentivize schools to invest still more of their resources in faculty scholarship—which, in turn, will raise tuition, reduce student discounts, and/or divert money from preparing students for their essential professional roles.

Scholarship is part of our commitment to the profession, clients, and society, but only one part. Over the last 20 years, I have seen law schools shift increasing resources to scholarship, while reducing teaching loads and raising tuition rapaciously. We produced excellent scholarship before 2000–scholarship that created fields like critical race theory, law and economics, feminist theory, and social science analyses of law-related issues. There is much still to explore, but why does today’s scholarship demand so many more resources? And will rankings further accelerate that trend?

, View Comment (1)

About Law School Cafe

Cafe Manager & Co-Moderator
Deborah J. Merritt

Cafe Designer & Co-Moderator
Kyle McEntee

ABA Journal Blawg 100 HonoreeLaw School Cafe is a resource for anyone interested in changes in legal education and the legal profession.

Around the Cafe

Subscribe

Enter your email address to receive notifications of new posts by email.

Categories

Recent Comments

Recent Posts

Monthly Archives

Participate

Have something you think our audience would like to hear about? Interested in writing one or more guest posts? Send an email to the cafe manager at merritt52@gmail.com. We are interested in publishing posts from practitioners, students, faculty, and industry professionals.

Past and Present Guests