A Better Bar Exam—Look to Upper Canada?

July 25th, 2017 / By

 

Today, tens of thousands of aspiring lawyers across the United States sit for the bar exam in a ritual that should be designed to identify who has the ability to be a competent new lawyer. Yet a growing chorus of critics questions whether the current knowledge-focused exam is the best way to draw that line and protect the public. As Professor Deborah Merritt has noted, “On the one hand, the exam forces applicants to memorize hundreds of black-letter rules that they will never use in practice. On the other hand, the exam licenses lawyers who don’t know how to interview a client, compose an engagement letter, or negotiate with an adversary.”

For years, the response to critiques of the bar exam has been, in effect: “It’s not perfect, but it’s the best we can do if we want a psychometrically defensible exam.” The Law Society of Upper Canada (LSUC), which governs the law licensing process for the province of Ontario, developed a licensing exam that calls that defense into question.

Overview of Law Society of Upper Canada Licensing Exam

The LSUC uses a 7-hour multiple-choice test consisting of 220 to 240 multiple-choice questions to test a wide range of competencies. For barristers (the litigating branch of the profession), that includes ethical and professional responsibilities; knowledge of the law; establishing and maintaining the lawyer-client relationship; problem/issue Identification, analysis, and assessment; alternative dispute resolution; litigation process; and practice management issues. A 2004 report explains how the LSUC identified key competencies and developed a licensing test based upon them.

Unlike the US exams, the LSUC exam is open-book, so it tests the ability to find and process relevant information rather than the ability to memorize rules. Most important, it tests a wider range of lawyering competencies than US exams, and it does so in the context of how lawyers address real client problems rather than as abstract analytical problems.

Below, we discuss how these differences address many of the critiques of the current US bar exams and make the LSUC exam an effective test of new lawyer competence. We also provide sample questions from both the LSUC and the US exam.

Open-Book Exam

Like all bar licensing exams in the United States (with the New Hampshire Daniel Webster Scholars Program as the sole exception), the LSUC exam is a pencil-and-paper timed exam. However, unlike any United States exam, including the Uniform Bar Exam, the LSUC licensing exam is open book.

The LSUC gives all candidates online access to materials that address all competencies the exam tests and encourages candidates to bring those materials to the exam. To help them navigate the materials, candidates are urged to create and bring to the exam tabbing or color-coding systems, short summaries of selected topics, index cards, and other study aids.

Lawyering is an open-book profession. Indeed, it might be considered malpractice to answer a legal problem without checking sources! As we have previously noted, good lawyers “…know enough to ask the right questions, figure out how to approach the problem and research the law, or know enough to recognize that the question is outside of their expertise and should be referred to a lawyer more well-versed in that area of law.” Actually referring a problem to someone else isn’t a feasible choice in the context of the bar exam, of course, but accessing the relevant knowledge base is.

The open-book LSUC exam tests a key lawyering competency untested by the US exam—the ability to find the appropriate legal information—and it addresses a significant critique of the current U.S. exams: that they test memorization of legal rules, a skill unrelated to actual law practice.

Candidates for the bar in Canada no doubt pore over the written material to learn the specifics, just as US students do, but they are also able to rely on that material to remind them of the rules as they answer the questions, just as a lawyer would do.

Testing More Lawyering Competencies

Like all bar exams in the US, the LSUC exam assesses legal knowledge and analytical skills. However, unlike US bar exams, the LSUC exam also assesses competencies that relate to fundamental lawyering skills beyond the ability to analyze legal doctrine.

As Professor Merritt has noted, studies conducted by the National Conference of Bar Examiners [NCBE] and the Institute for the Advancement of the American Legal System confirm the gaps between the competencies new lawyers need and what the current US bar exams test, citing the absence of essential lawyering competencies such as interviewing principles; client communication; information gathering; case analysis and planning; alternative dispute resolution; negotiation; the litigation process; and practice management issues.

The NCBE has justified their absence by maintaining that such skills cannot be tested via multiple-choice questions. However, as illustrated below, the LSUC exam does just that, while also raising professional responsibility questions as part of the fact patterns testing those competencies.

Testing Competencies in Context of How Lawyers Use Information

The LSUC exam attempts to capture the daily work of lawyers. Rather than test knowledge of pure doctrine to predict a result as the US exams tend to do, the LSUC used Bloom’s taxonomy to develop questions that ask how knowledge of the law informs the proper representation of the client.

The LSUC questions seek information such as: what a client needs to know; how a lawyer would respond to a tribunal if asked “x”; where a lawyer would look to find the relevant information to determine the steps to be taken; and what issues a lawyer should research. That testing methodology replicates how lawyers use the law in practice much more effectively than do the US exams.

The LSUC exam format and content addresses a significant critique of US bar exams—that those exams ask questions that are unrelated to how lawyers use legal doctrine in practice and that the US exams fail to assess many of the key skills lawyers need.

Sample Questions from the LSUC and the MBE

Here is a sampling of LSUC questions that test for lawyering skills in a manner not addressed in US exams. These and other sample questions are available on the Law Society of Upper Canada’s website:

  1. Gertrude has come to Roberta, a lawyer, to draw up a power of attorney for personal care. Gertrude will be undergoing major surgery and wants to ensure that her wishes are fulfilled should anything go wrong. Gertrude’s husband is quite elderly and not in good health, so she may want her two adult daughters to be the attorneys. The religion of one of her daughters requires adherents to protect human life at all costs. Gertrude’s other daughter is struggling financially. What further information should Roberta obtain from Gertrude?
(a) The state of her daughters’ marriages.
(b) The state of Gertrude’s marriage.
(c) Gertrude’s personal care wishes.
(d) Gertrude’s health status.
  1. Tracy was charged with Assault Causing Bodily Harm. She has instructed her lawyer, Kurt, to get her the fastest jury trial date possible. The Crown has not requested a preliminary inquiry. Kurt does not believe that a preliminary inquiry is necessary because of the quality of the disclosure. How can Kurt get Tracy the fastest trial date?
(a) Waive Tracy’s right to a preliminary inquiry and set the trial date.
(b) Bring an 11(b) Application to force a quick jury trial date.
(c) Conduct the preliminary inquiry quickly and set down the jury trial.
(d) Elect on Tracy’s behalf trial by a Provincial Court Judge.
  1. Peyton, a real estate lawyer, is acting for a married couple, Lara and Chris, on the purchase of their first home. Lara’s mother will be lending the couple some money and would like to register a mortgage on title. Lara and Chris have asked Peyton to prepare and register the mortgage documentation. They are agreeable to Peyton acting for the three of them. Chris’ brother is also lending them money but Lara and Chris have asked Peyton not to tell Lara’s mother this fact. Should Peyton act?
(a) Yes, because the parties consented.
(b) No, because there is a conflict of interest.
(c) Yes, because the parties are related.
(d) No, because she should not act on both the purchase and the mortgage.
  1. Prior to the real estate closing, in which jurisdiction should the purchaser’s lawyer search executions?
(a) Where the seller previously resided.
(b) Where the seller’s real property is located.
(c) Where the seller’s personal property is located.
(d) Where the seller is moving.

[These questions test the applicant’s understanding of: the information a lawyer needs from the client or other sources, strategic and effective use of trial process, ethical responsibilities, and knowledge of the real property registration system, all in the service of proper representation of a client. Correct answers: c, a, b, b.]

Compare these questions to typical MBE questions, which focus on applying memorized elements of legal rules to arrive at a conclusion about which party likely prevails. [More available here.]

  1. A woman borrowed $800,000 from a bank and gave the bank a note for that amount secured by a mortgage on her farm. Several years later, at a time when the woman still owed the bank $750,000 on the mortgage loan, she sold the farm to a man for $900,000. The man paid the woman $150,000 in cash and specifically assumed the mortgage note. The bank received notice of this transaction and elected not to exercise the optional due-on-sale clause in the mortgage. Without informing the man, the bank later released the woman from any further personal liability on the note. After he had owned the farm for a number of years, the man defaulted on the loan. The bank properly accelerated the loan, and the farm was eventually sold at a foreclosure sale for $500,000. Because there was still $600,000 owing on the note, the bank sued the man for the $100,000 deficiency. Is the man liable to the bank for the deficiency?
(a) No, because the woman would have still been primarily liable for payment, but the bank had released her from personal liability.
(b) No, because the bank’s release of the woman from personal liability also released the man.
(c) Yes, because the bank’s release of the woman constituted a clogging of the equity of redemption.
(d) Yes, because the man’s personal liability on the note was not affected by the bank’s release of the woman.
  1. A man arranged to have custom-made wooden shutters installed on the windows of his home. The contractor who installed the shutters did so by drilling screws and brackets into the exterior window frames and the shutters. The man later agreed to sell the home to a buyer. The sales agreement did not mention the shutters, the buyer did not inquire about them, and the buyer did not conduct a walkthrough inspection of the home before the closing. The man conveyed the home to the buyer by warranty deed. After the sale closed, the buyer noticed that the shutters and brackets had been removed from the home and that the window frames had been repaired and repainted. The buyer demanded that the man return the shutters and pay the cost of reinstallation, claiming that the shutters had been conveyed to him with the sale of the home. When the man refused, the buyer sued. Is the buyer likely to prevail?
(a) No, because the sales agreement did not mention the shutters.
(b) No, because the window frames had been repaired and repainted after removal of the shutters.
(c) Yes, because the shutters had become fixtures.
(d) Yes, because the man gave the buyer a warranty deed and the absence of the shutters violated a covenant of the deed

[Correct answers: d, c]

We Can Build a Better Bar Exam

As illustrated above, the LSUC exam shows that it is possible to test a far wider range of competencies than those tested in US bar exams.

Does the LSUC exam address all of the flaws of US bar exams? No—one problem that persists for both the LSUC and US exams is the requirement for rapid answers (less than 2 minutes per question), which rewards an ability and practice not associated with effective lawyering.

Does the LSUC exam fully address experiential skills? No—LSUC also requires applicants to “article” (a kind of apprenticeship with a law firm) or participate in the Law Practice Program (a four-month training course and a four-month work placement).

But the exam does what the NCBE has told us cannot be done. It is a psychometrically valid exam that assesses skills far beyond the competencies tested on US bar exams: skills such as interviewing, negotiating, counseling, fact investigation, and client-centered advocacy. And its emphasis is on lawyering competencies—using doctrine in the context of client problems.

Eileen Kaufman is Professor of Law at the Touro College, Jacob D. Fuchsberg Law Center.

Andi Curcio is Professor of Law at the Georgia State University College of Law.

Carol Chomsky is Professor of Law at the University of Minnesota Law School.

, No Comments Yet

The High Cost of Not Lowering the Bar

June 28th, 2017 / By

Gilbert A. Holmes is Dean and Professor of Law at the University of La Verne College of Law

In July of 2016, graduates from ABA-approved law schools in California had a first-time General Bar Examination pass rate of 62 percent, and all bar takers in the state had a first-time pass rate of 56 percent. These numbers are down from previous years, sparking debate, discussion and deliberation about the reason for the decline and what can be done about it.

Deans of ABA-approved law schools in California have been calling for a lowering of the cut score that serves as the basis for grading of the exam. California has the second highest cut score in the country. It also has the lowest pass rate, even though researcher Roger Bolus reported to the State Bar of California that the state’s bar exam takers perform higher than the national average on the only portion of the exam that every state except Louisiana administers—the MBE.

The State Bar has responded to this call by engaging in a number of studies about the content and validity of the bar exam and the California grading system. (more…)

, No Comments Yet

Bar Exam Scores and Lawyer Discipline

June 3rd, 2017 / By

Robert Anderson and Derek Muller have posted a provocative paper, The High Cost of Lowering the Bar, in which they argue that “bar exam score is significantly related to likelihood of State Bar discipline throughout a lawyer’s career.” Thus, they warn, “lowering the bar examination passing score,” as several California law deans have advocated, “will likely increase the amount of malpractice, misconduct, and discipline among California lawyers.”

Anderson and Muller urge the state bar to collect more detailed data on the relationship between bar scores and lawyer discipline–and then to consider the possible impact on attorney misconduct if the Supreme Court lowers the passing score. “The data we have collected,” they conclude, “should raise serious concerns about the effect on consumers of lowering the passing score.”

What type of correlation did Anderson and Muller identify? Should it affect decisions about the passing score for the bar exam? Let’s take a closer look.

(more…)

, View Comments (5)

More on the Bar Exam: Correlation and Competence

May 31st, 2017 / By

Derek Muller has identified an intriguing study of alternative ways to assess bar applicants. In 1980, the California bar examiners worked with a research team to explore the desirability of testing a wider range of lawyering skills on the bar exam. The researchers designed a two-day supplement to the bar exam and invited all July test-takers to participate in the supplemental exercise. More than 4,000 test-takers volunteered and, using appropriate sampling methods, the researchers chose 500 to participate. A few volunteers were unable to complete the exercise due to illness, so the final sample included 485 bar examinees.

These examinees completed the supplemental exercises in August 1980, shortly after taking the regular July exam. For two days, the examinees interviewed clients, drafted discovery plans, prepared letters, wrote trial briefs, cross-examined witnesses, and made arguments to mock juries. Each day’s work involved 5-6 tasks focused on a single client matter. Professional actors played the role of clients, and the researchers developed elaborate protocols for scoring the exercises.

How did results on the supplemental exam compare to those on the conventional test?

(more…)

, No Comments Yet

Reflections of a Bar Exam Skeptic

May 26th, 2017 / By

Robert Anderson has posted a thoughtful comment on the bar exam in which he dubs me a “bar exam skeptic.” I accept the label with pride: I have been deeply skeptical of the bar exam for years. I first wrote about the exam in 2001, when the national pass rate for first-time takers was a relatively high 77% (see p. 23). My skepticism today, with a national pass rate of 69%, is no greater or smaller. As I wrote recently, it’s time to convene a National Task Force to examine our bar admissions process.

Who Cares About the Bar Exam?

As Professor Anderson rightly observes, decanal concerns about the bar exam have risen as pass rates have fallen. That’s human nature. The content and scoring of the bar exam are boring subjects for alumni gatherings, graduation speeches, or law review submissions. Few legal educators spontaneously write about setting cut scores, scaling essay questions, or equating test scores over time. The bar exam is like plumbing: most people take it for granted until something goes wrong.

But now the bar exam pipes are leaking and people are paying attention. The leak doesn’t mean we should patch things up just to revive pass rates; the bar exam should measure competence, not admit a predetermined number of lawyers. But now that people are paying attention, this is a good time to consider whether we’re using the right type of filter and piping in our rather antiquated system.

(more…)

, No Comments Yet

Our Broken Bar Exam

May 4th, 2017 / By

The bar exam is broken: it tests too much and too little. On the one hand, the exam forces applicants to memorize hundreds of black-letter rules that they will never use in practice. On the other hand, the exam licenses lawyers who don’t know how to interview a client, compose an engagement letter, or negotiate with an adversary.

This flawed exam puts clients at risk. It also subjects applicants to an expensive, stressful process that does little to improve their professional competence. The mismatch between the exam and practice, finally, raises troubling questions about the exam’s disproportionate racial impact. How can we defend a racial disparity if our exam does not properly track the knowledge, skills, and judgment that new lawyers use in practice?

We can’t. In the language of psychometricians, our bar exam lacks “validity.” We haven’t shown that the exam measures the quality (minimal competence to practice law) that we want to measure. On the contrary, growing evidence suggests that our exam is invalid: the knowledge and skills tested by the exam vary too greatly from the ones clients require from their lawyers.

We cannot ignore the bar exam’s invalidity any longer. Every legal educator should care about this issue, no matter how many of her students pass or fail the exam. The bar exam defines the baseline of our profession. If the exam tests the wrong things, we have a professional obligation to change it.

*   *   *

For the rest of this essay, please see aalsnews. I discuss the concept of exam validity, our lack of agreement on “minimal competence,” and how educators and practitioners could work together to solve these serious problems.

, View Comments (3)

Law School Deans Ask For Extension On Exploitation

January 18th, 2017 / By

Originally published on Above the Law.

Laptop in classic libraryMore than 90 law school deans have asked their accreditor to halt new standards that would hold schools accountable for very low bar passage rates.

Last October, the Council of the ABA Section of Legal Education & Admissions to the Bar approved two new standards to stop exploitative admissions and retention practices. At a time when demand for law school decreased significantly, a minority of law schools began admitting swaths of students who, after three or more years of legal education, were not adequately equipped to pass the bar exam.

Why would a law school choose to do this? To keep tuition dollars flowing.
(more…)

, View Comment (1)

Council Approves New Bar Passage Standard

October 21st, 2016 / By

The Council of the ABA’s Section of Legal Education and Admissions to the Bar has approved a hotly debated proposal to tighten the accreditation standard governing bar passage rates. When the new standard takes effect, schools will have to demonstrate that seventy-five percent of graduates who choose to take a bar exam pass that exam within two years.

Opponents of the standard argued that it might reduce racial and ethnic diversity in the legal profession. Council members, however, largely rejected that argument. Raymond Pierce, former dean of the North Carolina Central University School of Law, distinguished between programs that give students “an opportunity” and those offering “a false chance.”

For more, see this story.

, View Comment (1)

ABA Moves Forward on Revised Accreditation Standards

September 10th, 2016 / By

The ABA Section of Legal Education’s Standards Review and Data Policy Committee voted unanimously today to recommend that the Section’s Council approve revisions to Standards 501 and 316.

This comes on the heels of a multi-month notice and comment period, which saw a number of comments about the revisions.

The committee recommended that the revised standards be adopted as proposed.

By taking this action, the committee acknowledges that its primary responsibilities are protecting the public and students, not law schools.

, No Comments Yet

The True Cost of the Georgia Bar Exam Error

September 8th, 2016 / By

To many, late October signals nothing more than fall in full swing, pumpkins, or costumes. In late May, we look forward to the Memorial Day holiday and long weekends. Yet, the last weekend of every October and May, Georgia bar takers anxiously await exam results. Some stalk the postman. Most spend the day refreshing a webpage, hoping and praying their name appears on the public pass list.

The stages of grief—denial, anger, bargaining, depression, and acceptance—are experienced by one who fails a state bar exam. Imagine discovering that a family member is alive after grieving their death for ten months. This week, 90 Georgia bar takers—45 from July 2015 and 45 from February 2016—were informed that the thing they grieved was, in fact, alive. Though their names failed to appear on that very public pass list, they indeed passed the Georgia bar exam.
(more…)

, No Comments Yet

About Law School Cafe

Cafe Manager & Co-Moderator
Deborah J. Merritt

Cafe Designer & Co-Moderator
Kyle McEntee

ABA Journal Blawg 100 HonoreeLaw School Cafe is a resource for anyone interested in changes in legal education and the legal profession.

Around the Cafe

Subscribe

Enter your email address to receive notifications of new posts by email.

Categories

Recent Comments

Recent Posts

Monthly Archives

Participate

Have something you think our audience would like to hear about? Interested in writing one or more guest posts? Send an email to the cafe manager at merritt52@gmail.com. We are interested in publishing posts from practitioners, students, faculty, and industry professionals.

Past and Present Guests