Following the Same Tack to Mediocrity

Posted on April 27, 2013 by noahwaisberg

The 2013 Am Law 100 revenue report is out, and it shows some firms doing well, others not. Profits per partner for the top 50 firms rose 8%. PPP fell by 3.3% for firms ranked 51–100. Some individual firms may have dropped because of practice area off-years, adding partners while keeping revenue constant, or other idiosyncratic factors. But many are down for more systematic reasons—they are consistently unable to convince clients to use or pay them. Interestingly, nearly all large firms follow the same strategy: attempting to provide high quality (though inefficiently delivered) bespoke services on an hourly rate basis. It is only working well for some.

Law schools display a similar pattern. Law schools generally exist to prepare their graduates to practice; graduate employment statistics are (imperfect) indicators of their success. Some law schools do very well at getting their graduates into high paying or otherwise desirable jobs. While they may be costly schools, graduates get jobs that allow them to repay their debts (or loan repayment assistance if they choose lower paying public service jobs). However, there aren’t many schools succeeding here. More typical are the horde of almost-equally-expensive schools with poor employed-at-graduation rates. Yet most law schools cost nearly the same, and teach the same material the same way. Here, again, what is working well for some schools is not working well for most.

Why are most non-leading law firms and law schools following the same tack to mediocrity? Is what’s right for the best firms and law schools right for others? In a less busy time in my life, I did a lot of sailboat racing. One popular strategy was to closely match, tack for tack, the leader in a race. This generally made sense for boats that were close to the leader. The close followers could all increase the likelihood of finishing near the top by matching moves. And, through better execution or a lucky break, perhaps win. But following the moves of other boats can be a non-winning strategy, especially for those not near the very front of the pack. Non-leaders sometimes need to try different things and take risks to have a chance of pulling ahead. To the extent non-leaders are systematically behind, they may not be suited to beating the leaders at their own game. That’s okay: underdogs often win. Just not by trying to do the same thing as favorites.

Law Firms

Nearly all Biglaw firms have essentially the same business model. A number appear to be not thriving—maybe doing fine, but not great. Apart from sector-specific (e.g., energy, patent litigation) and merger-driven gains, they are generally not growing. And they are all generally doing the same thing.* In contrast to the low growth general legal sector, Axiom and LPOs seem to be growing rapidly and thriving (though see Liam Brown’s persuasive post saying that not all LPOs are thriving). As Indiana Law’s Bill Henderson points out, LPOs are moving past being a labor market arbitrage play—they are actively trying to deliver better quality legal services more efficiently, including through well thought through processes, appropriate technology, and labor cost arbitrage.

Wachtell, Cravath, S&C, and a number of others manage to stay the course and do fine. There will always be a good market for exceptionally high quality, high priced legal advice.** But how many firms will get to continue to work in a price-insensitive way? Applying a top ten firm strategy at a top thirty firm is a recipe for pain.

Why are few, if any, of the Am Law 200 wholly devoted to delivering high value, lower cost legal services? Sure, there are firms who care about efficiency, that have strong knowledge management teams or others focussing on process improvement. And we know a number of individual law firm partners who really seem to care about efficiency. But it’s hard to find large firms working towards efficiency like many companies in other industries do. It’s hardly worth asking whether a car, plane, soap, food, computer, or wind turbine manufacturer cares about making their product more efficiently—of course they do. What about law firms? How many have this as a top priority? Axiom and other LPOs’ success and growth shows that there is large demand at a lower price point. By embracing process improvement, efficiency tools, cost arbitrage and more, firms can provide better quality services than they now do at lower prices, probably without even having to try very hard. Why aren’t many really trying this? I have a friend at a good quality and amazingly cheap (hourly rates range from $250–550) Am Law 200 firm. While they sell partly on price and seem generally pro-efficiency, I do not see them hammering this advantage, pushing all they can at efficiency, and trying to drive their value up even further. I don’t get it.

Law Schools

There have been a lot of recent articles on how many US law schools are in trouble: leaving students with high debt burdens and few employment prospects. Here’s an example from the New York Times:

Faced with profound and seemingly irreversible shifts, the legal profession is contemplating radical changes to its educational system, including cutting the curriculum, requiring far more on-the-ground training and licensing technicians who are not full lawyers.

The proposals are a result of numerous factors, including a sharp drop in law school applications, the outsourcing of research over the Internet, a glut of underemployed and indebted law school graduates and a high percentage of the legal needs of Americans going unmet.

Why does virtually every law school have the same strategy of teaching the same subjects the same way,*** for ever increasing costs? I get why it is sensible for my alma mater NYU Law to have raised its yearly tuition from $38,255 when I graduated in 2006 to $50,336 today (though these numbers still stun me). How many other law schools does this make sense for? Might it be possible to provide a good legal education for $25,000 per year? Couldn’t there be enough fat to cut to hit this price while still providing a high quality education? Doesn’t it seem like schools that accomplished this this might have a very appealing pitch to applicants? How many private law schools are trying?****

There is good news for law schools and law firms: since they have been run for so long without an eye to efficiency, there’s lots of room for improvement. And putting in the effort now may yield big gains down the road, especially against currently-thriving incumbents who aren’t under pressure to change. There appears to be vast demand for lower cost, high value legal services and legal education. Sellers of both would do well to try to meet it.

—————

* Note that many UK firms appear to be trying different things, such as DLA’s UK operation launching Riverview Law, and some US firms are trying innovative moves such as opening insourcing facilities in low cost locales or are otherwise trying different things (e.g., these).

** Putting aside for this post the question of whether the current typical Biglaw service delivery model gives the higest quality work product. Technology and/or outsourcing, for example, may give higher quality results on many junior Biglaw tasks. And incorporating expert systems should improve appropriate more complex work.

*** Will also exclude from this post the “teaching the same subjects the same way” part of the story—there are a number of law schools attempting interesting changes here, though they appear to be the exceptions.

**** Our suggestions are for schools that see their goal as being the best school they can be, or providing graduates with the best legal education possible (e.g., ”to maintain its position in the very first rank of American legal education“ (from a leading school)). Law schools that aim to maxamize revenues or profits, raising tuition to the maximum possible level is probably the right medium-term strategy, as long as they have enough applicants willing to pay. We also recognize that most law schools heavily discount their tuition for significant numbers of applicants.

(photo: http://www.flickr.com/photos/portofsandiego/5468840271/in/photostream/ )

How We Test DiligenceEngine’s Accuracy

Posted on February 12, 2013 by noahwaisberg

We say that DiligenceEngine finds 90% or more of the instances of nearly every contract provision it covers. How do we know this? Hint: it’s not an assumption based on our knowing we have good people with serious training who work really hard. We know our system’s accuracy through testing. This post describes how.

If interested in this subject it might be worthwhile to start with our earlier post explaining what we mean by “90% accurate” (short answer: recall).

How We Test Our Accuracy

The quick answer to how we test our accuracy is we measure how well our system finds provisions in real contracts. This may sound easy, but it is not—there are a number of ways to generate misleading accuracy results in this process. Read on if curious about the details.

In the course of building the DiligenceEngine due diligence contract review automation system, we reviewed and annotated a large and diverse pool of agreements. These contracts range from supply agreements, to employment contracts, to investment management agreements, to leases, to franchise agreements, to licenses, to many other types; while many are US-law-governed, a significant portion are not. All of these contracts were experienced-lawyer-word-for-word-reviewed, and a number were reviewed multiple times. We now have a large pool of annotated contracts, and test our accuracy by running the system on a big set of them which were not used to train the system. The resulting (recall) accuracy numbers are what we are referring to when we say our system finds 90% or more of nearly every provision covered.

How to Cheat at Accuracy Testing

One way to measure accuracy would be to test on the same documents used to build provision models. However, this is analogous to giving students the answers to a test beforehand. You can know that high scoring students are good at memorizing, but you cannot know if they’ve really learned the material. Computers are particularly good at memorizing, and thus you should never test on the training data to determine the accuracy of a learned model (unless the problem you are trying to evaluate is if a system can find already seen instances (which might be the case for an automated contract review system only intended to work on known documents like company forms)).

This requirement to test on “unseen” data is particularly difficult to meet for systems that use manual rules (i.e., human created rules, such as those built using Boolean search strings). If using a manual rules based system, the only way to obtain truly unbiased accuracy results is to keep testing documents secret from the people building the rules. This requires a great deal of discipline; it is very tempting to look at where rules are going wrong. When testing a machine learning built model, on the other hand, it is easier to make sure the computer does not improperly peek at the test questions!

Another potential pitfall can come through testing on a fixed set of testing data. It might be tempting to set aside a portion (e.g., 20%) of total training data to be used as testing data. Testing on a small and static testing set raises the risk of biasing models to perform well on the test set; final model accuracies may reflect accuracy on the test set and not reality. To avoid this, the test set should be varied across training data. The technical term for this technique is cross-fold validation.

A final thing to beware of is training data diversity. No clever accuracy testing technique can make up for training data that is itself not a good reflection of reality.

These observations apply to situations (like legal due diligence) where the agreements for review can be of unfamiliar form, and do not necessarily apply where provisions are known in advance. For example, if testing whether a system can successfully extract provisions from a form agreement, the form agreement itself might be used in training and would definitely be used in testing. While our system should perform well on form agreements, our goal is to help users find the provisions they seek in whatever agreements they happen to have at hand. This is a different problem that requires different testing.

All this said, the tests we care about most are whether our system finds provisions for our users. Try it out for yourself and see how it does!

(photo: http://www.flickr.com/photos/benchun/7140209563/ )

We’re Hiring a Clojure Web Developer

Posted on January 23, 2013 by diligenceengine

DiligenceEngine Inc. is a Toronto-based startup using machine learning to automate legal work. Our core tech finds information, but we know finding information is useless without a good way to use and display it. We’re looking for someone to help us make our data accessible by building out our Clojure-based web interface.

Knowing Clojure is a plus but not a requirement—we are happy to work with someone looking to learn. Our team is small, pragmatic, and inquisitive; we love learning new technologies and balance adoption with good analysis. As our primary web developer you’ll have both a lot of responsibility and a lot of freedom to make design decisions.

Web technology can be built better. If single-page web design driven by functional programming sounds interesting to you, get in touch!

Technologies to know include:

HTML, CSS, SASS,

Javascript, Coffeescript, jQuery,

Clojure + ClojureScript,

Reactive Programming,

Java, PostgreSQL, RabbitMQ.

Best practice web app security.

Tools:

Git, Linux.

Experience in visual design is also a plus.

This position starts immediately in the Toronto area, though we will consider those located elsewhere.

To apply, send an email to jobs@diligenceengine.com with your resume and links to previous work.

Note: We’re happy to report we have filled this position.

Legal Automation: Career Alternative for Laid-Off Law Firm Partners?

Posted on January 9, 2013 by noahwaisberg

The Wall Street Journal reports that layoffs have been hitting law firm partners, with more to come.

About 15% of roughly 120 firms surveyed by Wells Fargo Private Bank’s Legal Specialty Group intend to cut partners in the first quarter, continuing a three-year trend.

And 55% of the 113 managing partners and firm chairmen who responded to an American Lawyer magazine poll said they planned to ask between one and five partners to leave in the coming year. Though that proportion was roughly steady with the previous year, 5% intended to cut between 11 and 20 partners this year, up from 1.2%.

Non-rainmakers are at particular risk of losing their jobs. One laments:

“You’re only as secure as the amount of money you bring in,” says a partner who during the recession was asked to leave a large national law firm. He was let go from his subsequent firm last year.

It isn’t enough to be a good lawyer, he says. “The job is to make money for the firm.”

We have a career suggestion for laid-off partners who are highly skilled at legal work and less skilled at client development and management: find work in legal automation. Legal thinker Richard Susskind has argued that we will see more “legal knowledge engineers”. As a technically skilled practitioner, you could be perfect for this role. Companies like ours are engaged in teaching computers how to do legal work. Since the idea with automation is to get software to do the task right and then apply it repeatedly, people who can actually get the task right in the first place are needed. That doesn’t necessarily mean people with partner-level experience, but it could.

What type of legal automation work is out there?

There is a broad range of legal automation work being done. Since we’re most familiar with corporate law automation, I’ll keep the list to some areas within this. There also appear to be a number of companies working on improving legal research.

Automating form agreements (e.g., LegalZoom, Rocket Lawyer (both more consumer/small business targeted), Koncision (high-end drafting focussed)). This area is perfect for those with strong drafting skills. Draft clauses and agreements that will be used over and over. From what we read, some (but not all) of the companies in this area could use team members with more attention to document quality. Which seems like a great opportunity for highly skilled lawyers to step in and make an impact.
Creating drafting aides. Lawyers might typically draft by pulling from a few precedents. A few companies feel there’s a better way, through systems that, for example, pull relevant precedent clauses, build templates based off automatically reviewing multiple agreements (KMstandards), or integrating playbook information and providing enhanced blacklines (Baseline).
Building expert systems. Really know a complex area of law, or deeply understand a tricky statute? Use Neota Logic to build a system others can rely on to solve similar problems, accurately in less time.
Automating deal closing documents (e.g., signature page set creation). A number of senior associates and partners I worked with were obsessed with getting closings done right, and know that there are many opportunities for deals to go wrong at the end. If this is your thing, there is work being done. And more to do.
Automating due diligence/contract review. This is where I’m most experienced. Our software automatically finds user-specified contract provisions and puts them into summary charts. We spend a vast amount of time building provision models and reviewing lots of contracts to train our system and evaluate its performance. This involves lots of legal judgement, as well as working with technologists to translate lawyer thinking into computer models.

There is also a lot not yet automated. Consider areas you know, and think about what people spend time on, which tasks get done over and over, and where there’s room for improvement. This post gives characteristics of corporate law tasks we feel are ripe for automation.

Is this work challenging and interesting?

I find it very much so, and I love the work that goes in to improving our system. Stay tuned—I will write more on the legal automation parts of my job in a future post. For now, know that this work can be interesting, challenging and intellectually rewarding.

What work do lawyers actually do at legal automation companies?

Tasks will be project and individual specific. With us, I do many different things, including reviewing agreements and other tasks to improve our system, reviewing system results, helping set and improve our user interface, doing company legal work, sales, social media work, and much more. Only interested in some of these? For example, don’t think you’re great at selling? First off, don’t be so sure—we believe subject matter experts (i.e., people who have done and supervised lots of diligence) are the best people to sell our product. That said, we have no hard requirement that any team member do anything—there’s lots of work to go around. We highly value the legal judgement that goes into making our system better and better. It’s hard to convince people to trust software to do diligence when most experienced deal lawyers don’t trust their smart, well-educated and trained associates to do the task right (for good reason–people are imperfect at diligence in our experience). We think the way to earn trust is through giving high quality results. We have found high quality results have come to us through lots and lots of lawyer and technologist effort.

What about pay and benefits?

It’s not all good news. Your salary will likely be a lot lower. But you may be part of building something valuable. And hopefully that will someday pay off. As it is, your opportunity cost may be low. On the good side, you may find flexibility in how much you work. While most of our team members work a lot, our work is not as client- and deadline-driven as it was in my corporate law practice. So all-nighters and missed engagements are rare. And it’s our fault we work a lot—we decided to do it that way. Perhaps some laid-off partners are happy working 1,300 hour years. That may be fine with some legal automation companies, who could be happy to take that from a highly-skilled teammate. We, for example, have had a part-time employee. Virtual/telecomuting work arrangements, depending on the team, might also be acceptable.

One caveat to all of this: I left Biglaw as a midlevel associate, and perhaps what I find challenging would be easy for you. That said, you would not be the only ex-partner in legal automation.

Interested in learning more? I’m happy to talk—get in touch. And I suspect other legal automators would be happy to share their experiences too.

The Future Is Coming to Corporate Law Practice, Like It Or Not

Posted on December 31, 2012 by noahwaisberg

We see lots of changes coming to how large-scale corporate law is practiced. Technology will help (or take on) more and more of the process work that goes into getting deals done (e.g., contract drafting, signature page set creation, SEC research, due diligence). And more high volume work will be packaged off to lower cost, dedicated workers through in- and outsourcing. Lawyers should be excited about these changes. Practice specific technology and outsourcing can help lawyers produce higher quality work product in less time, spend more time on more interesting work, and allow firms that take advantage to make more money.

There’s a catch: some firms’ business models will have to change. If your firm’s business model is driven by junior lawyers billing lots of hours on work that can be well automated or outsourced, and you refuse to change with the times, you will be in trouble. It is really hard to compete with tech and low-cost labor in areas where each works well. Low-cost contract lawyers can do low-end legal work at the same high level as junior lawyers at Biglaw firms (if not better). Predictive coding software does e-discovery document review more accurately than humans in less time for much less money. Our software creates accurate due diligence contract review summary charts in seconds per page, where it takes a careful lawyer minutes per page. If your firm relies on making money off inefficiently doing tasks that can be better, faster and cheaper done otherwise, now is the time to figure out how to make money when this work goes elsewhere.

Happy New Year!

(photo: http://www.flickr.com/photos/dneuman/2629879722/ )

Force Majeure—What It Is, When & Why These Clauses Matter, And How We Can Help You Find Them Quickly

Posted on November 8, 2012 by noahwaisberg

Hurricane Sandy impacting the New York/New Jersey/Connecticut area serves as a good reminder of why force majeure clauses matter. Disasters occur regularly, whether natural or directly human-caused events like terrorist acts or war. Many agreements include language describing what happens to contractual obligations if a disaster or other unexpected event outside of the parties’ control occurs. These provisions are known as force majeure clauses. They rarely come up, but matter when they do.

What is a Force Majeure Clause?

A force majeure clause is a contractual provision that establishes what happens to performance under the contract if a “force majeure” event occurs. “Force majeure” events are events beyond the parties’ control. They are often but not always listed in a non-exclusive list in the force majeure clause itself. Force majeure events typically include: accidents, acts of war or terrorism, civil or military disturbances, and nuclear or natural catastrophes or acts of God. Strikes and work stoppages are also sometimes listed, typically with a caveat that parties are not required to resolve strikes. Parties are sometimes obligated to take efforts to avoid and remideate force majeure events (except labor-related ones). Economic occurrences, such as market crashes or price spikes, are sometimes explicitly excluded from being considered force majeure events. However, market crashes can occur as a result of disasters (e.g., post-September 11, 2001 stock market crash) and the underlying disaster could support a force majeure claim.

Note that force majeure clauses sometimes allow for contract termination in case of extended force majeure events.

Recent weeks have given several examples of force majeure events worldwide. Sandy caused companies such as Motiva to declare force majeure events at some facilities, different oil companies declared force majeure events related to operations in Nigeria and Angola, and a number of mining companies declared force majeure events related to labor disruptions in South Africa (e.g., Anglo American Platinum, Assmang).

Why You Might Need to Find Force Majeure Clauses In Contracts

Force majeure clauses can give an out for parties unable to meet their contractual obligations because of events beyond their control. Companies affected by force majeure events (or whose counterparties face force majeure events) should review their contracts to find out what the rules are in these specific situations. Force majeure clauses often require counterparty notifications (on the event of force majeure and its ending). And, as above, sometimes contracts are terminable if force majeure events last. It is important for companies that are experiencing force majeure events to find out what their agreeements actually say. Or even prepare in advance for unexpected events by keeping force majeure clause information in a contract management database.

How DiligenceEngine Can Help Quickly Locate Force Majeure Clauses Over a Large Set of Agreements

Force majeure events can occur unexpectedly and require a quick response. At moments where time is short, lawyers overextended, and answers needed, our software might be helpful to quickly determine the impact of a force majeure-type event. DiligenceEngine’s cloud-based software rapidly reads contracts for user-specified provisions and puts findings into summary charts. Our software reviews documents in seconds per page—even experienced lawyers are much slower. DiligenceEngine is cloud-based and requires no installation (though does require an internet connection, which Sandy made some of us realize is not necessarily easy to come by in a disaster). And DiligenceEngine is simple enough you should be able to use it without training.

If you are very interested in force majeure clauses, a number of examples from various agreement types are on the next page. Continue reading →

Did We Mention That DiligenceEngine Dramatically Speeds Up Legal Due Diligence Contract Review? (in case Sandy has anyone needing to catch-up)

Posted on November 1, 2012 by noahwaisberg

With Hurricane Sandy’s recent visit to the Northeastern United States knocking out power and transit systems, a lot of lawyers got an unplanned break from work this past week. While this may have been like a bad weather vacation for some (at least those who didn’t lose power or water (unlike me)), busy lawyers will have to pay for the time off with some serious catchup work. My guess is no one covered your deals while you were away. Work that should have been done Monday–Wednesday didn’t get done then and it didn’t go away. Deal deadlines aren’t necessarily going to get pushed out because of the storm.

We can’t help you get caught up on everything, but, to the extent you have a pile of due diligence contract review left undone because of the storm, our system can help. It takes a first pass at creating due diligence summary charts: quickly and accurately reading agreements for user specified provisions (e.g., assignment, change of control, amendment, license grant) and putting findings into summary charts. This saves associate time, and helps generate more accurate results in due diligence contract reviews in the process.

Perhaps you just need to get due diligence contract review done quickly and don’t have a hurricane to blame—a regular situation, in my Biglaw practice experience. Deals often need to get done in short time periods, and teams can have to grind out a lot of long nights to make them happen. (Putting aside how our software helps help catch errors made by over-tired people) DiligenceEngine can really speed up due diligence contract review. We’ve found it takes an experienced associate somewhere from one to five minutes per page to review a contract (if they are actually reading it). And creating and populating diligence summary charts takes time too; cutting and pasting verbatim language from contracts into summary charts doesn’t happen automatically (at least without our software)! DiligenceEngine reviews contracts in seconds per page. If you need to get a lot of contracts reviewed for due diligence, you really should check our software out. And, by the way, law firms (even ones that charge by hours billed) can make more money being more efficient.

While we’re on the subject of DiligenceEngine being fast, the system is also quick to get started on. It’s cloud based so there is nothing to install—all you need is an internet connection and web browser. And the system is intuitive enough to use without training (especially after our recent interface enhancements). Give it a try!

(photo: http://www.flickr.com/photos/shankbone/8139660357/in/photostream/ )

Follow “DiligenceEngine”