Warship Cost Estimating Theater2

Introduction

After a brief recap of my last post, I begin with some observations about the cost of weapons systems made by third parties (not critics of DoD, they’re not happy about anything) and finish with an introduction to the Planning Fallacy. I placed all references at the bottom of the first post in the series, which can be found here.

Review of my last post

Since some time elapses between my posts (figuring out what you want to write as you go takes time for reflection), here is a recap of my last post:

• the purpose of this series is to share research and theories about biases involved in predictions about project durations (things I thought a lot about on active duty),

• warship cost estimating isn’t sexy, but it can be interesting (at least that’s my aim, but my friends frequently told me that I was off kilter),

• nothing in this “Warship Cost Estimating Theater” series is intended to criticize Navy and Department of Defense acquisition processes, recommend process changes, or question the integrity of anyone involved in existing processes, and

• I will point out things that amuse me, which shouldn’t be construed as criticism, at least not directly.

What do we know about defense acquisition program costs?

In 2006, the RAND Corporation published a study for the CNO about the rise in new construction ship costs (Arena, Blickstein, Younossi, and Grammich, 2006). The report focused on the cost of warships and not on the accuracy of cost and schedule projections. The key takeaways were:

• The Navy’s preference for larger and higher complexity ships was a significant factor in the cost growth of new ship designs (this was a motivation for LCS, but that’s a story for another day).

• The researchers found that increases in light ship displacement (without fuel, weapons, and stores) and power density (i.e., the ratio of electric power generation capacity to displacement) correlated most strongly with ship costs. Remember this because it will come up again.

⠀

Increases in displacement and power density lead to costlier and more complex ships, particularly power density, which is a measure of the ship’s number of mission-related systems. Every electronic system (communications, navigation, surveillance, and weapons) or Hull, Mechanical, and Electrical (HM&E) system controlled with computers has to be programmed, integrated with others, and interface with subordinate systems for hydraulics, alarms, displays, electrical distribution, pumps and valves.

All of these components and systems interact, sometimes in unforeseen ways, and have to be tested in the shipyard as part of the construction process. In addition to needing to satisfy shipyard functional tests, many systems have to be certified by Navy agencies external to the shipyard such as the CVN 78 class Advanced Weapons Elevators, one of 23 new technologies developed for the Ford-class carriers that delayed its first deployment. Issues that arise because of conflicts between design and certification can be significant and also cause delays. The details are interesting but beyond the scope of this blog.

The RAND study’s authors concluded with this observation:

Over the past four decades, the cost of U.S. Navy ships has grown faster than the rate of inflation. This real growth in costs means that ships are becoming more expensive and outstripping the Navy's ability to pay for them. Given current budget constraints, the Navy is unlikely to see an increase in its shipbuilding budget. Therefore, unless some way is found to get more out of a fixed shipbuilding budget, ship cost escalation means that the size of the Navy will inevitably shrink. (p. xiii)

This is worth reflection, particularly for those claiming that the U.S. Navy urgently needs more ships. I leave that as an exercise for the reader.

In a RAND report for the Air Force, “Is Weapon System Cost Growth Increasing?,” the authors concluded:

• Yes,

• The increased “complexity of technology, software density, [and] system integration complexity … make the estimating a total system’s development cost … an increasingly challenging endeavor” (p. xxi).

• ”Most DoD defense development programs involve much higher levels of new technology adaptation [than private sector infrastructure projects] and therefore result in inherently higher levels of cost and schedule [emphasis added] uncertainty” (Younossi, Arena, Leonard, Roll, Jr., Jain, and Sollinger, 2007, p. xxi)

⠀

In another report for the Air Force, RAND authors studied Air Force acquisition program data for programs that were complete or nearly so. They concluded that there was “a systematic bias toward underestimating the costs and substantial uncertainty in estimating the final cost of a weapon system.” Since Air Force and Navy systems follow similar DoD requirements, there is no reason to expect that Navy programs would be significantly different.

In a report for the Institute for Defense Analyses, McNicol and Wu observed that changes to DoD acquisition policies and processes in the previous four decades did not have statistically significant effects cost growth of major defense acquisition programs (McNicol & Wu, 2014). So much for reducing risk with more meetings.

The Planning Fallacy

I conclude this post with a brief introduction to the Planning Fallacy. I plan to describe its implications and how accounting for it can inform schedule and cost estimates of new warship designs in more detail in future posts.

In their research on decision making under uncertainty, Kahneman and Tversky noted that the judgment errors people made were frequently systematic, not random. That is, the people they studied erred in predictable ways. This suggested that the source of the errors was bias and not confusion or mere lack of information (1977).

The profession or level of expertise of the predictors had no influence on the errors in judgment Kahneman and Tversky observed and the predictions of experts were no better than those of laymen. They recommended that decisions of project durations should be “guided by a critical and reflective assessment of reality, rather than by our immediate impressions, however compelling these may be” (Kahneman & Tversky, 1977, p.ii). This sounds wise and simple (at least critical and reflective assessments of reality should sound wise), but things get complicated when politics, DoD acquisition policies, Congressional appropriations, egos, and cost-estimating theater are involved. What is cost-estimating theater? Examples include saying you care about costs, making lots of public pronouncements about someone else needing to improve, and complaints that shipbuilders aren’t “investing enough” to make ships less expensive. Talking about needing to do better is always easier than changing what you do.

Kahneman and Tversky focused on predictions for the time to complete projects and not their cost per se, but the Planning Fallacy applies to any type of prediction for which distributional data exists. These data are universally undervalued by the people responsible for making the predictions (1977). Distributional, or reference-class, data, is the set of outcomes that have been observed for similar projects of the same general type. Before running to your shield labeled, “My Project is Different,” bear with me.

In project management, managers are responsible for:

• the actions (what’s to be done),

• the suppliers (sometimes there aren’t many),

• the order of the actions (the sequence), and

• the arrangement in time of multiple sequences of actions (the schedule).

Typical constraints on the schedule are resource availability (people, their training, and their experience, tools, and material) and space (how big the workspace is, typically much smaller on submarines than aircraft carriers, and other factors that facilitate work like drydocks and cranes). Estimating the durations of individual sequences is nothing more than an educated guess if the work hasn’t been done before. It isn’t possible either to predict the interactions across sequences (these were called “cross-ties” in the distant past) or the availability of resources. Knowing what’s required for your project is easier than getting it. Your workforce and suppliers have their own performance issues.

The result of over-reliance on plans and sequence predictions for new projects is what Kahneman and Tversky called the Planning Fallacy, which produces completion forecasts that are unreasonably optimistic. According to Kahneman, the Planning Fallacy leads to plans and forecasts that:

• are close to best-case scenarios (when do those ever happen?), and

• cannot account for “unknown unknowns” (Kahneman, 2011).

Donald Rumsfeld is credited for using the term unknown unknowns (usually forgotten is all the abuse he received from non-project managers for using it), but he was unlikely its author. There is no way for anyone’s project plan to accommodate the infinite variety of possible events that could delay it, although lack of resources and getting complicated new systems to pass operational tests are repeat offenders. Experience project managers know this even if they don’t usually say it out loud for fear of being derided as “too pessimistic.” Project managers considered pessimistic don’t get repeat work.

Conclusion

The Planning Fallacy suggests two things about project duration predictions. First, it is unwise (and erroneous) to expect that a detailed knowledge of a project’s sequence, however well you believe that the contractor has been “listening,” the number of risk management meetings you have held, your good feelings about what people understand, or any other vague nonsense are going to trump past performance on similar projects. Not only is it unwise, but you are most vulnerable to ignoring the relevance of similar projects precisely because you have detailed knowledge or involvement with yours (Kahneman&Tversky, 1977). Second, as soon as you find yourself saying, “But my project is unique and special” (i.e., there are no similar projects for comparison), look in the mirror and say out loud, “We’ll do what’s never been done before super well” or “We’ll achieve the best performance in the past fifty years.” If you really believe that, you should be playing the lottery more.

In my next post, I will begin a review of countermeasures for the Planning Fallacy. There are no cures.