To a decision analyst, a decision is formally defined to be "a near irrevocable commitment of resources" [1]. Decision analysis is "a set of formal methods, semiformal techniques for interacting with people, and bits of lore and craft" [2]. Successful, well-executed decision analyses provide a rational basis for important individual decisions and solid defensible justifications for high stakes government and corporate decisions. Or, they may build consensus among diverse stakeholders providing a foundation for coordinated execution of the group's decision.

One of the "semiformal techniques" in decision analysis is the general process that most decision analysts follow in conducting a decision analytic project. Figure 1 provides one view of this process. The process is highly iterative and intended to allow for the discovery of new relevant information throughout. The software products that are identified in this survey are all designed to support a single or multiple parts of this process. Properly used, decision analysis software is a powerful tool for connecting the lore and craft that help decision analysts work effectively with people to the formal methods that ensure the results of the process are rational.

Figure 1: The decision analysis process.

The first two stages of the process — identifying the problem and developing objectives — are largely brainstorming and communication facilitation activities where the analyst helps the decision-maker or stakeholders to formulate their "objective function," to identify the key intermediate variables that should be considered, and to develop a set of alternative courses of action. At this stage, features like support for distributed (Web-based) or group elicitation and ease of modifying model structure are very important.

When decomposing and modeling the problem, the tool should fit the decision situation under consideration. Choosing the highest valued option among a set of possibilities is a fundamentally different model formulation than constructing a portfolio of alternatives. The software one selects should accommodate this difference. Another major consideration in decision analytic modeling is the representation of uncertainty. How the tool supports the elicitation, presentation and computation of probabilities should be an important consideration. For example, Figure 2 is a sample of a screen from one package that allows for the input of probability information using either a probability wheel or a matrix. This type of feature can be very valuable in expert elicitation because it can be helpful to use different views of the same information. It is absolutely essential that the analyst understand which multi-attribute modeling techniques are implemented and how their elicitation is supported. There are various multi criteria decision techniques. For example, multi-attribute utility theory (MAUT) and the analytic hierarchy process (AHP) have different underlying axioms and different philosophies about how decision models should formulated. Both approaches have strengths, weaknesses and limitations that deserve some research before the techniques are applied.

Figure 2: Probability elicitation screen.

Sensitivity analysis is an essential stage in all good analyses, not just decision analyses. Good decision analysis software will facilitate the sensitivity analysis and support the communication of important findings to decision-makers and stakeholders. Figure 3 is a snapshot of a couple of visualizations that are potentially available. These graphics are telling a powerful story. Alternative 4 is preferred by a large margin; we are sure about it; and this preference holds regardless of the weights we place on the attributes. In this case we are likely ready to execute the decision.

Figure 3: Sample sensitivity analysis visualizations.

The Survey


This is the eighth survey OR/MS Today has done over the last dozen years for decision analysis software. The questions in this version were similar to those in the previous surveys. The survey responses in the following tables are presented "as is," in that we conducted no supplemental verification. The responses to these questions are intended to provide potential users of the software with some insight into the relative functionality of the software, costs and applicable computer operating environments. This information is a nice basis for initial product screening. Analysts who are interested in purchasing decision analysis software should supplement this information with further research and perhaps some software testing to ensure the software provides functionality that is consistent with their needs.

The Results


Twenty different companies submitted a total of 38 packages in response to the survey. The vendor pool is international with many of the packages originating in Europe. The advertised software prices ranged from free Web-based downloadable distribution to thousands of dollars (or dollar equivalents), with many of the respondents indicating that potential users should call for price quotes and most offering educational pricing options. The market for decision analysis remains stable in terms of the number of vendors, with many respondents having participated in multiple surveys. Most of the vendors have new versions of their products identified as available since the last survey. Most of the vendors offer two or more packages designed for different types of analyses that can be helpful in developing a compatible "full-service" suite of decision analysis tools. Many vendors have developed interfaces between their products and with standard tools (e.g. Excel) that allow users to efficiently share data across different software packages.

The vast majority of the software continues to be written for use with the Windows operating system. Fifteen packages in the survey will run on the Mac OS. This is a significant increase over previous surveys. Twelve packages either require UNIX or have UNIX versions available. This tool is a significant increase over previous surveys. Nine of the packages support decentralized elicitation, with four identifying a Web interface as being available. This is also an increase from the last survey. Survey respondents indicate that 28 of the software packages offer functionality that supports structuring and brainstorming activities. Nineteen packages offer functionality that support group elicitation.

There are varying levels of functionality for eliciting and manipulating a model's structure. The relative utility of these functions such as copying segments, generating printouts, limits to numbers of levels in a value hierarchy and constraints on the total number of variables in a model will depend heavily on individual circumstances. In general, if one is considering using the software for interactive elicitation, simple, highly visual graphical user interfaces are preferred.

Two survey questions explored how uncertainty is addressed in the software. The first question asked if there are features that support the explicit representation of probabilities. Twenty-one of the respondents indicated that this feature exists. The second question asked if probabilistic dependencies are represented. Sixteen packages are identified as having this feature. How uncertainty is treated is worthy of exploration during product evaluation. Some of the packages use Influence Diagrams to implement this feature. That approach is very strong if the method for model construction is largely elicitation from domain experts. Decision trees, supported by some packages, have the same advantage of transparency, but for smaller models. Other packages use Bayesian networks as the underlying foundation, which could work very well if the model is to be learned from data or if the probability distributions might change over time. Another approach vendors offer is spreadsheet-based models and Monte Carlo techniques as the foundation. This can be desirable if the analyst is looking for maximum flexibility and integration with other tools and techniques.

Twenty-three of the products indicate they support portfolio decisions. I find two factors helpful in selecting the right tool for the job. First, how is the analysis to be conducted? If it is to be accomplished interactively with a group of managers or executives, then the packages with strong graphics and relatively simple scoring strategies are better suited to the job. If the analysis is being executed by a group of analysts, with intermittent interaction with executives and if there are dependencies among alternative, then I prefer the packages that allow for more complicated models and more powerful sensitivity analyses. The trade-off is that these more powerful features often require more time to execute and interpretation or repackaging before they are presented to executives.

Twenty-three packages reported the ability to address sequential decisions or decision strategies. In most of the packages the software allows for the representation of uncertain events or variables that might present themselves between decisions. This can help analysts to work in dynamic environments where there are multiple decisions that unfold over time. Sensitivity analyses like computing the value of information or value of control are especially useful techniques that are automatically supported by some packages and relatively easily formulated in others.

The comments and targeted industry sections of the survey provide some insight into the priorities and focus markets for the vendors. Most of the vendors migrate toward the markets where decision analytic techniques are currently being most popular, with most vendors either offering educational versions or in some cases focusing on that market.

Conclusions


Decision analysis projects can range from a few daylong sessions to complicated studies lasting many months. Even in the smallest case the cost of software is likely the smallest component of the cost of implementing decision analysis techniques, even if one purchases the more expensive packages listed in the survey. That said, there is not always a correlation between the cost and the performance of the software or suitability to the analysis problem. In fact in addition to the commercial market place, there is a growing availability of open source decision analytic software. In general the software is available from universities and research organizations. The costs associated with using the software, similar to the respondents to this survey range, start with free downloads. One good Web site for starting the investigation of these options is: www.cs.ubc.ca/~murphyk/Bayes/bnsoft.html.

When shopping for decision analysis software, focus on the potential tool's ability to fit the specific problem or class of problems you face. Evaluate the software in relation to the parts of the decision analysis process that are most relevant to you and the way that you most often interact with clients. If your goal is to add a package or two to your general tool kit, then a package or combination of packages that provide balanced support across all stages of the process is likely your best investment. If the types of models you wish to employ involve multiple stakeholders and multiple competing attributes, then tools that emphasize group support and value elicitation are worth exploring. Problems involving large uncertainties, diagnosis, complex interdependencies or risk analysis would benefit most from tools like influence diagrams, Bayesian networks or one of the Monte Carlo modeling tools. Whichever tool(s) you select, they should be intuitive to the user, explainable to the client and support easy iteration among the various stages of the decision analysis process.

The 2006 Decision Analysis Software Survey




Dan Maxwell (dmaxwell@innovativedecisions.com) is a senior principal analyst at Innovative Decisions, Inc. of Centreville, Va. In that capacity he leads numerous decision analytic research efforts and provides applied decision analysis support to many senior government and corporate leaders. Maxwell holds a Ph.D. in Information Technology from George Mason University and has taught decision analysis at the graduate level for multiple universities.

References

1. Matheson, J. & Howard, R., 1968, "An Introduction to Decisions Analysis," in "Readings on the Principles and Applications of Decision Analysis," Howard & Matheson, editors, Strategic Decisions Group, USA.
2. Von Winterfildt, D. and Edwards, W., 1986, "Decision Analysis and Behavioral Research," Cambridge University Press, New York.

Acknowledgments The author would like to thank Peter Horner and Patton McGinley for conducting the survey as well as their assistance in the preparation of this article.

Table of Contents

OR/MS Today Home Page


OR/MS Today copyright © 2006 by the Institute for Operations Research and the Management Sciences. All rights reserved.


Lionheart Publishing, Inc.
506 Roswell Rd., Suite 220, Marietta, GA 30060 USA
Phone: 770-431-0867 | Fax: 770-432-6969
E-mail: lpi@lionhrtpub.com
URL: http://www.lionhrtpub.com


Web Site © Copyright 2006 by Lionheart Publishing, Inc. All rights reserved.