A few misconceptions about V&V …

• “Validation consists of comparing predictions to measurements.”

Wrong! While test-analysis correlation is an essential step of model validation, the later cannot be reduced to comparing predictions to measurements. The main outcome of a model validation study is to assess the prediction accuracy of a family of numerical simulations, especially away from configurations or settings that can be tested experimentally. This typically includes extrapolating the prediction accuracy away from the available test data, together with the quantification of its sources of uncertainty. Hence, validation cannot be reduced to comparing predictions to measurements.

• “A calibrated model is a validated model.”

Wrong! Calibration, also known as model updating or fitting, is a useful tool in model development. It nevertheless does not equate validation because a calibrated model provides no assessment of prediction accuracy away from those configurations or settings that have been tested experimentally. It has been recognized by many authors, for example, in the field of finite element model updating for structural dynamics simulations, that calibration can provide non-physical solutions. Calibration also leads to over-confidence. Both are detrimental to the ability to make predictions.

• “Models can be validated without data.”

Wrong! There is no validation without data because model validation must assess prediction accuracy relative to a physical reality. While code verification and calculation verification are concerned with the accuracy of the numerical implementation and convergence, respectively, validation activities focus on the adequacy of numerical simulations when applied to the description of reality, which requires experimental observations. We nevertheless recognize that the lack of test data can pose serious problems to model validation. Rigorously controlled expert elicitation techniques can provide information that is substituted to experimental testing in cases of severe lack of data and uncertainty.

• “Model validation is expensive.”

Not necessarily! Prior to engaging into a validation exercise, analysts should carefully define the scope and requirements of the study. This is achieved given the schedule and budget constraints, and depending on which decisions must be supported by the numerical simulation. For predictions to be credible, code and solution verification activities do not need to provide complete coverage of all aspects of the simulation. Likewise, validation testing can involve inexpensive, small-scale experiments that validate unit-level aspects of the physics. The bottom-line is that the goals of validation should be agreed upon by all stake-holders according to the level of credibility required for a specific application.

•  “Quantifying all sources of uncertainty is nearly impossible.”

Wrong! The aim of Uncertainty Quantification (UQ) is not necessarily to characterize the probability density functions for all predictions of the numerical simulation. While it is true that a fully probabilistic UQ can be nearly impossible, if not prohibitively expensive, other techniques can be brought to bear to investigate the effect of uncertainty on predictions and even reduce it. They include sampling-based methods for the inverse propagation of uncertainty, statistical effect screening, and variance reduction techniques. The quantification of sources of experimental and modeling uncertainty can involve activities of various nature, such as the elicitation of expert opinion, unit-level physical experiments, or the analysis of phenomenological models. In cases of severe lack-of-knowledge, non-probabilistic models can be developed that indicate the effect of uncertainty on predictions, which may be sufficient to demonstrate the credibility of numerical simulations and aid the decision-making process.

• “Analysts should be responsible for the validation of their models.”

Wrong! Analysts are not the sole stake-holders when it comes to V&V. Validation is a multi-disciplinary exercise that involves understanding the code and its capabilities; the hardware platform and its interactions with the software; the physical data and diagnostics available to validate various models; the statistical aspects of data analysis; and the purpose of the simulation in the decision-making process. Validation should therefore involve the code developers, computer scientists, experimentalists, statisticians, analysts, and application owners. Because of their expertise with testing, calibration, and the quantification of measurement error, test engineers can play a central role in the definition of validation experiments. Owners of the application, while soliciting the involvement of others, should play the central role because they are eventually responsible for making decisions based on simulation results. Guaranteeing a certain level of independence between V&V agents and other stake-holders is desirable because independence promotes credibility.