Total survey error is a conceptual framework describing statistical error properties of sample survey statistics. Early in the history of sample surveys, it arose as a tool to focus on implications of various gaps between the conditions under which probability samples yielded unbiased estimates of finite population parameters and practical situations in implementing survey design. While the framework permits design-based estimates of various error components, many of the design burdens to produce those estimates are large, and in practice most surveys do not implement them. Further, the framework does not incorporate other, nonstatistical, dimensions of quality that are commonly utilized in evaluating statistical information. The importation of new modeling tools brings new promise to measuring total survey error components, but also new challenges. A lasting value of the total survey error framework is at the design stage of a survey, to attempt a balance of costs and various errors. Indeed, this framework is the central organizing structure of the field of survey methodology.

This book provides up-to-date insight into key aspects of methodological research for comparative surveys. It discusses methodological considerations for surveys that are deliberately designed for comparative research such as multinational surveys. Topics covered include questionnaire development, translation of survey materials, cultural bias, quality assurance, analysis models, and global survey programs.

Survey quality is a multi-faceted concept that originates from two different development paths. One path is the total survey error paradigm that rests on four pillars providing principles that guide survey design, survey implementation, survey evaluation, and survey data analysis. We should design surveys so that the mean squared error of an estimate is minimized given budget and other constraints. It is important to take all known error sources into account, to monitor major error sources during implementation, to periodically evaluate major error sources and combinations of these sources after the survey is completed, and to study the effects of errors on the survey analysis. In this context survey quality can be measured by the mean squared error and controlled by observations made during implementation and improved by evaluation studies. The paradigm has both strengths and weaknesses. One strength is that research can be defined by error sources and one weakness is that most total survey error assessments are incomplete in the sense that it is not possible to include the effects of all the error sources. The second path is influenced by ideas from the quality management sciences. These sciences concern business excellence in providing products and services with a focus on customers and competition from other providers. These ideas have had a great influence on many statistical organizations. One effect is the acceptance among data providers that product quality cannot be achieved without a sufficient underlying process quality and process quality cannot be achieved without a good organizational quality. These levels can be controlled and evaluated by service level agreements, customer surveys, paradata analysis using statistical process control, and organizational assessment using business excellence models or other sets of criteria. All levels can be improved by conducting improvement projects chosen by means of priority functions. The ultimate goal of improvement projects is that the processes involved should gradually approach a state where they are error-free. Of course, this might be an unattainable goal, albeit one to strive for. It is not realistic to hope for continuous measurements of the total survey error using the mean squared error. Instead one can hope that continuous quality improvement using management science ideas and statistical methods can minimize biases and other survey process problems so that the variance becomes an approximation of the mean squared error. If that can be achieved we have made the two development paths approximately coincide.

The paper uses a statistical process control perspective to describe how paradata or process data can be used to monitor the survey process. We describe the data and analyses that are available and present several case studies of paradata use in different types of surveys and organizations.

The chapter treats quality aspects in multinational surveys. Quality on three levels are discussed: product, process and organization as well as measures used to control these levels.