Standards for Cross-National Research

CONSIRT promotes high quality standards for interdisciplinary cross-national research. To this end, CONSIRT is developing methodology and guidelines on issues that matter to researchers, professionals in governmental and non-governmental institutions, and funding agencies:

  • Increasing the effective and efficient use of extant cross-national survey and non-survey data
  • Standardizing the documentation of cross-national studies.

In the field of quantitative comparative research, CONSIRT is doing groundbreaking work in developing a new theoretical approach and corresponding methodology for increasing the effective use of extant cross-national data. This work, initiated in the Harmonization Project and continued in the SDR project, has important implications for the standardization of international survey projects’ documentation.

Our key idea is “data recycling.” In short, data recycling involves both the process of controlling for project-specific limitations of data quality by constructing control indicators for the quality of the source data and the process of expanding data coverage – in terms of time, space, number of observations, and types of indicators – via ex-post survey data harmonization. Separate control indicators for transformations of source variables as part of harmonization facilitate validity and reliability assessments of the target variables. A target variable is the harmonized common variable produced from variables pooled from surveys not a priori designed as comparable.

Survey quality controls and controls of the process of harmonization constitute two types of metadata that enable researchers to contend with basic methodological biases and errors of survey data in substantive analyses. Rather than disregarding, as some ex-post harmonization efforts do, projects that may be weaker on quality but very strong in terms of country and or/topic coverage, or “older” projects conducted before current standards were adopted, users can include the metadata in regression analyses to partial out their effects, thus “recycling” data of varying quality.

Alternatively, researchers could use the metadata as “filters,” that is, to select those datasets and items that best fit their research and data requirements. Another possibility is to use the control variables to construct weights for survey ‘importance’ so that information from high-quality surveys will carry more weight on the substantive results.

The survey data recycling (SDR) analytic framework has implications for survey documentation standardization.

Poorly documented data reduce user confidence, while inconsistencies between codebook and technical reports on one hand, and the data records in the computer files, on the other, make the data difficult to use. While evaluating the general documentation – study descriptions and technical reports – corresponding to the 1,721 national surveys of the 22 cross-national studies in the Harmonization Project database, we found that projects differ substantially in how transparent and accurate the descriptions of survey design and implementation, and of data coding, are.

The SDR schema of coding survey quality as reflected in the general documentation includes survey implementation stages that are key for ensuring high quality of the resulting data, such as type of the sample, details of the sample, response rate, control of the quality of the questionnaire translation, questionnaire pretesting, and fieldwork control. Administrators of on-going international survey projects and archiving institutions can rely on SDR guidelines and templates that follow Data Documentation Initiative standards (www.ddialliance.org) to produce consistent documentation of specific elements of survey implementation.