Standards for Cross-National Research

CONSIRT promotes high quality standards for interdisciplinary cross-national research. To this end, CONSIRT is developing methodology and guidelines on issues that matter to researchers, professionals in governmental and non-governmental institutions, and funding agencies:

  • Increasing the effective and efficient use of extant cross-national survey and non-survey data;
  • Assessing survey data quality;
  • Standardizing the documentation of international survey projects.

In the field of quantitative comparative research, CONSIRT does groundbreaking work in developing a new theoretical approach and corresponding methodology for increasing the effective use of extant cross-national data. This work, initiated in the Harmonization Project and continued in the SDR Project, has important implications for survey data quality assessment and the standardization of international survey projects’ documentation.

The key idea is “data recycling.” In short, data recycling involves both the process of accounting for project-specific quality of the survey data, by constructing variables that capture different aspects of source data quality, and the process of expanding data coverage – in terms of time, space, number of observations, and types of indicators – via ex-post survey data harmonization. Separate control indicators for transformations of source variables as part of harmonization facilitate validity and reliability assessments of the target variables. A target variable is the new, common, measure that is constructed from survey items pooled from datasets not a priori designed as comparable.

Survey quality controls and controls of the harmonization process constitute metadata that enable researchers to contend with basic methodological biases and errors of survey data in substantive analyses.[1]   

Rather than disregarding, as some research does, projects weaker on quality but strong in terms of country and or/topic coverage, or “older” projects conducted before current standards were adopted, users can include the metadata in regression analyses to partial out their effects, thus “recycling” data of varying quality.

Alternatively, researchers could use the metadata as “filters,” that is, to select those datasets and items that best fit their analysis and data requirements. Another possibility is to use the control variables to construct weights for survey “importance” so that information from high-quality surveys will carry more weight on the substantive results.

The survey data recycling (SDR) analytic framework has implications for data quality assessment and survey documentation standardization.

Poorly documented surveys reduce user confidence, while processing errors stemming from inconsistencies between documentation and data records in the computer files make the data difficult to use. Evaluating the general documentation – study descriptions, questionnaires, technical reports – corresponding to the 3,486 national surveys of the 23 cross-national studies in the SDR Project database shows that projects differ substantially in how transparent and accurate the descriptions of survey design and implementation, and of data coding, are (Kołczyńska and Schoene 2018; Oleksyienko et al. 2018; Tofangsazi and Lavryk 2018).[2]

The SDR schema of coding survey quality as reflected in the documentation includes survey implementation stages that are key for ensuring high quality of the resulting data, such as type of the sample, details of the sample, response rate, control of the quality of the questionnaire translation, questionnaire pretesting, fieldwork control, and interview mode. Administrators of on-going international survey projects and archiving institutions can rely on SDR guidelines and templates that follow Data Documentation Initiative standards (ddialliance.org) to produce consistent documentation of specific elements of survey implementation.

Endnotes

[1] By methodological biases and errors in survey data we mean consequences of (a) deviations from standards of documenting and preparing survey data suggested in the specialized literature and (b) inter-survey differences in harmonized items (Slomczynski and Tomescu-Dubrow. 2018. Ch. 43 in Advances in Comparative Survey Methods: Multinational, Multiregional, and Multicultural Contexts. Wiley).

[2] Kołczyńska M. and M. Schoene. 2018. Ch. 44; Oleksyineko, O., Wysmulek I., Vangeli A. Ch. 45 in Advances in Comparative Survey Methods: Multinational, Multiregional, and Multicultural Contexts. Wiley). Tofangsazi, B. and D. Lavryk. 2018. “We Coded the Documentation of 1700+ Surveys…” Harmonization: Newsletter on Survey Data Harmonization in the Social Sciences 4(2): 27–31.