A Handbook of Quality Guidelines for Statistical Production in Tanzania
8
2.2.2.1 Indicators for accuracy
2.2.2.1.1 Sampling error (i) Type of sample design (stratified, clustered, etc) (ii) Sampling unit at each stage of sampling - sampling unit to be defined at the beginning of the report (iii) Stratification and sub-stratification criteria (iv) Selection schemes (v) Sample distribution over time (vi) The effective sample size (vii) Coefficient of variation of estimates and a description of the method used to compute them (including software) (viii) An assessment of resulting bias due to the estimation method
2.2.2.1.2
Measurement errors
(ix)
A description of the methods used to assess measurement errors (any field tests,
reinterviews, split sample experiments, or cognitive laboratory results, etc)
(x)
A description of the methods used to reduce measurement errors
(xi)
Average time used to interview one person
(xii)
An assessment of the effect of measurement errors on accuracy
2.2.2.1.3 Processing errors (caused by instruments or human error) (xiii) A description of the methods used to reduce processing errors (xiv) A description of the editing systems (xv) The rate of failed edits for specific variables (xvi) The error rate of coding for specific variables and a description of the methodology followed for their estimation (xvii) A description of confidentiality rules and the amount of data affected by confidentiality treatment
2.2.2.1.4 Coverage errors (xviii) A description of the sampling frame (e.g. intended household is collective instead of individual)
A Handbook of Quality Guidelines for Statistical Production in Tanzania
9
(xix) Rates of over - coverage, under -coverage and mis -classification broken down
according to the sampling stratification
(xx) A description of the main mis -classification and un der- and over - coverage problems encountered in collecting the data
(xxi) A description of the methods used to process the coverage deficiencies
2.2.2.1.5 Non response errors (xxii) Unit non response rate (xxiii) Identification and description of the main reasons for non re sponse (e.g . non- contact, refusal, unable to respond, non-eligible, other reasons)
(xxiv) A description of the methods used to minimize non response (xxv) Item non response rates for variables (xxvi) A description of the methods used for imputation and/or weighting for non - response
(xxvii) Variance change due to imputation (xxviii) An assessment of resulting bias due to non response
2.2.2.2 Guidelines
2.2.2.2.1 Pre-test all the versions of the survey instrument s to ensure that they adequately convey
the intended research questions and mea sure the intended attitudes, values, reported
facts and /or behaviors.
2.2.2.2.2 In order to reliably project from the sample to the larger population with known levels of certainty/precision, use probability sampling.
2.2.2.2.3 Provide a report on each variable in the dataset of the selected elements to check correct overall sample size and within the stratum sample size, distribution of the sample elements by other specific groups such as census enumeration areas, extreme values, nonsensical values, and missing data.
2.2.2.2.4 If possible, assess accuracy by looking at the differences between the study estimates and any available true standard values. A Handbook of Quality Guidelines for Statistical Production in Tanzania
10
2.2.3 Timeliness and punctuality The quality of timeliness and punctuality are supposed to be obser ved at all stages of data production and dissemination. The experience has shown that , most of data production in NSS follows the accepted work plans which explain time to be taken by each activity. For example, time for designing instruments, dispatching and receiving dates of instruments, time for data processing, analysis and reporting; and date and time for dissemination. It is through this scenario, data producer can decide to measure the timeliness and punctuality based on the type of data production by selecting indicators elaborated below:
2.2.3.1 Indicators of timeliness and punctuality
2.2.3.1.1 The legal deadline imposed on respondents This is an indicator which measure s time set for a respondent to answer questions or reply to data producer based on the type of assignment.
2.2.3.1.2 The date instruments i.e. questionnaires were dispatched This is an indicator used by data producers to monitor movement of instruments from the centre to the outreach offices or sub- locations or field.
2.2.3.1.3 Starting and finishing dates of fieldwork
This is a life span indicator which measure s duration set for starting and finishing of
data collection exercise either from the field or office. For example, if time set for data
collection is one month, then, data quality assurance team should observe whether field -
work is completed within or beyond a month.
2.2.3.1.4 Starting and finishing dates of data processing
This is also a life span indicator which measure s duration set for starting and finishing
of data processing exercise including data entry, editing and processing. For example, if
time set for data processing is six months, then , data quality assurance team should
observe if time period was correctly observed.
2.2.3.1.5 Dates for preliminary and final results computed and disseminated This is an indicator which s ets dates for the release of preliminary and final computed results to the public. This indicator depends on the completion of prior activities including field-work and data processing exercises. For example, if data is disseminated later than required by the regulation or contract, the average delay in days or months in A Handbook of Quality Guidelines for Statistical Production in Tanzania
11
the transmission of results with reference to the legal deadline violate the quality of timeliness and punctuality.
2.2.3.2 Guidelines It is important to note that, timeliness and punctuality is a qualitative indicator which depends on the availability of other factors including resources in terms of human, financial and infrastructure to be used in data production process. It is advised that data producers should not be over ambitious in setting time for each activity at the planning stage in order to reduce bias ness. For example, data producer should be careful in setting:
2.2.3.2.1 Time for each activity should also be s et by observing external factors which may hinder its performance properly; and
2.2.3.2.2 Data producer should create a study timeline, production milestones, and deliverables with due dates.
2.2.4 Accessibility This is the ability to retrieve data for the benefit of as many users as possible. Very often , collection of data consumes a lot of financial and human resources; therefore , should not be locked in cabins. Data must be made available in hard copies and soft copies through various media such as libraries, CD Rom, websites and even e-mailing to regular users.
2.2.4.1 Indicators of accessibility
2.2.4.1.1 Description of how to locate and access publications based on analysis of the data. The storing of data in a systematic and classified order in dicating the subject and time series the data covers. The system can be used for easy access of data by users in a library, website etc.
2.2.4.1.2 Information on what results are sent to the reporting units included in the survey. This is the set of resul ts sent to units that supply the data for a given survey. It is a feedback that creates trust between the data collector and supplier which is important as it creates the two way information traffic.
2.2.4.1.3 Information on the dissemination scheme of the results. A Handbook of Quality Guidelines for Statistical Production in Tanzania
12
This is the policy statement indicating the set of data and time it is to be made available and accessible to users. It is important for data producers to make data available as per the dissemination scheme in order to create trust among the users. If the dissemination time -table is not adhered to, then, users lose trust on the data and sometimes when data are made available and accessible they may be obsolete and therefore , useless irrespective of how much resources were spent in their collection. An example of the dissemination time-table is the Tanzania Master Plan Release Calendar.
2.2.4.1.4 A list of Variables required but not available for Reporting.
This is the list of attributes or quantitative value s required by data users which is not
available for reporting. Attributes can be Identification Number, Establishment ID while
quantitative value is expressed in numeric value.
2.2.4.1.5 Reasons why variables are not available. Reasons of why variables are not availab le for reporting include ; the anonymity for individual data sets based on profession al ethical principles for protecting privacy and maintaining confidentiality. A good example is the NBS confidentiality on individual entity data sets. Other reasons could be that the information is not collected due to budgetary constraints.
2.2.4.2 Guidelines
2.2.4.2.1 Save all data files and computer syntax in different statistical software packages during sample design and data processing in safe and well labeled fold ers for future reference and use.
2.2.4.2.2 Establish procedures early in the survey lifecycle to ensure that all important files are preserved.
2.2.4.2.3 Test the archived files periodically to verify user accessibility.
- 2.4.2.4 Create electronic versions of all project materials on a regular basis at each stage of
statistical production.
2.2.4.2.5 Produce and implement procedures to distribute restricted use files, if applicable. (Removing identifiers, off-setting GPS coordinates, etc) A Handbook of Quality Guidelines for Statistical Production in Tanzania
13
2.2.5 Interpretability The process of data collection and subsequent processing should have an impact to the measure of accuracy that is being represented. The information collected often includes underlying concepts, variables and classification on how it is to be used. The interpretability of data is directly related to its availability to be aggregated with other pieces of information necessary to interpret it appropriately such as metadata and paradata.
Metadata is defined as data providing information ab out one or more aspects of the data while paradata are data about the process by which the survey data were collected. For example, paradata topics about a survey include the time of day interviews that were conducted, how long the interviews took, how ma ny times there were contacts with each interviewee or attempts to contact the interviewee, the reluctance of the interviewee, and the mode of communication (such as phone, Web, email, or in person).
2.2.5.1 Indicator of Interpretability
2.2.5.1.1 A copy of methodological documents (sampling design, classifications, instructions manual, codebook) relating to the statistics provided
2.2.5.1.2 A metadata embedded with the data file may inc lude the information on means o f creation of the data, purpose of dat a, time and date of creation and location on a computer network.
2.2.5.1.3 A paradata document shows how each observation in the survey affect the costs and management of a survey, the findings of a survey, evaluations of interviewers, and inferences one might make about non -respondents. Sometimes paradata is called "administrative data about the survey".
2.2.5.1.4 Visualization tools are among the most effective ways to display development indicators through graphs and charts. It is a visual display of d ata which makes comparisons easier and promotes a better understanding of trends. Such charts include bubble diagrams etc.
A Handbook of Quality Guidelines for Statistical Production in Tanzania
14
2.2.5.2 Guidelines
2.2.5.2.1 At the data processing stage of the study, create a codebook that provides question- level metadata matched to variables in the dataset. Metadata include variable names, labels, and data types as well as basic study documentation, question text, universes (the characteristics of respondents who were asked the question) of the number of respondents who answered the question, and response frequencies.
2.2.5.2.2 Make available the system that stores all the necessary information collected during data collection.
2.2.6 Coherence The quality of coherence ensures that, data can be combined with other st atistical information for various secondary purposes. Two or more different statistical data -sets can be used together for measuring and or determining some other statistical information, e.g. GDP with the economic growth; GDP with population growth rate, inflation rate which determines people’s purchasing power and their economic welfare, etc.
Therefore, this ensures that the combined data together with other statistical information can be used in various secondary purposes.
2.2.6.1 Indicators of coherence
2.2.6.1.1 A description of every pair of statistics (statistical unit, indicator, domain, and breakdown) for the survey(s) that should be coherent.
2.2.6.1.2 A description of any of the differences that are not fully explained by the accuracy component.
2.2.6.1.3 A description of the reported lack of coherence, for specific statistics.
2.2.6.2 Guidelines
2.2.6.2.1 Create a clear, concise description of all survey implementation procedures to assist secondary users.
A Handbook of Quality Guidelines for Statistical Production in Tanzania
15
2.2.6.2.2 Provide data files in all the major statistical software packages and test all thoroughly before they are made available for dissemination.
2.2.6.2.3 Designate resources to provide user support and training for secondary researchers.
2.2.7 Comparability The observed data from different geographical locations of the same dimension can be compared with other data-sets, if at all they explain the same phenomenon, e.g. Inflation Rate of Tanzania can be compared with those of other East African countries or even beyond the African boundaries as one of the SNA objectives states. This will ensure as much as possible that, the same statistical data from different sources or different geographical locations are comparable.
2.2.7.1 Indicators of comparability
2.2.7.1.1 Time (temporal) (i) The differences, if any, in concepts and methods of measurement between last and previous reference periods.
(ii) A description of the differences, including an assessment of their effect on the estimates.
2.2.7.1.2 Geographical (spatial) (i) All differences between local practices and national standards (if such standards exist)
(ii) An assessment of the effect of each reported difference on the estimates
2.2.7.1.3 Domains (i) A description of the differences in concepts and methods across cross-cultural surveys (e.g., in classifications, statistical methodology, statistical population, methods of data manipulation, etc.)
(ii) An assessment of the magnitude of the effect of each difference