A Handbook of Quality Guidelines for Statistical Production in Tanzania
16
2.2.7.2 Guidelines
2.2.7.2.1 Define comparable tar get populations and verify that the sampling frames provide adequate coverage to enable the desired level of generalization.
2.2.7.2.2 Minimize the amount of measurement error attributable to survey instrument design, including error resulting from context effects.
2.2.7.2.3 Minimize or account for the impact of language differences resulting from potential translations.
2.2.7.2.4 Minimize the effect interviewer attributes have on the data through appropriate recruitment, selection, and case assignment; minimize the effect that interviewer behavior has on the data through formal training.
2.2.7.2.5 Identify potential sources of unexpected error by implementing pretests of translated instruments.
2.2.7.2.6 Reduce the error associated with non-response as much as possible.
2.2.7.2.7 Minimize the effect that coder error has on the data through appropriate coder training.
2.2.7.2.8 Provide variables definitions to minimize comparability of incomparable variables.
2.3 Survey process quality management
This approach focuses on quality at three levels; the organization, the process and the product.
Quality products cannot be produced without quality processes, and having quality processes
requires an organization that manages for quality. A focus on the survey process quality is to ensure
the quality of survey production processes and consequently the survey data throughout the
statistical production lifecycle, as well as a clear and comprehensive documentation of study
methodology and to provide indicators of the process and data quality.
The guidelines below illustrate Survey Process Quality Management that allows users to assess the quality of processes throughout the statistical production lifecycle.
A Handbook of Quality Guidelines for Statistical Production in Tanzania
17
2.3.1 Develop a sustainable quality management plan.
2.3.1.1 Rationale Developing planned, systematic quality assurance and quality control activities helps to ensure that the study and survey data meet the client or user requirements. It also facilitates development of a quality profile or quality rep ort, which documents survey methodology, key indicators of quality, lessons learned, and recommendations for improvement.
2.3.1.2 Procedural steps
2.3.1.2.1 Review the existing quality profiles and lessons learned from other studies. Use the standardized quality profiles and protocols to establish sustainable quality management.
2.3.1.2.2 Review the study requirements for quality assurance and quality control. These may be developed at the study design stage by the survey organization.
2.3.1.2.3 Review the study goals and objectives, required products and deliverables, study timeline and budget.
2.3.1.2.4 Through analysis of the process in the statistical production lifecycle (process analysis), identify characteristics of survey products (e.g. coded d ata) that could vary during the process (e.g. verification failures). For example,
(i) Use tools to analyze a process, to determine what steps in the process need to be monitored to ensure quality, and to identify quality indicators to monitor.
(ii) Identify key indicators of the quality of the process products in terms of TSE and other dimensions of quality, as well as factors such as cost, burden, and the risk of not meeting quality requirements.
(iii) Define measurement and reporting requirements for use during quality assurance and quality control, and determine who would be responsible for ensuring that quality assurance and quality control activities are carried out.
A Handbook of Quality Guidelines for Statistical Production in Tanzania
18
(iv) Assess whether the requirements can be met through the current procedures and system and with currently collected data and if not, develop a process improvement plan.
(v) Create cost/error trade-off decision rules about how to alter the features of the study design if the goals are not met.
2.3.1.2.5 Use quality planning tools to help determine what performance analyses and assessments should be used. For example,
(i) A cost-benefit analysis of potential quality management procedures and activities; that is, evaluating their benefits in relation to the cost of performing them relative to the overall study costs.
(ii) Benchmarking, that is, comparing planned activities against those of similar
studies, and the outcomes of those activities, to form a basis for performance
measurement.
(iii) Statistical analysis of factors that may influence indicators of the process or
product quality.
2.3.1.2.6 Develop a quality assurance plan, which could include:
(i)
The process improvement plan.
(ii) Performance and product quality baselines.
(iii) Process checklists.
(iv) A training plan.
(v)
Recommended performance analyses and assessments, for example quality
assurance procedures for verifying interviews and evaluating interviewer
performance (PES).
2.3.1.2.7 Develop a plan for continuous monitoring of processes to ensure that they are stable and that products are meeting the requirements (Quality Control). Such a plan could include: (i) The process improvement plan. (ii) Performance and product quality baseline. (iii) Quality indicators identified in the process of analysis and planning for the design. (iv) Performance analyses and assessments to monitor the process.
A Handbook of Quality Guidelines for Statistical Production in Tanzania
19
(v) Tools to monitor the process and product quality, e.g. statistical process control charts. (vi) Reports to prepare performance measurement, such as interviewer training certification. 2.3.1.2.8 Develop procedures to ensure that throughout the statistical production lifecycle, all documentation, reports and files related to quality planning and assurance, quality monitoring and control, and process improvement are retained.
2.3.1.2.9 Develop procedures for updating the quality management plan as needed during the statistical production lifecycle.
2.3.2 Perform quality assurance activities.
2.3.2.1 Rationale Quality assurance is the planned procedures and activities an organization uses to ensure that the study meets the process and product quality requirements. It specifies ways in which quality can be measured.
2.3.2.2 Procedural steps
2.3.2.2.1 Perform quality assurance activities as outlined in the quality management plan.
2.3.2.2.2 Carry out performance and product quality assessments. For example: (i) Certification of interviewers after training (e.g. rate of certification and rate of certification after follow-up training) that is, based on evaluation of interviews, determination that the interviewer is ready to work on the study.
(ii) Verification of coded questionnaires (rate of verification failures).
2.3.2.2.3 Generate indicators of quality for each assessment, based on baselines established in quality planning, and create reports on performance and quality assessments, which can be used for both quality monitoring and control.
2.3.2.2.4 Provide documentation for: (i) Performance and quality assessments (ii) Recommended corrective actions and corrective actions taken (iii) Changes to quality assurance plan.
A Handbook of Quality Guidelines for Statistical Production in Tanzania
20
2.3.3 Perform quality control activities.
2.3.3.1 Rationale Quality control is the planned system of process monitoring, verification and analysis of indicators of quality, and updates to quality assurance procedures, to ensure that quality assurance works.
2.3.3.2 Procedural steps
2.3.3.2.1 Perform quality monitoring and control activities as outlined in the quality management plan, such as: (i) Monitor the process of quality indicators (ii) Analyze and report on the results of quality assurance activities such as interviewer training certification, data entry verification and checking that a process met the specifications.
2.3.3.2.2 Determine whether there is a need to: (i) Recommend corrective actions (ii) Modify the process improvement plan (iii) Modify the quality management plan
2.3.3.2.3 Provide documentation for: (i) Performance and quality assessments (ii) Recommended corrective actions and corrective actions taken (iii) Changes to the quality management and quality assurance plans
2.3.4 Create a quality profile
2.3.4.1 Rationale A quality profile (quality report) combines information from other sources, documenting survey methodology used throughout the statistical production lifecycle, providing indicators of the process and data quality, lessons learned and recommendations for improvement. It provides to the user all the information available to help assess data quality in terms of fitness for the intended use and total survey error.
A Handbook of Quality Guidelines for Statistical Production in Tanzania
21
2.3.4.2 Procedural steps
2.3.4.2.1 Document procedures and methodology used for key stages or processes in the
statistical production. For example, for sample design, this would include:
(i)
Time dimension of the design
(ii)
Target and survey population definitions, including inclusion/exclusion criteria
(iii) Sampling frame descriptions
(iv) Maps and protocol used in field listing
(v)
Description of all stages of selection, including sample sizes, stratification,
clustering and number of replicates fielded at each stage
(vi) Documentation of procedures to determine probabilities of selection and weights
for each stage of selection
(vii) Tables of the precision of the estimates of key survey statistics
For Each documented process should include: (i) Quality assurance procedures (ii) Quality control procedures (iii) Corrective actions taken
2.3.4.2.2 Document lessons learned and make recommendations for improvement in studies of the same design, and if possible, make recommendations for methodological research that could inform design of similar studies in the future.
A Handbook of Quality Guidelines for Statistical Production in Tanzania
22
PART THREE QUALITY INPUTS AND GUIDELINES IN DATA PRODUCTION STEPS
3.0 Introduction This section brings together guidelines and checklists on major issues that need to be considered in the pursuit of quality objectives in the execution of statistical activities. Its focus is on how to assure quality through effective and appropriate design and implementation of a statistical programme from inception through data evaluation, documentat ion and dissemination. It is organized in sub -sections that correspond to the main activities of a typical survey. All the sub- sections follow the same structure, describing the inputs, guidelines and quality indicators related to each activity.
Guidelines
These are known good practices that have evolved in the design and implementation of statistical
surveys. However, not all of these guideline s can be applied to every statistical production. The
guidelines provide checklists to aid survey designs.
Quality indicators These consist of information which is a by-product of the statistical process. They do not measure quality directly but can provide enough information to offer valuable insight into quality. It will be of interest to directors, statistical production managers and data users, who will use the indicators to assess and compare the quality of various statistical products. It will also provide a basis to directors and managers of different program areas for monitoring performance in terms of qual ity of the processes and products in the program areas.
3.1 Coverage and frames Goal: The survey population should be reasonably consistent with the target population in order for the survey results to be relevant. Coverage is the completeness of the info rmation for the target population that would be derived if all the frame units were to be surveyed. The frame should conform to the survey population and should contain minimum under -coverage and over-coverage. Frame data should be up -to-date and accurate because of their use in stratification, sample selection, collection follow -up, data processing, imputation, estimation, quality assessment and analysis.
A Handbook of Quality Guidelines for Statistical Production in Tanzania
23
3.1.1 Quality inputs 3.1.1.1 Define the target and survey population 3.1.1.2 Description of the frame and its coverage errors
3.1.2 Guidelines 3.1.2.1 Frames should be tested at the planning stage of a survey for their suitability and quality, assess the coverage of the frame and of the target collection units.
3.1.2.2 Ensure that the frame is as up to date as possible relative to the reference period for the statistical production.
3.1.2.3 Where possible, use the same frame for surveys with the same target population to avoid inconsistencies, to facilitate combining estimates from the surveys and to reduce costs of frame maintenance and evaluation.
3.1.2.4 Implement survey procedures to detect and correct coverage errors from the frame, provide feedback to up-date and maintain the frame.
3.1.2.5 Monitor the frame between the time of sample selection and the survey reference period.
3.1.2.6 Implement training and procedures for data collection and data processing staff aimed at minimizing coverage error.
3.1.2.7 Minimize frame errors through effective training of staff, putting emphasis on the importance of coverage, and the implementation of quality assurance procedures of frame and related activities.
3.1.2.8 Implement procedures to detect and minimize errors of omission and mis-classification that can result into under-coverage, and to detect and correct errors of mistaken inclusion and duplication resulting into over-coverage.
3.1.2.9 Monitor the frame quality by periodically assessing its coverage and the quality of the information on the characteristics of the units.
3.1.3 Quality indicators Main quality dimensions and elements: Accuracy – Coverage error and Relevance.