Statistical Methods, Standards and Guidelines 9
(vi)
Consultations with potential stakeholders to identify their requirements and
expectations.
(b)
A review of related studies, surveys, and reports of Tanzania and non-Tanzania sources to
ensure that part or all of the survey would not unnecessarily duplicate available data from an
existing source, or could not be more appropriately obtained by adding questions to existing
Tanzania statistical surveys. The goal here is to minimize unnecessary use of limited
resources available for surveys in the country and minimize burden to data producers;
(c) A review of the confidentiality and privacy policy of an existing Statistics Act. on surveys that will collect individually-identifiable data from any survey respondent;
(d) A complete review of all survey data items, the justification for each item, and the means of measurements (e.g., through questionnaires, tests, or administrative records);
(e) A plan for pre-testing or cognitive interviewing, if applicable;
(f)
A plan for quality assurance during each phase of the survey process to permit monitoring
and assessing performance during implementation;
(i)
The plan should include possibility to modify the survey procedures if design
parameters appear unlikely to meet expectations (for example, if low response rates
are likely);
(ii)
Should contain general specifications for an internal project management system that
identifies critical activities; and
(iii)
Key milestones of the survey that will be monitored, and the timeframes among
them.
(g) A plan for evaluating survey procedures and results;
(h) An analysis plan that identifies analysis issues, objectives, key variables and proposed statistical tests;
(i) An estimate of resources and target timeframe needed for completion of the survey cycle; (j) A dissemination plan that identifies target audiences, proposed major information products, and the timing of their release; and
(k) A data management plan for the preservation of survey data, documentation, and information products as well as the authorized disposition of survey records.
Statistical Methods, Standards and Guidelines 10
Guideline 2.1.1.3: Include standard elements of project management in the plan, including target completion dates, the resources needed to complete each activity, and risk planning.
Guideline 2.1.1.4: To maintain a consistent data series over time, use consistent data collection procedures for ongoing data collections on core statistics. Continuous improvement efforts sometimes result in a trade-off between the desire for consistency and a need to improve a data collection. If changes are needed in key variables or survey procedures for a data series, consider the justification or rationale for the changes in terms of their usefulness for policymakers, conducting analyses, and addressing information needs. Develop adjustment methods, such as crosswalks and bridge studies that will be used to preserve trend analyses and inform users about the effects of changes.
2.1.2 Survey Designing
Standard 2.1.2: MDAs, LGAs and other stakeholders must develop a survey design, including
defining the study frame, target population, sampling plan, identify the data collection instruments
and methods, developing a practical timetable, estimating survey cost, and selecting samples using
accepted statistical methods (e.g., probabilistic methods that can provide estimates of sampling
error). Any use of non-probability sampling methods (e.g., judgmental, Quota and Snowball etc.
samples) must be justified statistically and be able to measure estimation error. The size and
design of the sample must reflect the level of detail needed in tabulations and other data products,
and the precision required of key estimates. Documentation of each of these activities and
resulting decisions must be maintained in the project files for use in documentation.
The guidelines for this standard are:
Guideline 2.1.2.1: Include the following in the survey design:
(a)
Frame for selection;
(b)
Proposed target population;
(c)
Stratification levels/domain of study and analysis;
(d)
Response rate from previous survey or expected response rate;
(e)
Survey frequency;
(f)
Timing of data collection;
(g)
Data collection modes (such as paper and pencil, mail survey, telephone survey,
etc);
Statistical Methods, Standards and Guidelines 11
(h)
Sample design;
(i)
Precision requirements;
(j)
Effective sample size determination based on power analyses for key variables; and
(k)
Overall sample size.
Guideline 2.1.2.2: Ensure the sample design will yield the data required to meet the objectives of
the survey. Include the following in the sample design:
(a)
Identification of the sampling frame (address, name, location);
(b)
Identify the sampling unit used (at each stage if a multistage design);
(c)
Identify sampling strata;
(d)
Power analyses to determine sample sizes;
(e)
Effective sample sizes for key variables by reporting domains (Urban/Rural where
appropriate);
(f)
Criteria for stratifying or clustering, sample size by stratum, and the known
probabilities of selection;
(g)
Response rate goals (see Standard 2.1.3); estimation and weighting plan; variance
estimation techniques appropriate to the survey design; and
(h)
Expected precision of estimates for key variables.
Guideline 2.1.2.3: When a non-probabilistic sampling method is employed, include the following
in the survey design documentation:
(a)
A discussion of what options were considered and why the final design was
chosen;
(b)
An estimate of the potential bias in the estimates.
Guideline 2.1.2.4: Include a statement of confidentiality along with instructions required to complete the survey.
Guideline 2.1.2.5: Include the following in the data collection plans:
(a)
Frequency and timing of data collection;
(b)
Methods of collection for achieving acceptable response rates;
(c)
Training of enumerators and persons key to the survey, coding and editing the data;
(d)
Cost estimates, including the costs of pretests;
(e)
Non-response follow-up; and
Statistical Methods, Standards and Guidelines 12
(f) Evaluation studies.
2.1.3 Response Rates
Standard 2.1.3: MDAs, LGAs and other stakeholders must design the survey to achieve the
highest rates of response to ensure that survey results are representative of the target population so
that they can be used with confidence to inform decisions. Non-response bias analyses must be
conducted when unit or item response rates or other factors suggest the potential for bias to occur.
The guidelines for this standard are: Guideline 2.1.3.1: Calculate sample survey unit response rates without substitutions.
Guideline 2.1.3.2: Design data collections that will be used for sample frames for other surveys (e.g., the Population and Housing Census enumeration areas (EAs), and the Statistical Business Register of Establishments) to meet a target unit response rate of at least 80 percent, or provide a justification for a lower anticipated rate.
2.1.4 Focus Group Discussions (for instrument development) Standard 2.1.4: MDAs, LGAs and other stakeholders must ensure that the survey collects the required information for their intended producers and users. The purpose of this standard is to get key issues regarding the planned survey before developing the survey questionnaire.
The guidelines for this standard are: Guideline 2.1.4.1: Identify key stakeholders in the subject matter area who will participate in focus group discussion.
Guideline 2.1.4.2: Prepare semi-structured (focused) discussion with members of the target population to expose what they know about the study that the questionnaire will cover, how they think about the study and what terms they use in talking about the study topics/variables.
Guideline 2.1.4.3: Recruit volunteers (10-20 from data collectors, producers and users side) who are at least familiar or are expected to be data producers or users of the study; to participate in a systematic discussion guided by a moderator about the survey topic(s) (questions for discussion
Statistical Methods, Standards and Guidelines 13
should be prepared prior to convene volunteers). Lessons learned from the discussion will be the basis for questionnaire design.
2.1.5 Designing Survey Instrument(s) (Questionnaire(s))
Standard 2.1.5: Based on the experiences and lessons drawn from the literature review and focus
group discussion, but mainly reflecting on the objectives of the proposed study, MDAs, LGAs and
other stakeholders should design a questionnaire that will capture the intended information to be
collected. The instrument shall probe and systematically record comprehensive information that
answers the study questions.
The guidelines for this standard are: Guideline 2.1.5.1: Identify subject matter specialists (mostly statisticians, researchers, sociologists, economists, etc) who will draft the questionnaire.
Guideline 2.1.5.2: Review literatures and instruments from previous similar studies for comparability purposes.
Guideline 2.1.5.3: Check the identification and demographic variables of the existing instruments if they meet the requirements of the intended study and update accordingly.
Guideline 2.1.5.4: Design the questionnaire using the available questions bank in Tanzania (e.g. TNADA) or outside Tanzania.
Guideline 2.1.5.5: Prepare instruction manuals for data collectors and supervisors.
2.1.6 Pre-testing of Survey Instruments Standard 2.1.6: MDAs, LGAs and other stakeholders must ensure that the draft questionnaire is pre-tested to randomly chosen participants by interviewers to probe the understanding of the study questions to respondents, time to complete one questionnaire and an attempt to learn how they formulate their answers. Then the recording of the outcome of the interview is done for the purpose of questionnaire improvement. By conducting a pretest of the survey components, measurement error will be controlled.
Statistical Methods, Standards and Guidelines 14
The guidelines for this standard are: Guideline 2.1.6.1: Randomly choose participants to participate in pre-test interviews.
Guideline 2.1.6.2: Key researchers and survey desk officers should participate fully in the cognitive interview and if possible recording the interview for quality checking.
Guideline 2.1.6.3: Arrange technical meeting to discuss the experiences learned from the cognitive interviews and use the results to improve the questionnaire.
Guideline 2.1.6.4: Record starting and finishing times for questionnaire interviews to determine the average time spent per questionnaire. The technical committee can then allocate number of questionnaires to be completed per interviewer per day.
2.1.7 Training of Trainers (TOT), Supervisors and Training of Enumerators (TOE) Standard 2.1.7: MDAs, LGAs and other stakeholders should recruit field staff to participate in TOT and TOE on the basis of their competence and experience in the planned data production exercise such as a census or a survey.
The guidelines for this standard are: Guideline 2.1.7.1: Identify key staff (trainers) who will train supervisors and enumerators
Guideline 2.1.7.2: If supervisors and enumerators are not enough or available within an organization, consider to hire and recruit them. In addition, recruit reserve supervisors and enumerators for replacement in case of dropouts during main survey.
Guideline 2.1.7.3: Prepare conducive environment for training in terms of geography, conference facilities and accommodation for participants.
Guideline 2.1.7.4: A maximum group of 5 enumerators should be supervised by one supervisor.
Guideline 2.1.7.5: Prepare mock exam to test the understanding of trainees.
Statistical Methods, Standards and Guidelines 15
Guideline 2.1.7.6: Prepare form for trainers’ evaluation. This will help to show relationships of trainers-trainees and understanding of areas for improvement.
2.1.8 Pilot Testing
Standard 2.1.8: MDAs, LGAs and other stakeholders must administer at least a sample covering
zonal areas using all field procedures similar to the main census or survey instruments and hold
debriefing between interviewers and project team.
The guidelines for this standard are: Guideline 2.1.8.1: Test all field instruments, logistical procedures and human resource capacity. Items to be tested include frame development, sample selection, questionnaire design, data collection, edit specifications, data processing, estimation, file creation and tabulations. A complete test of all components is desirable for all surveys that cover the whole country.
Guideline 2.1.8.2: Tabulate pilot results to see patterns of missing data values, recording and behavior coding to detect patterns of questions that are difficult to answer.
Guideline 2.1.8.3: Use the lessons learned from the pilot to improve the instruments and the approach to the study.
2.2
DATA COLLECTION
2.2.1 Sampling Frames
Standard 2.2.1: MDAs, LGAs and other stakeholders must ensure that the frames for the planned
sample survey or census are suitable for the study design and are assessed against the target
population for quality checking.
The guidelines for this standard are:
Guideline 2.2.1.1: Describe target populations and associated survey or sampling frames. Include
the following items in this description:
(a)
Describe the approach in which the frame was created and the methodology of
frame updating;
(b) Describe exclusions that have been applied to frame and target populations;