A Handbook of Quality Guidelines for Statistical Production in Tanzania
24
3.2 Sample design Goal: In each survey, to select an optimal, cost -efficient probability sample that is representative of the target population and allows users to make inf erences to the target population, and to standardize sample designs without hampering optimal designs.
3.2.1 Quality inputs 3.2.1.1 Target and survey population description.
3.2.1.2 Sampling frame definitions, including definitions of strata and sampling units, and any up-dating of the frame that was needed.
3.2.1.3 Desired level of precision overall and for specific sub-groups.
3.2.1.4 Sample size based on specified levels of precision.
3.2.1.5 Selection procedure(s) and estimates of probabilities of selection at each stage.
3.2.1.6 Field listing standard procedures and minimum requirements of field listers.
3.2.1.7 Unique, sample identification codes for each selected sampling unit.
3.2.2 Guidelines 3.2.2.1 When determining the sample size, take in to account the required levels of precision needed for the survey estimates, the type of design and estimators to be used, the availability of auxiliary information, budgetary constraints as well as both sampling and non-sampling factors.
3.2.2.2 Alter the survey design during data collection to minimize costs and errors.
3.2.2.3 For longitudinal panel surveys, determine its duration of time in the sample by balancing the need for duration data with sample attrition and conditioning effects.
3.2.2.4 For periodic surveys, develop procedures to monitor the quality of the sample design over time. Set up an up -to-date strategy for selective redesign of the strata that have suffered serious deterioration.
A Handbook of Quality Guidelines for Statistical Production in Tanzania
25
3.2.2.5 For periodic surveys, make the design as flexible as possible to deal with future changes such as increases or decreases in sample size, re -stratification, re -sampling and up - dating of selection probabilities.
3.2.2.6 Establish an expected response rate using a pre -test or data from previous occasio ns of the same or similar surveys, which can in turn be used in sample size determination.
3.2.3 Quality indicators Main quality dimensions and elements: Accuracy – coverage error, sampling error as described previously.
3.3 Questionnaire design Goal: To maximize the comparability of survey questions across different surveys and cultures and reduce measurement error related to questionnaire design.
3.3.1 Quality inputs
3.3.1.1 Survey objectives and research questions.
3.3.1.2 Review of literature and any relevant studies to identify useful material.
3.3.1.3 Documentation templates.
3.3.1.4 Documentation of the origin of any existing questions or materials to be considered for re- use.
3.3.2 Guidelines
3.3.2.1 Design self -completed questionnaires to be attractive and easy to complete, give a positive first impression in the cover letter and front cover and make the questionnaire appear professional and business-like.
3.3.2.2 Choose questionnaire design and wording that encourage respondents to com plete the questionnaire as accurately as possible. The questionnaire must focus on the topic of the survey, be as brief as possible and flow smoothly (including skip patterns) from one question to the next. A Handbook of Quality Guidelines for Statistical Production in Tanzania
26
3.3.2.3 Consulting major data users during the q uestionnaire design process for clear understanding of how the data are to be used. Undertake a review of the existing subject matter literature and surveys both nationally and internationally for a well designed questionnaire that meets the users` needs.
3.3.2.4 In the introduction to the questionnaire, provide the title of the survey, explain the purpose of the survey, identify the sponsor, indicate the authority under which the survey is being executed, the confidentiality protection measures, and reque st the respondent’s cooperation.
3.3.2.5 Harmonize concepts and wording with those already in use, when appropriate re -use questions from other surveys.
3.3.2.6 With respect to questionnaire layout, provide headings for each section of the questionnaire, instructions and answer spaces that facilitate accurate answering of the questions, use colour, shading, illustrations and symbols to attract attention and guide the respondents or interviewers to the parts of the questionnaire that are to be read and to indicate where answers are to be placed. At the end of the questionnaire, provide space for additional comments by respondents and include an expression of appreciation to the respondents.
3.3.2.7 Consider two phases of questionnaire testing (mock intervi ews and pre -testing). This involves testing the questionnaire at an early stage of its development, making revisions to the questionnaire based on the findings, and then testing the revised questionnaire.
3.3.2.8 Hold de -briefing sessions with interviewer s after testing the questionnaire. Let the interviewers discuss their experiences in interviewing respondents and how the questionnaire performed. They can identify potential sources of response and non - response errors as well as areas where the questionnaire can be improved.
3.3.2.9 Conduct pilot-testing after a thorough questionnaire test to observe how all the survey operations, including the administration of the questionnaire and survey logistics work together in practice. The pilot test provides an o pportunity to fine-tune the questionnaire and logistics before their use in the main survey.
A Handbook of Quality Guidelines for Statistical Production in Tanzania
27
3.3.3 Quality indicators Main quality dimensions and elements: Accuracy – measurement error, Relevance and Coherence.
3.4 Translation of survey instruments Goal: To create and follow optimal procedures to standardize, assess, and document the processes and outcomes of survey questionnaire translation.
3.4.1 Quality inputs
3.4.1.1 Source questionnaire and any material to be translated. 3.4.1.2 Templates of translation development, as relevant. 3.4.1.3 Delivery schedule including any further refinements proposed that relate to translation (procedure such as language harmonization, adaptation, pre -testing and any required adjudication steps).
3.4.1.4 Back translation.
3.4.2 Guidelines 3.4.2.1 Create translation team, briefing, training and monitoring. 3.4.2.2 Produce draft translations, checking translator output at an early stage of production. 3.4.2.3 Maintain documentation at each stage. 3.4.2.4 Review and adjudicate the translations. 3.4.2.5 Pre-test the translations. 3.4.2.6 Repeat any translation refinement step as needed.
3.4.3 Quality indicators Main quality dimensions and elements: Accuracy – measurement error and Interpretability.
3.5 Interview recruitment and training Goal: To improve the overall quality of the survey data by minimizing interviewer effects while controlling costs by optimizing interviewer efficiency.
3.5.1 Quality inputs
3.5.1.1 Recruitment and training timeline A Handbook of Quality Guidelines for Statistical Production in Tanzania
28
3.5.1.2 Minimum standards for survey staff employment 3.5.1.3 Study specific requirements (e.g. gender, language, etc) 3.5.1.4 Assessment tests for employment 3.5.1.5 Minimum interviewer requirements checklist 3.5.1.6 Criteria for dismissal of follow-up training
3.5.2 Guidelines 3.5.2.1 Train the trainers before they train the interviewers 3.5.2.2 Complete the checklist during candidate screening 3.5.2.3 Take attendance during the training 3.5.2.4 Certify the candidates 3.5.2.5 Dismiss or retrain candidates who fail certification 3.5.2.6 Maintain written records of the candidates certification tests results
3.5.3 Quality indicators Main quality element: Relevance and Accuracy – measurement error.
3.6 Pre-testing Goal: To ensure that the versions of the survey instruments adeq uately convey the intended research questions, measure the intended attitudes, values, reported facts and behaviours, and that the collections of data are conducted according to the specified study protocols in every survey.
3.6.1 Quality inputs 3.6.1.1 Pre-testing plan, including pretest goals, evaluation techniques, timelines, and budget 3.6.1.2 Standard procedures for cognitive interviews
3.6.2 Guidelines
3.6.2.1 Identify what the pre -test should achieve and choose a pre -test design that best fits the
study goals.
3.6.2.2 Combine pre -testing techniques to create a comprehensive design plan that takes
advantage of the strengths and minimizes the weaknesses of each method.
3.6.2.3 Train or hire staff members who are able to adequately implement the chosen pre-testing
technique(s).
A Handbook of Quality Guidelines for Statistical Production in Tanzania
29
3.6.2.4 Conduct the pre-test in the same mode of data collection (interviewer administered or self-administered) as the main survey. 3.6.2.5 Conduct the pre-test with the same target population as the target population for the survey. 3.6.2.6 Pre-test the survey instrument or part of it in each country and in each language. 3.6.2.7 Document fully, the pretesting protocol and findings.
3.6.3 Quality indicators Main quality dimensions and elements: Accuracy – measurement error.
3.7 Data collection Goal: To achieve an optimal statistical survey data collection design by maximizing the amount of information obtained per monetary unit spent within the allotted time, while meeting the specified level of precision and producing data of good quality.
3.7.1 Quality inputs 3.7.1.1 Target outcome rates (e.g. response, refusal, non contact), and completion rates 3.7.1.2 Target hours per interview 3.7.1.3 Re-contact or re-interview the respondents 3.7.1.4 Percentage of interviewer cases to be verified 3.7.1.5 Verification of questions 3.7.1.6 Interviewer performance checklist
3.7.2 Guidelines
3.7.2.1 Interviewers are critical to the success of most of the data collection. Interviewer manuals and training must be carefully prepared and planned since they provide the best way to guarantee data quality, the comprehension of survey efforts and subject matter, as well as to ensure proper answers to the questions from the respondents.
3.7.2.2 Careful planning of the data collection process should include the establishment of roles and responsibilities regarding all aspects linked to data collection, in order to reduce respondent’s burden and collection cost, and maximize timeliness and data accuracy
A Handbook of Quality Guidelines for Statistical Production in Tanzania
30
3.7.2.3 Establish appropriate sample control procedures for all data collection operations. Such procedures track the status of sampled units from the beginning through the completion of data collection so that data collection managers and interviewers can assess progress at any point in time.
3.7.2.4 Establish and maintain good respondent’s relationships in order to obtain a good response rate. Such measures can include advertising the up-coming survey, an introductory letter to inform the respondents that they will be part of the survey, an informative brochure with key statistics to maintain their interest in participating in the survey and a letter thanking them for their participation. These will help to sensitize the selected units in the sample to participate in the survey.
3.7.2.5 Ensure that the respondent within the responding household is contacted at the appropriate time so that the information is readily available. Allow the respondents to provide data in a method and format that is convenient to them. This will help to increase response rates and improve the quality of the information obtained from the respondents.
3.7.2.6 Tracking should be conducted to locate and contact the respondents when the available contact information on the survey unit is likely to be out-dated. Tracking increases response rate and also helps in determining if the sampled unit is still in the scope.
3.7.3 Quality indicators Main quality dimensions and elements: Accuracy – Non response error, Timelines and Punctuality and Interpretability.
3.8 Data processing and statistical adjustment Goal: To code and capture data from their raw state to an edited data file that can be used within the survey organization for quality assessment of the survey implementation and harmonized with other surveys' data files in preparation for statistical adjustment, dissemination, and eventually substantive research.
3.8.1 Quality inputs
3.8.1.1 Percentage of manually entered questionnaires to be verified