42
Programme implementation (GSBPM Phases 4 to 7)(ct’d) Quality Element UN NQAF Element Link to existing policy, guidelines or practice How the quality element is assured and recommendation (if any) 4.7.7 Statistical discrepancies in intermediate data are assessed and investigated
15.15
SMSG
Documentation is needed
4.7.8
Preliminary and revised data are clearly identified.
15.16
SMSG and June 2010
Dissemination and
Pricing Policy
Documentation is needed
4.7.9
Information on the size and direction of revisions for
key indicators is provided and made public
15.17
June 2010
Dissemination and
Pricing Policy.
Needs to be developed 4.8: Making sure that the statistics produced by the programme are timely and available to users with a minimal delay
4.8.1 Agreements are made with data providers about the schedule for the delivery of data
16.16 Not yet in place This needs to be developed and extended to all data providers 4.8.2 Procedures exist that ensure the effective and timely flow of data from providers.
16.17 Not documented This is done on an ad-hoc basis and a more systematic approach is needed 4.8.3 When preliminary data are released, they are clearly identified as such, and users are provided with appropriate information to be able to assess the quality of the preliminary data.
16.19 June 2010 Dissemination and Pricing Policy This is done but a more systematic approach is needed 4.8.4 A published policy exists that describes the revisions for those key outputs that are subject to scheduled revisions.
16.20 June 2010 Dissemination and Pricing Policy. Needs to be developed 4.9: Making sure that the statistics produced by the programme are accessible and clear to users 4.9.1 The mix of printed publications, electronic releases and data available on request is considered to be appropriate, given the needs of the principal users
17.22 June 2010 Dissemination and Pricing Policy This is a practice
43
Programme implementation (GSBPM Phases 4 to 7)(ct’d) Quality Element UN NQAF Element Link to existing policy, guidelines or practice How the quality element is assured and recommendation (if any) 4.9.2 Statistics are presented in a clear and understandable manner 17.23 June 2010 Dissemination and Pricing Policy This is a practice 4.9.3 Explanatory texts that accompany the data are reviewed for clarity and readability 17.24 June 2010 Dissemination and Pricing Policy Documentation is needed 4.9.4 Meaningful comparisons are included in the publications when appropriate 17.25 June 2010 Dissemination and Pricing Policy
This is a practice 4.9.5 Users are informed about the methodology of the statistical processes and the use of administrative data 17.26 SMSG Documentation is needed 4.9.6 Different levels of metadata detail are made available to users to meet their requirements 17.27 June 2010 Dissemination and Pricing Policy
This is done but a more systematic approach is needed 4.9.7 Policies are in place for archiving statistics and metadata 17.30 June 2010 Dissemination and Pricing Policy
Documentation is needed 4.10: Making sure that the statistics produced by the programme are coherent and can be compared with other sources and over time 4.10.1 Specific procedures and guidelines are in place to ensure that outputs are internally coherent
18.14 SMSG Documentation is needed 4.10.2 Changes in methods are clearly identified and measured to facilitate reconciliation
18.15 SMSG Documentation is needed 4.10.3 Breaks in the series are explained and the methods for ensuring reconciliation over a period of time are made publicly available 18.16 SMSG This is done on an ad-hoc basis and a more systematic approach is needed
44
Part 5: Post collection and evaluation (GSBPM Phases 8 and 9) Part 5 of the Framework looks at how quality is maintained and improved for activities within individual statistical programmes at the post collection stage. These activities are mostly concerned with the archiving of data and metadata and evaluating programmes from time to time. There are 22 quality elements divided into 8 categories or sections as follows. 5.1 Evaluating the usefulness of the programme; 5.2 Evaluating the standards used for the programme; 5.3 Monitoring the use of micro-data, especially when these have been made available for further research and analysis outside the agency; 5.4 Evaluating the quality of the statistical programme; 5.5 Evaluating the relevance of the statistical programme; 5.6 Evaluating the accuracy and reliability of the data generated by the statistical programme; 5.7 Evaluating the timeliness and accessibility of the data generated by the statistical programme; 5.8 Evaluating the extent to which the statistics produced by the programme are coherent and can be compared with other sources and over time.
45
Post collection and evaluation (GSBPM Phases 8 and 9) Quality Element UN NQAF Element Link to existing policy, guidelines or practice How the quality element is assured and recommendation (if any) 5.1: Evaluating the usefulness of the programme 5.1.1 It is possible for stakeholders to evaluate whether a statistical product is needed
2.28 User satisfaction surveys This is done on an a regular basis, but a more systematic approach is needed 5.2: Evaluating the standards used for the programme 5.2.1 Statistical standards are reviewed regularly and revised, if necessary, to ensure that they remain relevant, coherent and clear
3.21 CDOS Documentation is needed 5.3: Monitoring the use of micro-data 5.3.1 The use of microdata sets are monitored to identify whether data confidentiality may have been compromised and, if so, immediate action is taken to redress the situation
7.14 June 2010 Dissemination and Pricing Policy This is not yet done, but is important 5.4: Evaluating the quality of the statistical programme 5.4.1 Metadata and indicators of quality are provided to users to help them assess the quality of the released data
8.17 SMSG This is done on an ad-hoc basis and a more systematic approach is needed 5.4.2 Where relevant, outside experts are called in to conduct quality reviews
8.18 SMSG Documentation is needed 5.4.3 Mechanisms are in place to collect and follow up on users’ reactions and feedback
8.19 TSMP and User satisfaction surveys This is done but a more systematic approach is needed 5.4.4 User satisfaction surveys are implemented regularly, and their results are made public
8.20 User satisfaction surveys Documentation is needed 5.4.5 The costs of the statistical production process are accurately assessed 9.12 SMSG Documentation is needed
46
Post collection and evaluation (GSBPM Phases 8 and 9)(ct’d) Quality Element UN NQAF Element Link to existing policy, guidelines or practice How the quality element is assured and recommendation (if any) 5.4.6 Cost-benefit analyses are conducted
9.13 SMSG Documentation is needed 5.4.7 Follow up with users is carried out to seek their views on the extent to which the statistical outputs meet their needs
10.28 User satisfaction surveys This is done but a more systematic approach is needed 5.5: Evaluating the relevance of the statistical programme
5.5.1 Assessments are undertaken to find out if there were any problematic aspects of the questionnaire design and its implementation.
13.11 Ad hoc This is done on an ad-hoc basis and a more systematic approach is needed 5.5.2 Where relevant, post-collection evaluations are carried out to obtain feedback that can be used in the planning for future such collections.
14.20 SMSG This is done but a more systematic approach is needed 5.5.3 Action plans are prepared and implemented to improve relevance and meet emerging needs
14.21 SMSG Documentation is needed 5.6: Evaluating the accuracy and reliability of the data generated by the statistical programme
5.6.1 Any errors that are discovered in published statistics are corrected at the earliest possible date and are publicized
15.18 SMSG This is a practice 5.6.2 An analysis of revisions is performed and used to improve the statistical process.
15.19 Not yet in place Needs to be developed 5.6.3 Sampling and non-sampling errors are analysed from time to time and improvement actions are taken as a result
15.20 SMSG This is done partially, a more systematic approach is needed
47
Post collection and evaluation (GSBPM Phases 8 and 9)(ct’d) Quality Element UN NQAF Element Link to existing policy, guidelines or practice How the quality element is assured and recommendation (if any) 5.7: Evaluating the timeliness and accessibility of the data generated by the statistical programme
5.7.1 Indicators of timeliness and punctuality are calculated regularly and are monitored, published and followed up 16.21 SMSG This is done on an ad-hoc basis and a more systematic approach is needed 5.7.2 Assessments are carried out from time to time to ensure that the arrangements for disseminating official statistics are meeting user needs
17.31 TSMP and User Satisfaction Survey Documentation is needed 5.7.3 User-oriented quality reports are made available to keep users informed about the quality of the statistical outputs
17.32 Not in place Needs to be developed 5.8: Evaluating the extent to which the statistics produced by the programme are coherent and can be compared with other sources and over time 5.8.1 Effects of changes in methods on published statistics are assessed and appropriate information is provided to users
18.17 SMSG and CDOS This is the practice, but more documentation is needed 5.8.2 Statistical outputs are compared with other data sources and divergences are identified and explained to users
18.18 SMSG and CDOS This is done but a more systematic approach is needed 5.8.3 Quality reports include a section on internal consistency and comparability over time and make comparisons with other subject matter areas 18.19 SMSG and CDOS This is done on an ad-hoc basis and a more systematic approach is needed
48
Attachment 1: Overview of the Elements of the UN Generic National Quality Assessment Framework NQAF Topic National Statistical System Agency level Programme design Programme implementation Post- collection evaluation
GSBPM 1 - 3 GSBPM 4 - 7 GSBPM 8 - 9 NQAF1 Coordinating the NSS 1.1 - 1.10
NQAF2 Relationships with data users and providers
2.1 - 2.18 2.19 -2.24 2.25 - 2.27 2.28 NQAF3 Managing statistical standards
3.1 - 3.6 3.7 - 3.11 3.12 - 3.20 3.21 NQAF4 Professional independence
4.1 - 4.10
NQAF5 Impartiality and objectivity
5.1 - 5.15
NQAF6 Transparency
6.1 - 6.5
NQAF7 Statistical confidentiality and security
7.1 - 7.6 7.7 - 7.8 7.9 - 7.13 7.14 NQAF8 The quality commitment
8.1 - 8.11 8.12 - 8.15 8.16 8.17 - 8.20 NQAF9 Adequacy of resources
9.1 - 9.6 9.7 - 9.9 9.10 9.11 - 9.13 NQAF10 Methodological soundness
10.1 - 10.10 10.11 - 10.18 10.19 - 10.27 10.28 NQAF11 Cost-effectiveness
11.1 - 11.11 11.12 - 11.14 11.15 - 11.17 11.18 - 11.19 NQAF12 Soundness of implementation
12.1 - 12.5 12.6 - 12.16 12.17 - 12.28
NQAF13 Managing the respondent burden
13.1 - 13.5 13.6 - 13.9 13.10 13.11 NQAF14 Relevance
14.1 - 14.10 14.11 - 14.15 14.16 - 14.19 14 20 - 14.21 NQAF15 Accuracy and reliability
15.1 - 15.6 15.7 - 15.8 15.9 - 15.17 15.18 - 15.21 NQAF16 Timeliness and punctuality
16.1 - 16.10 16.11 - 16.15 16.17 - 16.20 16.21 NQAF17 Accessibility and clarity
17.1 - 17.17 17.18 - 17.21 17.22 - 17.30 17.31 - 17.32 NQAF18 Coherence and comparability
18.1 - 18.9 18.10 - 18.13 18.14 - 18.16 18.17 - 18.19 NQAF19 Managing metadata
19.1 - 19.6