CYOpenData_Lisbon_2018%20-%20Final.pdf

Open Data in Cyprus The National Open Data Portal Lisbon, 27th September 2018 Dimitris Michail (PAPD) Christos Rodosthenous (OUC) CYPRUS

 Population: 840,000  Independent since 1960  EU member since 2004  Open Data Awareness: Medium  First PSI re-use Law: 2006  Current law: voted in Dec. 2015  National Open Data Portal re- launched in 2018 (dkan) The Republic of Cyprus

The Cyprus PSI Team Dimitris Michail (PAPD) Mantalena Tsoukka (PAPD) Christos Rodosthenous (OUC) Michalis Epiphaniou (OUC)

Main Challenges and Advantages of a Small Country  Limited Resources  human  financial  Low Demand  small population  small local market  few opportunities for commercial use  Very small or no ecosystem around open data  Absence of Local Initiatives / Portals  Absence of comprehensive FOI Law (Cyprus)  coming into effect in 2020  Centralized Government System  ease of communication across government  relatively easy to form networks  Small Population → easier to reach / inform  Large number of universities (8)
compared to population  Mainly Services oriented economy  open to initiatives that enhance business environment Main Challenges / Barriers Main Advantages

Fostering the use of open data | Hackathon 2.0 5 transposed to the “Re-use of public sector information Law N. 132(I)/2006” PSI Directive (2003/98/EC) Launch the 1st edition of National OD Portal Revision of the National PSI Law in line with the revised PSI Directive New National PSI Law May 2018 Revamped National OD portal launch PAPD commissions Deloitte Cyprus to study the potential impact of PSI re-use in Cyprus and develop a road map for the practical implementation of the PSI Directive. The results of the study formed the basis for the 2012-2016 National Action Plan launched the following year. Socioeconomic Study National Action Plan 2012-2016 Open Data Forum June 2016 2006 2012 2015 2018 2011 2014 2016 The project funded by the European Commission’s Structural Reform Support Service (SRSS) to assist the PSBs in implementing the revised directive 2013/37/EU and promote public sector information reuse to the public and private/ business sector in Cyprus. CY PSI Project Workshops for the public sector July 2016 1st Open Data Hackathon in Cyprus September 2016 2nd Open Data Hackathon in Cyprus September 2018 2017 National Action Plan 2017-2021 Open Data in Cyprus o Legislation o Infrastructure (portal) o Provision of Training to PSBs o Promotion of re-use

Situation Analysis
Identification of Users’ Requirements Implementation Design based on OS Software Finalised Partnership with OUC formed March 2017 June 2017 January 2018 October 2017 September 2017 Developing the New Open Data Portal

Situation Analysis www.data.gov.cy Situation Analysis (2017)  CY NODP developed in-house by DITS between 2014-2016  Hosted on CY GOV Network  In general it served the purpose it was built for  As demand for open data increased various issues arose:  Relatively slow navigation  Limited Filters for data search  Absence of Datastore API (only metadata API)  Absence of data visualization tools

Identification of User Requirements Improvements Infrastructure  Network speed  Access from everywhere  Flexibility Infrastructure  Extendibility  Scaling Improvements Functionality  Accessibility  Visualizations  Publish Workflow Process  APIs  Supported metadata schema Improvements Training/Monitoring  Workshops  Webinars  Surveys

Partnership with the OUC The Open University of Cyprus Public University Distance Learning State-of-the-art eLearning Platform Support FOSS Open Technologies Access to Academic Network Advanced IT Infrastructure September 2017 Memorandum of Understanding

Searching for a suitable solution Selection criteria:  Meet the requirements  License-free  Promote openness  Successful deployments  Technologies that can be supported by the team Why DKAN Feature-rich Opensource - GNU General Public License v2.0 Based on a popular CMS (Drupal) Deployed in many Countries and Large Organizations Technologies that can be supported by the team Support team expertise Readymade API infrastructure Visualizations Modular architecture

DKAN Installations

92 documented worldwide installations Source: http://docs.getdkan.com/en/latest/introduction/dkan-sites.html

DKAN Open Data Platform-Features DKAN is a community-driven, free and open source open data platform that gives organizations and individuals ultimate freedom to publish and consume structured information.

DKAN Open Data Platform-Support Community

The OUC Infrastructure 100Mbps [1Gbps] CYNET Academic Network / GEANT

DKAN Installation Current deployment state:  DKAN 1.15.3  Drupal 7.59  Ubuntu 16.04 LTS  Apache 2.4  MariaDB  PHP 7

The Cyprus Portal Development Process Design Finalization Implement Infrastructure Launch Setup Production data.gov.cy Setup Sandbox October 2017 November 2017 May 2018 January 2018 December 2017 Data Migration Data Validation December 2017 January 2018

18  Format Transformation - Compatibility with Technical Framework o Machine Readable o Open Formats CSV instead of XLS, PDF, DOC 480 datasets 1350  700 datasets Carried out by the PSI Team (not an automated process)  Assessment of all published datasets o Approve / Discard o Organize (Merged to one or Split to many) o Checks for Metadata Completeness +300 datasets Identification / Preparation & Publication of New Datasets Migration of Data and Open Formats The Cyprus National Open Data Portal Frontpage • Dataset Topics • Information Dataset • Visualization • Search APIs • Datastore • Other Backend • Administrative • Publish Workflow Support • Documentation • Training material

The Cyprus National Open Data Portal - FrontPage List of Topics (12) Datastories News Feed and social media Language Selection Direct links training and support

The Cyprus National Open Data Portal - Datasets  Datasets [1041]  From 87 Public Sector Organizations  Resources [2691]  Majority CSV  Machine-readable  JSON  XML

Organizing and searching datasets Filter data per:  Topic  Organization  Format  Tag  License Type

Dataset Metadata DCAT-AP Machine-readable  JSON  RDF

Visualizing Data Using the Portal Interface Grid format Graph format
Map format

Share datasets and visualizations in Datastories

Access to Data and Metadata using APIs The data.gov.cy provides access to the portal data and metadata using four (4) APIs:  Datastore API: The DKAN Datastore houses any CSV files that have been uploaded and then imported into the Datastore and provides an endpoint to access them  CKAN API: Harvest and index the Portal from other CKAN-based portals (CKAN harvester)  Open Data APIs: Two Open Data Standards APIs: DCAT-AP and Project Open Data  Dataset REST API: Access all Portal content nodes

Datastore API - Endpoint

Datastore API - Parameters Parameters •resource_id (mixed) – id (string) or ids (array) of the resource(s) to be searched against. •filters (mixed) – array or string of matching conditions to select •q (string) – fulltext search •offset (int) – offset this number of rows •limit (int) – maximum number of rows to return •fields (array or comma separated string) – fields to return (default: all fields in original order) •sort (string) – comma separated field names with ordering •join (array) – array of fields to join from multiple tables •group_by (array) – array of fields to group by Aggregation functions •sum (string) – field to compute the sum •avg (string) – field to compute the average •min (string) – field to compute the maximum •max (string) – field to compute the minimum •std (string) – field to compute the standard deviation •variance (string) – field to compute the variance Source: https://docs.getdkan.com/en/latest/apis/datastore-api.html

Datastore API - Example https://www.data.gov.cy/api/action/datastore/search.json?resource_id=b0de1b05-88e7-4350-96d0-ad1266bf2b35&limit=5&language=en

Administration and Backend  User Permission Controls  Administrators (OUC & PAPD)  Public sector users (publishers and moderators)  Public (Registered users)  Simple forms to add/update Datasets and Resources

Publish Workflow Roles  Contributors (each organization)  Moderators (currently very few)  Super Moderator (PAPD) Organizations www.data.gov.cy Approve and Publish Reject and Review Submit dataset and send for approval

Monitoring and Logging • Notification system (email)  Error reporting  Broken link reporting  Custom reports (e.g., non-updated datasets)

Portal Statistics (Period: 5 May 2018 – 21 September 2018) Visitors: 14,600 (4,171/month) Visitors’ Countries : 90 % Visitors from Abroad : 32% Page Views = 103,000 Page Views per Day = 755 % Access via API = 20%

Portal Statistics (Period: 5 May 2018 – 21 September 2018)

Support, Documentation and Training Guides for PSBs  Data Selection  Data Preparation  Publishing Datasets  Utilizing the DS APIs  Roles and Levels of Authorization  Creating CSVs Developers Corner  Data Cleaning Tools  Programming Languages  Base Maps  Visualization Tools  DKAN Guide  DKAN API Guides Technical Guidelines  Data Selection  Licensing  Charges  Formats  Metadata  Standards Other Material  Guides  Information on DCAT-AP  Presentations from Events/ Conferences  Videos Support, Training and Promotion

Working with DKAN  Powerful Tool  Easy to set up/support/upgrade  Try to stay close to the core  Good Documentation  Easily Extendable  Bugs (found <10 in 1 year)  Very good community support  Issues Resolved Quickly (sometimes within days)  Users engaged in prioritizing further dkan development  CY Team among contributors

What's Next  Dataset statistics dashboard  Extend search capabilities for datastores  Translations (metadata, etc.)  Create webinars for the public  Create video guides for PSBs  Increase engagement using datastories  Challenges with prizes (substitute hackathons) Examples of Datasets

Location and Measurements of E/M Radiation of Radio-communication Stations Publisher: Department of Electronic Communications Themes: Science and Technology Formats: CSV, API (json) Frequency: bi-annual Tags: EM radiation

Real-Time Recordings of Meteorological Stations Publisher: Department of Meteorology Themes: Science and Technology Formats: API (XML) Frequency: Hourly Related datasets: Weather Forecasts by Area

Real-Time and Historical Recordings of Air-Quality Stations (3 datasets) Publisher: Department of Labour Inspection Themes: Environment Formats: API (xml) – real time data CSV, API (json) – historical Frequency: Hourly, Daily

Companies, Partnerships and Business Names Registry Publisher: Department of Registrar of Companies and Official Receiver Themes: Economy and Finance Formats: CSV, API (json) Frequency: Monthly Related Datasets: statistics on registered companies, trade marks, etc

Public Transport Datasets Collection of 7 datasets (one per Bus Service Operator) that include:  OSEL  EMEL  OSYPA  OSEA Publisher: Departments of Public Works Themes: Transport Formats: csv, shp, kml, json EMEL Routes (shp)  ZINON  Nicosia Small Buses  INTERCITY Collection of Geospatial Data Collection of 200+ Layers from the:  CY Cadastral Map  CY Topographic Map  CY Hydrographic Map  CY Administrative Map + CSS, hospitals, fishing shelters, natural trails, picnic sites, etc…. Publisher: mainly Department of Lands and Surveys Themes: Regions, Cities and Zoning, Environment Formats: wms, wfs, gml, shp, kml

CY NODP Satisfaction Survey (conducted between 15-24 September) System Usability Scale (SUS)=Good (72)

Dimitris Michael Open Data Team Department of Public Administration and Personnel Ministry of Finance Tel: +357 22 60 1523 / 1515 | Fax: +357 22 602763 Web: www.data.gov.cy | Twitter: @OpenDataCY Christos Rodosthenous Open Data Team Information, Technology and Library Unit Open University of Cyprus Tel: +357 22 411783 Web: www.data.gov.cy | Twitter: @OpenDataCY Thank you…