The deliverable D7.8: Data Collection and Management Guideline Report was submitted by Future Analytics Consulting, with contributions from NTNU, LCCC, TK, MAI, MP, SB, SMO, VORU, IESRD, POW, EAP, UL, R2M, and ISOCARP in May 2021. The executive summary of the deliverable is available below and the full deliverable at the end for download:
“Task 7.4 sets out the requirement to create practical recommendations and guideline reports which detail, analyse and interpret the aggregated data collected during the M&E process, and deliver technical recommendations related to the Information and Communication Technologies (ICT), energy, community, mobility, environmental and regulatory interventions implemented in the project. Deliverable 7.5 provided initial guidelines for these processes to which future iterations of the report would be structured. The guidelines provided in D7.5 were refined in this document to develop a framework for the evaluation of project activities and interventions, and set out a way to report on the feedback and provide actionable guidelines and recommendations.
The implementation of project interventions and initiatives generates data that is captured by project partners. Through the KPI framework developed for the project, partners are able to capture specifically defined quantitative KPI-related data that would suit the requirements in the calculation of the KPI. The consistent capturing and sharing of data to the Monitoring and Evaluation Reporting Tool (MERT) enables accurate capturing and processing of quantitative KPI data which, alongside broader quantitative, qualitative and experiential data collected throughout the overall project work, is used in the analysis and derivation of insights.
Mainly, partners will be using the manual data upload process through the individual interface of each KPI in the MERT. This might cause additional workload for partners until the automated process has been enabled. A limited number of partners currently share data through the automated process using Application Programming Interface (API) connections between partner and MERT servers. The way data is shared is not only dependent on the type of data available, but also on when the data is available. Uncertainty relating to the sharing and availability of data is being worked on.
A first level of insights derived from project data is based on the analysis of the KPI monitoring data, compared to the baseline situation and the expected and targeted impact for the indicator. The comparison of these points makes it possible to assess and track the indicators’ performance over time to identify trends, track growth, and highlight significant changes, from which quantitative insights can be formulated. The insights drawn from the data are then used to formulate recommendations on M&E topics such as data monitoring, data modelling, evaluation of the expected impact, performance of the indicator, and technical points regarding the implementation of a particular indicator.
To get a better understanding of the overall impact of project activities or interventions it is important to have partners’ feedback and input that describe the quantitative data and other impacts or results achieved. Feedback will be sourced from partners through various processes, including feedback forms, interviews, learning sessions, informal discussions, and the review of relevant project documentation. The monitoring data will be used together with the additional information provided by the project partners and the intervention results documented in formal project deliverables, to derive insight and inform recommendations. As data capturing and management using online ICT systems and tools such as the +CityxChange MERT and Smart Cities Marketplace (SCM) (formerly known as the Smart Cities Information System (SCIS)) Self Reporting Tool (SRT) will continue throughout and beyond the project lifecycle, it is important to consider the best possible way of integration and interoperability between these systems. Seamless interoperability (as documented in Deliverable 1.31: Report and catalogue on the ICT data integration and interoperability) will ensure efficient use of the available systems. In support of this ideal, the MERT will be reviewed periodically to determine whether all functionalities are still fit for purpose. In addition, other tools such as the +CxC Decision Support Tool (DST) and the SRT will be reviewed alongside the MERT to determine the level of possible interoperability and ease of use for the systems.
The ongoing sharing and analysis of data, engagement with partners, and use of data management and modelling tools will result in suggestions for improvement in efficiency of execution and processes being generated. The extraction of insights and development of recommendations are therefore not only based on quantitative data, but also the coherent insight and additional validating information informed by it.
Ongoing collaboration between Work Packages 7, 8, 9 and other partners will refine a framework for the accumulation of quantitative and qualitative data and information. The framework will set out the process for the identification of project activities to be evaluated, and demonstrate the evaluation procedures (feedback forms, workshops, interviews, informal interactive group discussions, review of relevant project documentation) that can be applied in the evaluation process. The application of the framework will ensure a collaborative methodology to analyse and interpret the evaluation of project activities to enable the extraction of useful insights and recommendations to inform future decision-making in the project. This framework development is described in this report.”
