| Work package | WP5 - Evaluation |
|---|---|
| Dissemination level | Restricted |
| Delivery date | M46 |
In the culmination of the NAVMAT project, the second and final evaluation stage employs cutting-edge performance metrics akin to those utilized in contemporary information retrieval systems. This evaluation scrutinizes the efficacy of keyword and incident similarity searches, employing a multi-user engagement approach for comprehensive assessment. By leveraging state-of-the-art metrics, the evaluation aims to gauge the system's efficiency and effectiveness in delivering accurate results, thus contributing valuable insights into its real-world applicability and user satisfaction.
The evaluation reveals promising results, particularly in the performance of the ranking algorithm as indicated by the normalized Discounted Cumulative Gain (nDCG) statistics, showcasing highly relevant items positioned towards the top of the list. Similarly, the Spearman metric also demonstrates a positive correlation between the top retrieved documents and their ideal re-orderings provided by the domain experts. The free term evaluation indicates that overall, the domain experts were able to retrieve the incidents, reports, and publications that correspond to specific ontology terms (and their synonyms). The value of incorporating the domain-oriented knowledge to the NAVMAT system was proven by testing the performance of the system, which, after ceasing to leverage domain-oriented knowledge (by removing some of the AI capabilities), dropped significantly.