Evaluation of reference models

Evaluation of reference models

1. Conceptual bases

   • Quality criteria
     • Ad-Hoc
       • Plain text based
         • Easy to perform
         • Discusses the characteristics, strengs and weakness of a reference model verbally, evaluation more o less structured. It does not use specific characteristics
       • Feature based
         • Weakness: Evaluation and selection of specific criteria is subjective
         • Define a set of features which can be used to describe and characterize reference models. The feature set is introduced ad-hoc and not derived from theory
       • Metric-Based Evaluation
         • Metrics define how to measure characteristics of a reference model.
           • Metric types
             • Metrics for the modeling process
               • Adoption guide
             • Metrics for the product of the modeling process
               • Softwar product line adoption inside an enterprise
           • This approach assumes that each metric is operationalized and can be determined objectively
           • Problem:the objective measurement of the introduced metrics. While determining some metrics is easy (e. g. number of entity-types as a measure for the model size), the measurement of other aspects is difficult (e. g. adaptability or usability of a reference model)
       • criteria are just introduced for the purpose of the evaluation approach without referring to a specific reference theory
     • Theory-Based Evaluation
       • Criteria are derived from and founded on a specific theory, a so-called reference theory
       • Types
         • Metamodel
           • It is possible to check whether the reference model is syntactically correct
           • The strength of this approach is that it is based on clear quality criteria, though it allows just a technical evaluation.
         • Paradigmatic Evaluation
           • Analyzes the meta-theoretical assumptions of a reference model
           • aspects
             • ontological
             • epistemological
             • linguistic
             • contextual
         • Master Reference Model-Based
           • a master reference model allows to evaluate reference models. We define a master reference model as a reference model that does not represent a specific class of domains, but all classes of enterprise domains.
           • This evaluation approach analyzes whether the facts represented in a reference model are equivalent to the representation in the master reference model.
           • Problems
             • How the quality of the master reference model is ensured
             • Currently no master reference model for our adoption guide is known
   • Based on
     • Fettke, P. & Loos, P., 2003. Multiperspective Evaluation of Reference Models – Towards a Framework. Conceptual Modeling for Novel Application Domains: ER 2003, 2814, pp.80–91. Available at: http://dx.doi.org/10.1007/978-3-540-39597-3_9.
   • Research method
     • Analytical
       • Based on logical conclusions
     • Empirical
       • Based on exeperiences
         • Survey, laboratory, experiment, field sutdy case duty, action reserch
   • Evaluation point of view
     • user-oriented reference model should be flexible and adaptable, so its fitness for the intended application is very high.
     • enterprise-oriented point of view, the purchase and usage of a reference model should make the development of enterprise systems more efficient.
     • Economical oriented Reference models should increase the productivity of the national economy.
     • science-oriented point of view, a reference model should represent a specific industry-type completely, precisely, consistently and correctly