Measuring Models

By Monperrus Martin, J. Jézéquel, J. Champeau, B. Hoeltzener. 2009

Model-Driven Engineering (MDE) is an approach to software development that uses models as primary artifacts, from which code, documentation and tests are derived. One way of assessing quality assurance in a given domain is to deﬁne domain metrics. We show that some of these metrics are supported by models. As text documents, models can be considered from a syntactic point of view i.e., thought of as graphs. We can readily apply graph-based metrics to them, such as the number of nodes, the number of edges or the fan-in/fan-out distributions. However, these metrics cannot leverage the semantic structuring enforced by each speciﬁc metamodel to give domain speciﬁc information. Contrary to graph-based metrics, more speciﬁc metrics do exist for given domains (such as LOC for programs), but they lack genericity. Our contribution is to propose one metric, called σ , that is generic over metamodels and allows the easy speciﬁcation of an open-ended wide range of model metrics.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Many-shot Jailbreaking (2024, background): “We observe that in-context learning follows predictable power laws on the following safety-unrelated datasets: LogiQA (Liu et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), TriviaQA (Joshi et al., 2017), Common-senseQA (Talmor et al., 2019), CREAK (Onoe et al.,…”
Steering Llama 2 via Contrastive Activation Addition (2023, background): “It is possible that better steering effects could be achieved via prompting alone if more effort were applied to finding effective prompts.”
Zephyr: Direct Distillation of LM Alignment (2023, methodology): “We also evaluate Z EPHYR -7B on the Open LLM Leaderboard (Beeching et al., 2023), which measures the performance of LMs across four multiclass classification tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and Truthful QA(Lin et al., 2022).”
Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation (2025, unknown): “While several NLP datasets are designed for multiple choice-style tasks (Joshi et al., 2017; Rajpurkar et al., 2016; Lin et al., 2022; Reddy et al., 2019), the options for open-ended long-form responses are much more limited.”