Are there differences between behavioral measurement methods? A comparison of the predictive validity of two ratings methods in a working dog program

By E. Wilsson, D. Sinn. 2012

88 citations recorded by the source corpus.

Abstract Consistent behavioral variation within and between individuals is ubiquitous in all working dog populations. Most working dog programs have recognized this fact, and have subsequently attempted to quantify behavior through the use of standardized tests. Standardized tests may employ several measurement methods, but two common ones are behavioral ratings and subjective ratings. The former is characterized by a rating for a behavior (e.g., reaction to a noise) usually based on a single observation or test; the latter is characterized by a rating for a trait (e.g., confidence) that is based across multiple observations of behavior. The main difference between the two rating methods is the level of aggregation, or intuition, that is required by the human observer. Measurement theory predicts that ratings based on multiple observations (i.e., subjective ratings) should be more reliable because measurement error is reduced. However, subjective ratings, by definition, may be susceptible to observer bias, in which case ratings based on fewer, but better-defined observations (i.e., behavioral ratings) could result in greater reliability. In either case, the ultimate criterion of most working dog programs is the predictive validity of measured behaviors in standardized tests. To the best of our knowledge, the relative predictive validity of subjective and behavioral ratings has yet to be tested within the same working dog population. Here we analyzed behavioral test results along with training outcomes from a large sample (∼400) of German shepherd dogs, aged 15–18 months, all bred at the Swedish Armed Forces breeding kennel. Behaviors observed in the test were measured using 25 behavioral and 13 subjective ratings. Data reduction and confirmatory techniques identified five underlying dimensions in behavioral ratings (confidence, physical engagement, social engagement, aggression, and environmental sureness) and three in subjective ratings (engagement, confidence, and aggression). Both ratings methods correctly classified a high percentage of dogs that did/did not complete training (70.3–78.3%). However, only minor differences in predictive validity were observed between the two measurement methods (1.7–6.6%). Engagement and confidence, irrespective of measurement method, were the strongest predictors of training completion, but the two ratings methods identified different aspects of engagement and confidence that may be important to training outcomes in this and other working dog programs. Taken together, our results suggest that in some cases, the use of subjective versus behavioral ratings may be inconsequential from the standpoint of prediction to training criterion. Further empirical verification is needed, along with improvements in the explicit definition and measurement of ‘success’ in this and other working dog programs.

Subfields: Experimental and Cognitive Psychology, Genetics, Small Animals

View Paper: https://www.semanticscholar.org/paper/a99ccc294e62976c83ec2a65ba53b2da58073702

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Experimental and Cognitive Psychology

In Genetics

In Small Animals

Citations in the corpus

Minimizing fear and anxiety in working dogs: A review (2016, background): “The DMA (Wilsson and Sinn, 2012) used in Scandinavia presents a stranger, a dark room, loud metallic noises, an unstable surface, dummies, human “ghosts” and gunfiredall of which are potentially fear-provoking.”
Behaviour and experiences of dogs during the first year of life predict the outcome in a later temperament test (2014, methodology): “Nevertheless, there is considerable agreement between this and previous analyses of the same behaviour test as the one used here (Wilsson and Sinn, 2012), where the PCA generated five PCs with eigenvalues >1, named “Confidence”, “Physical Engagement”, “Social Engagement”, “Aggression” and…”