Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review.

By R. Pace, Pierre Pluye, G. Bartlett, A. Macaulay, J. Salsberg, J. Jagosh, Robbyn Seller. International Journal of Nursing Studies, 2012

1,210 citations recorded by the source corpus.

BACKGROUND Systematic literature reviews identify, select, appraise, and synthesize relevant literature on a particular topic. Typically, these reviews examine primary studies based on similar methods, e.g., experimental trials. In contrast, interest in a new form of review, known as mixed studies review (MSR), which includes qualitative, quantitative, and mixed methods studies, is growing. In MSRs, reviewers appraise studies that use different methods allowing them to obtain in-depth answers to complex research questions. However, appraising the quality of studies with different methods remains challenging. To facilitate systematic MSRs, a pilot Mixed Methods Appraisal Tool (MMAT) has been developed at McGill University (a checklist and a tutorial), which can be used to concurrently appraise the methodological quality of qualitative, quantitative, and mixed methods studies. OBJECTIVES The purpose of the present study is to test the reliability and efficiency of a pilot version of the MMAT. METHODS The Center for Participatory Research at McGill conducted a systematic MSR on the benefits of Participatory Research (PR). Thirty-two PR evaluation studies were appraised by two independent reviewers using the pilot MMAT. Among these, 11 (34%) involved nurses as researchers or research partners. Appraisal time was measured to assess efficiency. Inter-rater reliability was assessed by calculating a kappa statistic based on dichotomized responses for each criterion. An appraisal score was determined for each study, which allowed the calculation of an overall intra-class correlation. RESULTS On average, it took 14 min to appraise a study (excluding the initial reading of articles). Agreement between reviewers was moderate to perfect with regards to MMAT criteria, and substantial with respect to the overall quality score of appraised studies. CONCLUSION The MMAT is unique, thus the reliability of the pilot MMAT is promising, and encourages further development.

Subfields: Aerospace Engineering, Applied Mathematics, Biomedical Engineering

View Paper: https://www.semanticscholar.org/paper/1079a0b445f682b2343f745406d72537b67de1e3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study. (2019, background): “Previous studies on the interrater reliability of the MMAT reported that agreement scores ranged from poor to perfect [12,13].”
Factors influencing acceptance of technology for aging in place: A systematic review (2014, background): “It was decided not to exclude articles based on quality assessment because there is little empirical evidence on which to base exclusion decisions in mixed studies systematic reviews [35–37].”
The relationships between school climate and adolescent mental health and wellbeing: A systematic literature review (2018, positive): “The Mixed Methods Appraisal Tool (MMAT), developed by Pace et al. (2012) (see also Souto et al., 2015) was used for this purpose.”
What cognitive mechanisms predict travel mode choice? A systematic review with meta-analysis (2017, methodology): “We therefore applied three criteria that were highlighted across six previous studies recommending bias assessment in correlational studies (Effective Public Health Practice Project [EPHPP], 1998; Gauthier, 2003; NHLBI, 2014; Pace et al., 2012; von Elm et al., 2007; Wong, Cheung, & Hart, 2008).”
Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT) (2018, methodology): “These studies showed that it is relevant to researchers and decision/ policymakers and feasible for them to use,(20) and that there is a variability of agreement of the items ranging from poor to perfect and a need for further testing and refinement of this tool.(20,21) To further the development an”
The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers (2018, neutral): “Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”