Reporting standards for research in psychology: Why do we need them? What might they be?

By H. Cooper. Reporting quantitative research in psychology: How to meet APA Style Journal Article Reporting Standards (2nd ed.)., 2020

399 citations recorded by the source corpus.

Problem under investigation Participants or subjects; specifying pertinent characteristics; in animal research, include genus and species Study method, including: Sample size Any apparatus used Outcome measures Data-gathering procedures Research design (e.g., experiment, observational study) Findings, including effect sizes and confidence intervals and/or statistical significance levels Conclusions and the implications or applications Introduction The importance of the problem: Theoretical or practical implications Review of relevant scholarship: Relation to previous work If other aspects of this study have been reported on previously, how the current report differs from these earlier reports Specific hypotheses and objectives: Theories or other means used to derive hypotheses Primary and secondary hypotheses, other planned analyses How hypotheses and research design relate to one another Method Participant characteristics Eligibility and exclusion criteria, including any restrictions based on demographic characteristics Major demographic characteristics as well as important topic-specific characteristics (e.g., achievement level in studies of educational interventions), or in the case of animal research, genus and species Sampling procedures Procedures for selecting participants, including: The sampling method if a systematic sampling plan was implemented Percentage of sample approached that participated Self-selection (either by individuals or units, such as schools or clinics) Settings and locations where data were collected Agreements and payments made to participants Institutional review board agreements, ethical standards met, safety monitoring Sample size, power, and precision Intended sample size Actual sample size, if different from intended sample size How sample size was determined: Power analysis, or methods used to determine precision of parameter estimates Explanation of any interim analyses and stopping rules Measures and covariates Definitions of all primary and secondary measures and covariates: Include measures collected but not included in this report Methods used to collect data Methods used to enhance the quality of measurements: Training and reliability of data collectors Use of multiple observations Information on validated or ad hoc instruments created for individual studies, for example, psychometric and biometric properties Research design Whether conditions were manipulated or naturally observed Type of research design; provided in Table 3 are modules for: Randomized experiments (Module A1) Quasi-experiments (Module A2) Other designs would have different reporting needs associated with them 842 December 2008 ● American Psychologist standards for reporting either (a) a study involving random assignment of participants to experimental or intervention conditions (Module A1) or (b) quasi-experiments, in which different groups of participants receive different experimental manipulations or interventions but the groups are formed (and perhaps equated) using a procedure other than random assignment (Module A2). Using this modular approach, the JARS Group was able to incorporate the general recommendations from the current APA Publication Manual and both the CONSORT and TREND standards into a single set of standards. This approach also makes it possible for other research designs (e.g., observational studies, longitudinal designs) to be added to the standards by adding new modules. The standards are categorized into the sections of a research report used by APA journals. To illustrate how the tables would be used, note that the Method section in Table 1 is divided into subsections regarding participant characteristics, sampling procedures, sample size, measures and covariates, and an overall categorization of the research design. Then, if the design being described involved an experimental manipulation or intervention, Table 1 (continued) Paper section and topic Description Results Participant flow Total number of participants Flow of participants through each stage of the study Recruitment Dates defining the periods of recruitment and repeated measurements or follow-up Statistics and data analysis Information concerning problems with statistical assumptions and/or data distributions that could affect the validity of findings Missing data: Frequency or percentages of missing data Empirical evidence and/or theoretical arguments for the causes of data that are missing, for example, missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR) Methods for addressing missing data, if used For each primary and secondary outcome and for each subgroup, a summary of: Cases deleted from each analysis Subgroup or cell sample sizes, cell means, standard deviations, or other estimates of precision, and other descriptive statistics Effect sizes and confidence intervals For inferential statistics (null hypothesis significance testing), information about: The a priori Type I error rate adopted Direction, magnitude, degrees of freedom, and exact p level, even if no significant effect is reported For multivariable analytic systems (e.g., multivariate analyses of variance, regression analyses, structural equation modeling analyses, and hierarchical linear modeling) also include the associated variance–covariance (or correlation) matrix or matrices Estimation problems (e.g., failure to converge, bad solution spaces), anomalous data points Statistical software program, if specialized procedures were used Report any other analyses performed, including adjusted analyses, indicating those that were prespecified and those that were exploratory (though not necessarily in level of detail of primary analyses) Ancillary analyses Discussion of implications of ancillary analyses for statistical error rates Discussion Statement of support or nonsupport for all original hypotheses: Distinguished by primary and secondary hypotheses Post hoc explanations Similarities and differences between results and work of others Interpretation of the results, taking into account: Sources of potential bias and other threats to internal validity Imprecision of measures The overall number of tests or overlap among tests, and Other limitations or weaknesses of the study Generalizability (external validity) of the findings, taking into account: The target population Other contextual issues Discussion of implications for future research, program, or policy 843 December 2008 ● American Psychologist Table 2 Module A: Reporting Standards for Studies With an Experimental Manipulation or Intervention (in Addition to Material Presented in Table 1) Paper section and topic Description Method Experimental manipulations or interventions Details of the interventions or experimental manipulations intended for each study condition, including control groups, and how and when manipulations or interventions were actually administered, specifically including: Content of the interventions or specific experimental manipulations Summary or paraphrasing of instructions, unless they are unusual or compose the experimental manipulation, in which case they may be presented verbatim Method of intervention or manipulation delivery Description of apparatus and materials used and their function in the experiment Specialized equipment by model and supplier Deliverer: who delivered the manipulations or interventions Level of professional training Level of training in specific interventions or manipulations Number of deliverers and, in the case of interventions, the M, SD, and range of number of individuals/units treated by each Setting: where the manipulations or interventions occurred Exposure quantity and duration: how many sessions, episodes, or events were intended to be delivered, how long they were intended to last Time span: how long it took to deliver the intervention or manipulation to each unit Activities to increase compliance or adherence (e.g., incentives) Use of language other than English and the translation method Units of delivery and analysis Unit of delivery: How participants were grouped during delivery Description of the smallest unit that was analyzed (and in the case of experiments, that was randomly assigned to conditions) to assess manipulation or intervention effects (e.g., individuals, work groups, classes) If the unit of analysis differed from the unit of delivery, description of the analytical method used to account for this (e.g., adjusting the standard error estimates by the design effect or using multilevel analysis) Results Participant flow Total number of groups (if intervention was administered at the group level) and the number of participants assigned to each group: Number of participants who did not complete the experiment or crossed over to other conditions, explain why Number of participants used in primary analyses Flow of participants through each stage of the study (see Figure 1) Treatment fidelity Evidence on whether the treatment was delivered as intended Baseline data Baseline demographic and clinical characteristics of each group Statistics and data analysis Whether the analysis was by intent-to-treat, complier average causal effect, other or multiple ways Adverse events and side effects All important adverse events or side effects in each intervention group Discussion Discussion of results taking into account the mechanism by which the manipulation or intervention was intended to work (causal pathways) or alternative mechanisms If an intervention is involved, discussion of the success of and barriers to implementing the intervention, fidelity of implementation Generalizability (external validity) of the findings, taking into account: The characteristics of the intervention How, what outcomes were measured Length of follow-up Incentives Compliance rates The “clinical or practical significance” of outcomes and the basis for these interpretations 844 December 2008 ● American Psychologist Table 2

Subfields: Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/cf0a898715e5041e74a97c00645cd3e971126ef6