OXBench: A benchmark for evaluation of protein multiple sequence alignment accuracy

By Gajendra P. S. Raghava, Stephen M. J. Searle, Patrick Audley, Jonathan D. Barber, Geoffrey J. Barton. 2003

186 citations recorded by the source corpus.

BackgroundThe alignment of two or more protein sequences provides a powerful guide in the prediction of the protein structure and in identifying key functional residues, however, the utility of any prediction is completely dependent on the accuracy of the alignment. In this paper we describe a suite of reference alignments derived from the comparison of protein three-dimensional structures together with evaluation measures and software that allow automatically generated alignments to be benchmarked. We test the OXBench benchmark suite on alignments generated by the AMPS multiple alignment method, then apply the suite to compare eight different multiple alignment algorithms. The benchmark shows the current state-of-the art for alignment accuracy and provides a baseline against which new alignment algorithms may be judged.ResultsThe simple hierarchical multiple alignment algorithm, AMPS, performed as well as or better than more modern methods such as CLUSTALW once the PAM250 pair-score matrix was replaced by a BLOSUM series matrix. AMPS gave an accuracy in Structurally Conserved Regions (SCRs) of 89.9% over a set of 672 alignments. The T-COFFEE method on a data set of families with <8 sequences gave 91.4% accuracy, significantly better than CLUSTALW (88.9%) and all other methods considered here. The complete suite is available from http://www.compbio.dundee.ac.uk.ConclusionsThe OXBench suite of reference alignments, evaluation software and results database provide a convenient method to assess progress in sequence alignment techniques. Evaluation measures that were dependent on comparison to a reference alignment were found to give good discrimination between methods. The STAMP ScScore which is independent of a reference alignment also gave good discrimination. Application of OXBench in this paper shows that with the exception of T-COFFEE, the majority of the improvement in alignment accuracy seen since 1985 stems from improved pair-score matrices rather than algorithmic refinements. The maximum theoretical alignment accuracy obtained by pooling results over all methods was 94.5% with 52.5% accuracy for alignments in the 0–10 percentage identity range. This suggests that further improvements in accuracy will be possible in the future.

Subfields: Materials Chemistry, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Materials Chemistry

In Molecular Biology

Citations in the corpus

Unified rational protein engineering with sequence-based deep representation learning (2019, positive): “To confirm our cluster assignments, we picked a small set of three cytochromeoxidase (1,2,3) families from OXBench, eight proteins in total.”
DECIPHER: harnessing local sequence context to improve protein multiple sequence alignment (2015, unknown): “This collection includes OXBench [41], PREFAB (v4.”
Multiple sequence alignment. (2006, unknown): “Recently, several new benchmarks have appeared, including OXBENCH [11], PREFAB [12 ], SABmark [13], IRMBASE [14] and a new, extended version of BALIBASE (http://www-bio3d- igbmc.u-strasbg.fr/balibase/).”
A Monte Carlo approach successfully identifies randomness in multiple sequence alignments: a more objective means of data exclusion. (2009, unknown): “Sequence alignments of unrelated sequences deliver meaningless site homology statements—which is trivial—however, alignments of related but highly divergent sequences might face similar problems (Thompson et al. 1999b; Raghava et al. 2003).”
FAMSA: Fast and accurate multiple sequence alignment of huge protein families (2016, methodology): “The experiments on smaller benchmarks (BAliBASE, PREFAB, OXBench-X, SABmark) concerned also top consistency-based algorithms: MSAProbs, 22 QuickProbs, 13,14 and GLProbs.”
3DCoffee: combining protein sequences and structures within multiple sequence alignments. (2004, unknown): “(a) Each box indicates the average accuracy difference between TC-3D and TCoffee when computing HOM39 MSAs with various structure/sequence ratios: [0–20] (6 values), [21–40] (27 values), [41–60] (44 values), [61–80] (44 values), [81–100] (20 values).”
MAFFT-DASH: integrated protein sequence and structural alignment (2019, unknown): “878 test cases were collected from the BAliBase (22), HomFam (23), OXFam (an extended version of OXBench (24)), Mattbench (25) and SISYPHUS (26) benchmark sets.”
MAFFT version 5: improvement in accuracy of multiple sequence alignment (2005, unknown): “TCoffee (20) employs progressive strategy but achieved the highest accuracy (1,21,22).”
Parallelization of MAFFT for large-scale multiple sequence alignments (2018, unknown): “1 a ), HomFam (Sievers et al. , 2011), OXFam (Raghava et al. , 2003; Yamada et al. , 2016) and ContTest (Fox et al. , 2016) (Supplementary Table S1).”