The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation

By G. Doddington, Alexis Mitchell, Mark A. Przybocki, Lance A. Ramshaw, Stephanie Strassel, Ralph M. Weischedel. 2004

1,438 citations recorded by the source corpus.

The objective of the ACE program is to develop technology to automatically infer from human language data the entities being mentioned, the relations among these entities that are directly expressed, and the events in which these entities participate. Data sources include audio and image data in addition to pure text, and Arabic and Chinese in addition to English. The effort involves defining the research tasks in detail, collecting and annotating data needed for training, development, and evaluation, and supporting the research with evaluation tools and research workshops. This program began with a pilot study in 1999. The next evaluation is scheduled for September 2004.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection (2022, methodology): “Experiment Settings We conduct experiments on two benchmark datasets: ACE05-EN (Doddington et al., 2004) and MAVEN (Wang et al.”
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks (2018, methodology): “For CR, EMD and RE, we use the Automatic Content Extraction (ACE) program ACE05 corpus (Doddington et al. 2004).”
Automated Fact Checking: Task Formulations, Methods and Future Directions (2018, background): “While some of these claims could be represented as triples, they typically require more complex representations; for example, events typically need to be represented with multiple slots (Doddington et al., 2004) to denote their various participants.”
Event Extraction by Answering (Almost) Natural Questions (2020, methodology): “We conduct experiments on the ACE 2005 corpus (Doddington et al., 2004), it contains documents crawled between year 2003 and 2005 from a variety of areas such as newswire (nw), weblogs (wl), broadcast conversations (bc) and broadcast news (bn).”
Distant supervision for relation extraction without labeled data (2009, background): “…often only looked at the hypernym (is-a) or meronym (part-of) relation (Girju et al., 2003; Snow et al., 2005), while those based on the ACE program (Doddington et al., 2004) have been restricted in their evaluation to a small number of relation instances and corpora of less than a million words.”
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, methodology): “The four datasets are used in their work: ACE04 dataset [28], CoNLL04 EC tasks [115], Dutch Real Estate Classifieds (DREC) dataset [12], and Adverse Drug Events (ADE) [47].”
Embedding Methods for Fine Grained Entity Type Classification (2015, background): “Standard entity type classiﬁcation tasks use a small set of coarse labels, typically fewer than 20 (Hirschman and Chinchor, 1997; Sang and Meul-der, 2003; Doddington et al., 2004).”
MAVEN: A Massive General Domain Event Detection Dataset (2020, methodology): “As stated in Section 2, we follow the ED task definition specified in the ACE challenges, especially the ACE 2005 dataset (Doddington et al., 2004) in this paper, which requires ED models to generally detect the event triggers and classify them into specific event types.”
DocRED: A Large-Scale Document-Level Relation Extraction Dataset (2019, methodology): “, 2010), ACE 2003-2004 (Doddington et al., 2004), TACRED (Zhang et al.”
Generating Coherent Event Schemas at Scale (2013, background): “Until recently, all event schemas in use in NLP were hand-engineered, e.g. , the MUC templates and ACE event relations (ARPA, 1991; ARPA, 1998; Doddington et al., 2004).”
Open domain event extraction from twitter (2012, background): “Previous work in event extraction [21, 1, 54, 18, 43, 11, 7] has focused largely on news articles, as historically this genre of text has been the best source of information on current events.”
A Unified MRC Framework for Named Entity Recognition (2019, background): “ACE 2004 and ACE 2005 (Doddington et al., 2005; Christopher Walker and Maeda, 2006) 5: The two datasets each contain 7 entity categories.”
Resource Interoperability for Sustainable Benchmarking: The Case of Events (2018, methodology): “…from their successor OntoNotes 5.0 (Weischedel et al., 2013 The key content extraction tasks of the Automatic Content Extraction (ACE) program (Doddington et al., 2004; Strassel et al., 2008), which ran between 1999 and 2008, were deﬁned as the automatic detection and characterization of…”
The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution (2018, background): “The earliest evaluation corpora were the Message Understanding Conferences (MUC) (Grishman and Sundheim, 1996) and the ACE (Doddington et al., 2004).”