Pointwise Prediction for Robust, Adaptable Japanese Morphological Analysis

By Graham Neubig, Yosuke Nakata, Shinsuke Mori. 2011

257 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Structured Attention Networks (2017, methodology): “Japanese word segmentation is done using the KyTea toolkit (Neubig et al., 2011).”
OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles (2016, methodology): “For Japanese and Chinese, the KyTea 4 word segmentation library is used for the tokenisation (Neubig et al., 2011) along with pre-trained models.”
Neural Machine Translation via Binary Code Prediction (2017, unknown): “To prepare inputs for training, we used tokenizer.perl in Moses (Koehn et al., 2007) and KyTea (Neubig et al., 2011) for English/Japanese tokenizations respectively, applied lowercase.perl from Moses, and replaced out-of-vocabulary words such that rank( w ) > V − 3 into the UNK marker.”
Learning to Parse and Translate Improves Neural Machine Translation (2017, methodology): “We tokenize each Japanese sentence with KyTea (Neubig et al., 2011) and preprocess according to the recommendations from WAT’16 (WAT, 2016).”
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models (2018, methodology): “For all language pairs, tokenization was carried out using the Moses tokenizer, 2 except for Japanese, where word segmentation was performed using the KyTea tokenizer (Neubig et al., 2011).”
Towards Burmese (Myanmar) Morphological Analysis (2019, methodology): “Typical SVM-based works are as follows: Japanese morphology analysis by Kudo and Matsumoto [22] 8 and Neubig et al. [31].”
Models of Visually Grounded Speech Signal Pay Attention to Nouns: A Bilingual Experiment on English and Japanese (2019, unknown): “gner [18] and Maus Forced Aligner [19] for English and Japanese respectively), resulting in alignments at word and phone level. We also tagged each dataset using TreeTagger [20] for English and KyTea [21] for Japanese. As the tagset of both taggers differs, we mapped each POS to its Universal POS eq”
Incorporating Discrete Translation Lexicons into Neural Machine Translation (2016, methodology): “1 k for KFTT. NMT Systems: We build the described models us-ing the Chainer 2 toolkit.”
Overview of the 4th Workshop on Asian Translation (2017, unknown): “6 (Neubig et al., 2011) with full SVM model13 and MeCab 0.”
The Kestrel TTS text normalization system (2014, methodology): “Deciding whether to use the dative or some other form of the date is outside the scope of these grammars: that is handled by a separate morphosyntactic tagger, a pointwise predictor that uses the grad-boost algorithm to predict each attribute (Duchi and Singer 2009; Neubig, Nakata and Mori 2011).”
On the Relation between Position Information and Sentence Length in Neural Machine Translation (2019, unknown): “0.4.2) (Neubig et al., 2011) for Japanese segmentations.”
Overview of the 1st Workshop on Asian Translation (2014, methodology): “…gu re 1: T he su bm is si on w eb pa ge fo rp ar tic ip an ts For Japanese segmentation we use three different tools, which are Juman version 7.0 (Kurohashi et al., 1994), KyTea 0.4.6 (Neubig et al., 2011) with Full SVM model 7 and MeCab 0.996 (Kudo, 2005) with IPA dictionary 2.7.0 8.”
Overview of the 5th Workshop on Asian Translation (2019, methodology): “6 (Neubig et al., 2011) with full SVM model13 and MeCab 0.”
Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021, methodology): “The Ja-En dataset is obtained from the Kyoto Free Translation Task (KFTT) word alignment data (Neubig, 2011), and the Japanese sentences are tokenized with the KyTea tokenizer (Neubig et al., 2011).”
An Empirical Study of Mini-Batch Creation Strategies for Neural Machine Translation (2017, methodology): “The English and German texts were tokenized with tokenizer.perl4, and the Japanese texts were tokenized with KyTea (Neubig et al., 2011).”
An Empirical Study of Adequate Vision Span for Attention-Based Neural Machine Translation (2016, methodology): “perl” for English side, and Kytea tokenizer (Neubig et al., 2011) for Japanese side.”
OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora (2018, methodology): “Tokenisation : Once the sentences are segmented, they are tokenised, using either the tokenisation scripts from Moses (Koehn et al., 2007) or the Kytea toolkit for Chinese word segmentation (Neubig et al., 2011).”
Tree-to-Sequence Attentional Neural Machine Translation (2016, methodology): “For the target language, i.e., Japanese, we used KyTea (Neubig et al., 2011), a Japanese segmentation tool, and performed the pre-processing steps recommended in WAT’15.”