Assessing the impact of minor modifications on the interior structure of GRU: GRU1 and GRU2


Yigit G., AMASYALI M. F.

CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE, vol.34, no.20, 2022 (SCI-Expanded) identifier identifier

  • Publication Type: Article / Article
  • Volume: 34 Issue: 20
  • Publication Date: 2022
  • Doi Number: 10.1002/cpe.6775
  • Journal Name: CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE
  • Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Aerospace Database, Applied Science & Technology Source, Communication Abstracts, Compendex, Computer & Applied Sciences, INSPEC, Metadex, zbMATH, Civil Engineering Abstracts
  • Keywords: curriculum learning, gated recurrent units, recurrent neural networks, Seq2seq, short-term dependency
  • Yıldız Technical University Affiliated: Yes

Abstract

In this study, two GRU variants named GRU1 and GRU2 are proposed by employing simple changes to the internal structure of the standard GRU, which is one of the popular RNN variants. Comparative experiments are conducted on four problems: language modeling, question answering, addition task, and sentiment analysis. Moreover, in the addition task, curriculum learning and anti-curriculum learning strategies, which extend the training data having examples from easy to hard or from hard to easy, are comparatively evaluated. Accordingly, the GRU1 and GRU2 variants outperformed the standard GRU. In addition, the curriculum learning approach, in which the training data is expanded from easy to difficult, improves the performance considerably.