Optimal RAG System Design for Turkish Textbooks: A Comprehensive Evaluation and Performance Enhancement Study

Oner E. N., Ceyhun S., Yildiz M. H., Goncharova A., Yucel T. S., KESGİN H. T., ...More

2025 Innovations in Intelligent Systems and Applications Conference, ASYU 2025, Bursa, Turkey, 10 - 12 September 2025, (Full Text) identifier

  • Publication Type: Conference Paper / Full Text
  • Doi Number: 10.1109/asyu67174.2025.11208464
  • City: Bursa
  • Country: Turkey
  • Keywords: educational AI systems, RAG systems, retrieval-augmented generation, textbook question-answering, Turkish natural language processing
  • Yıldız Technical University Affiliated: Yes

Abstract

Retrieval-augmented generation enables precise educational question-answering by combining retrieval with natural language generation. Limited evaluation exists for educational RAG systems, particularly for curriculum-based applications in diverse languages. We introduce systematic evaluation using Turkish Ministry of Education textbooks, comparing embedding architectures, generation models, and optimization techniques across educational questions. Results establish multilingual-e5-large-instruct and BAAI/bge-m3 as optimal embedding models, demonstrate Qwen3 and Turkish-Gemma-9b's superior generation performance, and show ensemble methods yield 3.3% improvements. This provides evidence-based recommendations for educational RAG development across different languages and curricula.