When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation
1st International Workshop on Code Translation, Transformation, and Modernization, ReCode 2026, Rio de Janeiro, Brezilya, 12 - 18 Nisan 2026, ss.38-43, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1145/3786180.3788314
- Basıldığı Şehir: Rio de Janeiro
- Basıldığı Ülke: Brezilya
- Sayfa Sayıları: ss.38-43
- Anahtar Kelimeler: Code Translation, In-context Learning, LLMs, Many-shot prompting, Software Engineering
- Yıldız Teknik Üniversitesi Adresli: Evet
Özet
Large Language Models (LLMs) with vast context windows offer new avenues for in-context learning (ICL), where providing many examples ("many-shot"prompting) is often assumed to enhance performance. We investigate this assumption for the complex task of code translation. Through a large-scale empirical study of over 90,000 translations, we systematically evaluate the impact of scaling in-context examples from zero-shot to many-shot configurations of up to 625 examples, with prompts spanning from 100K to 800K tokens. Our findings reveal a "many-shot paradox": while static similarity metrics may modestly improve with more examples, functional correctness consistently peaks with few-shot prompting (5-25 examples). Providing substantially more examples often degrades this crucial functional performance. This study highlights that for code translation, the quality of a few well-chosen examples outweighs sheer quantity, challenging the universal efficacy of "more is better"for ICL and underscoring the task-dependent nature of optimal prompting strategies. Our results have significant implications for effectively leveraging LLMs in software engineering.