Orthogonal time frequency space (OTFS) technique is a recent two-dimensional (2-D) modulation aiming to exploit both time and frequency selectivity of doubly dispersive wireless channel, which significantly affects the reliability of conventional wireless communication systems. In OTFS transmission, delay-Doppler domain is utilized for conveying the data symbols. Besides channel characteristics, the performance of wireless communications systems is severely limited due to the impairments at the RF front-end. One of these impairments is the in-phase and quadrature (IQ) imbalance that occurs due to inevitable imperfections existing between in-phase and quadrature branches at either transmitter (Tx) or receiver (Rx) architecture. The aim of this paper is to reveal and investigate the physical effect of IQ impairment in delay-Doppler domain, which is unknown in the literature. Hence, we provide theoretical models to explicitly understand the impact of IQ imbalance at different stages of OTFS-based communication systems including Tx, Rx, and jointly Tx-Rx. Importantly, we show that although the performance of OTFS transmission depends on both delay and Doppler axes, IQ imbalance leads to an interference originating only from the mirror Doppler axes, which we name mirror Doppler interference (MDI), along with power degradation. Moreover, we assess the performance of IQ imbalanced OTFS under various communication scenarios considering ideal and practical pulse shapes with maximum likelihood (ML) and minimum mean square error (MMSE) detectors, respectively.