© 2021 IEEE.As most of the fraud detection systems take part on the side of banking sector, aviation industry is one of the important sectors which faces seriously damaging fraud cases. As fraud cases increase exponentially, airline companies started to face with huge financial losses caused by chargeback expenses. Up to the present, rule-based systems have been used widely to struggle with fraud in aviation. However, their inefficacy, especially against zero-day fraud attacks, encourage the researchers to exploit machine learning approach in aviation. In this study, we created a fraud detection mechanism using a suite of machine learning algorithms on an airline payment dataset. We also proposed several techniques to overcome most common challenges of fraud detection and to point some misapplications in literature. We emphasized the importance of chronological order train-test sets with comparative experiments, then we used undersampling technique to resolve imbalanced dataset problem, afterwards the success of fraud detection system was evaluated from a cost-based perspective and cumulative training set technique was applied to determine the optimal timewise distance between train and test sets. Finally, we carried out a set of feature selection trials to observe its impact on final results.