Diabetes Mellitus is one of the major non-communicable diseases that occurs when the pancreas does not produce enough insulin or the body does not respond effectively to the insulin, which causes blood sugar levels to increase. Uncontrollable diabetes may lead to many serious complications, including damage to the eye, kidney, nerves, heart and peripheral vascular system. Early diagnosis is vital to enable patients to be treated early, thus avoiding and reducing the risk of complication. Fuzzy Inference System (FIS) is widely used to predict disease at an early stage where it imitates human thinking by incorporating the IF-THEN rules to solve a problem systematically. However, this method can cause computational complexity when involving many attributes. In this study, Hierarchical Fuzzy Inference System (HFIS) is proposed to overcome this limitation in diabetes prediction. There are eight attributes under consideration and decomposed into three subsystems based on their similar characteristics which represent the first level of the HFIS. The outputs of the subsystems are then used as input variables for the main system at the second level to generate the output indicating the diabetes severity. The proposed HFIS significantly reduced the number of generated rules without compromising the accuracy of the prediction and can be developed into a more comprehensive system for predicting diabetes.