Linear and nonlinear normal mode motions may provide promising information about the condition of mechanical structures under small and large amplitude vibrations, respectively. In this view, this study investigates the nonlinear dynamics of cracked beams through use of the nonlinear mode motion and extends the crack identification methods that utilize the linear characteristics to nonlinear vibrating structures. At first, the nonlinear normal modes of the intact and cracked beams are calculated by a continuation algorithm. A finite element model of a geometrically nonlinear prismatic beam was created based on crack stress intensity. Subsequently, a method based on normal mode motion and minimization of strain energy, which is valid for linear and nonlinear vibrating beams, was developed as an optimization problem. To this end, hybrid optimization was also used due to its capability in finding global minimum along with its computational efficiency. It was shown that the proposed crack detection technique is applicable to beams vibrating in linear and/or nonlinear regimes and well capable of detecting both crack location and severity.