© 2022, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.Because the importance of the optical wave propagation in fibers, we seek for the optical travelling wave solutions and effect of the third-order dispersion parameter for the extended nonlinear Schrödinger equation (NLSE) which is used to describe the femtosecond(fs) pulse propagation in optical fiber. We have used the modified F-expansion method because it is effective, sufficient and offers more solutions for this kind of problems. First, we obtain the nonlinear ordinary differential form (NODE) by using the wave transformation on the investigated extended nonlinear Schrödinger equation. Then, we propose the modified F-expansion method to get solution of the equation as the NODE form, and by calculating the balancing constant over the NODE form. By substituting the proposed solution function and its derivatives in the NODE form, we obtain an algebraic equation system over this NODE. With the solution of this system, we obtain the appropriate solution sets for the undefined parameters, and then, using the appropriate sets, the proposed solution function, the F-expansion solution functions, and the wave transform, together, we obtain the solution functions of the investigated problem. In order to contribute to the physical interpretation of the problem, 3D, 2D and contour graphs of the obtained solution functions were plotted and interpreted. The problem has not been examined by considering two different situations with the effect of third order dispersion parameter as in this study, before. In this respect, the results obtained in the study are new, and it is believed that they will contribute to the studies in this field.