The occurrence of ferroresonance in electrical systems including nonlinear inductors such as transformers will bring a lot of malicious damages. The intense ferromagnetic saturation of the iron core is the most influential factor in ferroresonance that makes nonsinusoidal current and voltage. So the nonlinear behavior modeling of the magnetic core is the most important challenge in the study of ferroresonance. In this paper, the ferroresonance phenomenon is investigated in a single phase transformer using the finite element method and considering the hysteresis loop. Jiles-Atherton (JA) inverse vector model is used for modeling the hysteresis loop, which provides the accurate nonlinear model of the transformer core. The steady-state analysis of ferroresonance is done while considering different capacitors in series with the no-load transformer. The accurate results from copper losses and iron losses are extracted as the most important specifications of transformers. The validity of the simulation results is confirmed by the corresponding experimental measurements.