[Objective] With China’s ongoing electricity market reforms and the pursuit of carbon peaking and neutrality goals， renewable energy penetration in the power system is rapidly increasing. While supporting clean energy transition， this also introduces marked electricity price volatility and market uncertainty， highly complicating the development of bidding strategies by power producers， relying in particular on spot trading. In response to the development of the optimal bidding strategies by traditional thermal power enterprises and diverse energy market players in the joint energy and frequency regulation ancillary services market， a bidding strategy optimization method based on proximal policy optimization （PPO） is proposed. [Methods] First， a bi-level optimization model is established for the joint energy-frequency regulation market， integrating multiple generation types and renewable energy storage， where storage smooths price fluctuations through charge-discharge control， elevating the risk response capability of market players such as wind-storage unions. In this framework， the upper-level power producers develop bidding strategies aiming at profit maximization， while the lower-level market clearing model achieves joint dispatch with the objective of minimizing system operating costs. Second， the bidding problem is formulated as a Markov decision process （MDP） within a deep reinforcement learning （DRL） framework， where PPO algorithm is employed to achieve autonomous learning and dynamic optimization of bidding strategies. [Results] Comparative analysis against the theoretical optimal solution in typical cases demonstrates that the proposed approach effectively boosts thermal power enterprises’ revenues， mitigates the risks resulting from renewable energy price fluctuations， reduces system operating costs， and enhances frequency regulation efficiency. [Conclusions] The proposed approach demonstrates superior economic performance and higher real-time computational efficiency in a joint market compared with benchmark solutions.