Abstract: The special-shaped section waveguide has special mode transmission characteristics. In theory, it can expand the existing mode conversion technology and promote the development of multi-frequency point, broadband and compact mode converter. However, due to its complex structure, it is difficult to directly obtain the analytical expression to accurately describe the mode coupling process through mathematical methods, which inhibits the development of this kind of technology. Based on Maxwell's equations, the coupled wave equations of curved waveguide and the expression of coupling coefficient based on vector waveform function are deduced and given. Combined with the numerical solution of vector waveform function, the optimization problem of special-shaped waveguide mode converter can be solved. In order to verify the effectiveness of this technical approach, this paper attempts to design an elliptical waveguide TM₀₁-TE₁₁ mode converter working in X-band. The simulation and numerical results show that the mode conversion efficiency of this mode converter is higher than 95%, the bandwidth is 10%, and the maximum conversion efficiency is higher than 99%. It has better performance than classical circular waveguide mode converters.