The XBeach-NH model is used to add the drag term based on the Morison equation, and the numerical simulation and analysis of the wave motion characteristics and the drag coefficient of the reef surface is carried out. The model is verified by the time series of free surface at different positions on the smooth and rough reef surfaces, including the distribution of wave height and mean water level along the reef. The validated numerical model is used to compare and analyze the spatial and temporal distribution characteristics of free surface near the reef edge and on the reef flat when the smooth reef surface and the rough reef surface exist. Finally, the results of XBeach-NH model simulation are used to examine the optimal drag coefficient values of all rough reef surface experimental conditions. The results show that the XBeach-NH model can satisfactorily simulate the wave motion along the reef, and the drag term based on the Morison equation can reasonably simulate the resistance characteristics of the rough reef surface. The existence of the roughness element causes the wave energy to be subjected to greater bottom frictional damping, and the wave height increase along the reef on the rough reef surface is significantly smaller than that on the smooth reef surface. Higher harmonics and free waves appear in the process of wave propagation, and the second or even higher harmonics appear on both smooth and rough reef surfaces. The existence of the reef roughness unit will significantly reduce the magnitude of the second harmonic. On the rough reef surface, the wave energy is obviously dissipated by the roughness element. The drag coefficient is larger when the water depth and the wave height are both small, which is related to the increase of flow resistance in shallow water. The drag coefficient is larger when the period is large (long wave), and the change of drag coefficient value depends on the water depth of the reef flat when the wave height is large (strong nonlinearity). Studying the wave motion characteristics under the influence of large roughness of coral reef surface can provide scientific basis for disaster prevention and mitigation of coral reef coast under the impact of extreme waves such as typhoon waves.