Abstract: In order to improve spectrum efficiency of communication system, a novel time-frequency overlapped hybrid division multiplexing system was researched. A rapid detection algorithm based on Gabor transform and an improved detection algorithm using error back propagation method was proposed. The simulation results show that the proposed rapid detection algorithm obtains significant increase in detection speed at the expense of some performance compared with maximum likelihood sequence estimation algorithm, and that the improved detection algorithm is a good compromise between system performance and detection speed and its performance is still better than the high level modulation with the same spectrum efficiency at a high signal to noise ratio. This shows that improving the transmission data rate with overlapped signals can achieve greater gains than using high-level QAM modulation with the same spectrum efficiency.