As multiple signal classification (MUSIC) algorithm is unable to estimate the direction of arrival (DOA) of highly correlated or coherent signal, based on array processing and decomposition of covariance matrix of incident signals due to its lack of source identification. In the proposed hybrid model, we have considered expectation maximization (EM) algorithm for precise identification of the DOA of highly correlated signals in wireless communication applications. The proposed model is analysed, simulated and verified using two different source signals. The mathematical analysis shows substantial closeness with the simulated results. The robustness of the proposed algorithm is further verified by adding noise to the incident signals. The utilization of the EM algorithm in the proposed hybrid approach reduces the time requirement and mathematical complexity of MUSIC algorithm for DOA estimation. By exploiting the EM algorithm, the original transmitted signal and its arriving angle on the antenna element are estimated from the mixture of interferer’s signal, transmitted signal, multipath component and noise. Hence it is straightforward to recognise the desired signal.
