Ibn Umar said:
I asked Bilal: How did you find the Messenger of Allah (ﷺ) responding to them when they gave him a salutation while he was engaged in prayer. He replied: In this way, and he spread his Palm.
Shaykh Al Bassam said:
The obligation of returning the Salam, due to Allah Ta’ala statement:
و اذا حييتم بتحية فحيوا باحسن منها او رُدُّوهَا 
86. When you are greeted with a greeting, greet in return with what is better than it, or (at least) return it equally. 
The indication for the one praying is the best which he is capable upon in regards to returning the Salaam.
Tawdeehul Ahkaam 2/58 
Note The Narration Shaykh Albani authenticated it
Majid Jawed al Afghanee