He started to pray, then he wasn’t sure whether he formed the intention to pray or not; does he have to repeat it?

The intention (niyyah)is one of the conditions of prayer being valid. If a person starts to praywithout having formed the intention, then his prayer is not valid.

Ibn Qudaamah (may Allah have mercy on him) said: We do not knowof any difference of opinion among the scholars concerning the necessity offorming the intention to pray; prayer cannot be done without it. The basicprinciple concerning that is the words of Allah, may He be exalted (interpretationof the meaning): “And they were not commanded except to worship Allah,[being] sincere to Him in religion” [al-Bayyinah98:5]. Sincerity is an action of the heart, and this refers to theintention, seeking Allah alone and no other. And the Prophet (blessings andpeace of Allah be upon him) said: “Actions are but by intentions, and each onewill have but that which he intended.” And the seat of the intention is theheart.

End quote from al-Mughni,1/287

But it is very unlikely that this thing (prayingwithout intention) could happen. The intention could accompany the opening takbeer (when one says “Allahu akbar” to start the prayer), or it may be shortly beforeit.

When the Muslim hears the adhaan, then he gets up, does wudoo’and goes to the mosque, and sits down waiting for the iqaamah(call immediately preceding the prayer), then when the iqaamahis given he gets up and stands in the row and so on, then he has definitelyformed the intention to offer the prayer for which the call was given and the iqaamah was recited, so how could he have any doubts abouthis intention after that?

Based on that, what appears to be thecase is that the doubt you are feeling comes under the heading of whispers fromthe Shaytaan (waswaas) whowants to distract you from your prayer and prevent you from pondering andfocusing properly. So you should not pay any attention to it.

For more information on dealing with waswaas, please see the answer to question no. 100268 and 62839.

It should be noted that if there is a greatdeal of doubt, you should not pay any attention to it, because it is whispersfrom the Shaytaan.

Shaykh Ibn ‘Uthaymeen (may Allah havemercy on him) said: If a person experiences so many doubts that he can hardlydo anything without doubting it – if he does wudoo’he doubts it, and if he prays he doubts it, and if he fasts he doubts it – thenthis also does not mean anything, because this is a kind of sickness. What wediscussed above applies to the healthy person who is free of sickness; if aperson is experiencing a great deal of doubt like this, then it is to beassumed that his mind is not stable, so no attention should be paid to thesedoubts.

End quote from ash-Sharh al-Mumti‘, 3/379

And Allah knows best.