I, and evidently many others, find clear personal space important for prayer. It is often difficult to get to this in a church, especially during the service; I think they attract those who need contact!
It seems to me that the voice of God is easily drowned out by the voice of people, and the feeling of contact with God is easily pushed aside by the feeling of contact with people. (Fortunately, that feeling is probably less important than we tend to make it. )
It seems to happen only when the person talking to you seems to particularly want contact, as though it is the use for feeding of needs that breaks the peace, more than the act of talking.
See also The glorious 8 o'clock