A Charlie Adam own goal put Rovers in the lead after a weird decision to head back to the keeper not very accurately from close range.
Substitute Matt Phillips looked to have secured a share of the spoils for the hosts when he equalised after 86 minutes, but Brett Emerton sent the traveling Rovers fans wild when he came up with the winner four minutes into injury time.
Allardyce said: "You know there is no time for the opposition to come back then. We have got a very valuable three points and are very pleased to get our first away win.
"You always want to do well against one of your old clubs if you've managed them, but it was a long time ago. It is great to see them up here. It is a bit of a fairytale for them."