There's one right at London Bridge station - you know, if you come out by the bus station and look up to where there are a few shops around the arches? There's that big bike shop, a phone repair shop and so on. On the upper storey there's a record shop - either a Fopp or a Richer Sounds (can't remember which), or at least there was about six months ago when I noticed it. I don't think there's one inside the station itself, though.

I haven't been that way for a while but there used to be a music shop on the north side of the bridge in the parade of shops between the river and the traffic lights at the end of the bridge. Mainstream kind of place.
The closest music shop is probably HMV at Leadenhall Market, a 10-15 minute walk from London Bridge station.