I don't own a great deal of vinyl so won't claim to be a dedicated follower, I have though lived with a couple of true vinyl junkies and know well enough just how serious an addiction it becomes. It's safe to say that what you own, ultimately begins owning you. Just seeing one of these guys attempt to move house was proof enough of that.
Having a portable record player in your home is the perfect ametuers starting point, not only can you find some vintage/retro options but it is a excellent coffee table/home accessory.
Junk markets are a good place to start and then there's Ebay which offers a conveyerbelt of choices. Be warned though, they're usually very heavy so postage prices may take the edge of the shopping experience.
I bought mine from Deptford Junk Market for £9 off a man called Peter.