It sounds like your cat5 is your pots line. Do you know how many drops per outlet are there? If you have only one I would bet your phone is on the cat5 and you can't do that. There are tricks to fixing this you would have to wire half of you cat 5 drop for telco and the other half for network. Telco only uses one pair computer will only use 2 pair. I have done this many times but it's not easy if you don't understand how it works. You will have to use a 110 punch down splice to make it work and make it clean.