Three things that were the causes for the problems: The blower's resistor, the switch and the blower relay. My explanations:
The cause could be just a defected relay, why don't you replace it in the first place if the blower DID NOT shut off with no key in the ignition (Sticking relay).
The next cause that I would check will be the switch to turn the blower on or off. You did not say anything about if you've ever turned the switch to other positions to see how it work: LOW-MEDIUM-HIGH speeds.
Next one is a defected resistor that works only on the Medium and High speeds and not the Low speed. A total failed resistor will put the blower motor in default mode: High speed.
Look at the wiring diagrams I've included below. You'll have to scroll to the right one, I'm looking at number 9 diagram for a J body 1995. Works the same just differ in wires color.
http://www.autozone.com/autozone/repairinfo/repairguide/repairGuideContent.jsp?pageId=0900c1528003cfdd
Look at the diagrams or let someone help you how the circuit run, it'll be more than one failed components like I've explained above or it could be in the wiring. That where the wiring diagrams for: Helping you to see where you're going just like a road map.
Hope it help you, the circuit is simple. Feels good when you repair your own car. Happy wrenching.HTH
Edit: Don't hesitate to send a message if you need more help.