Ok, here's the skinny.
Fear gets a bad wrap far too often. Factors that your going into an alien invironment, regardles of previous dives, the fact you must take life support systems with you to do this, the inner fear of sharks, drowning, claustrophobia, loss, etc manifests it's self in fear about SCUBA. Fear is not necessarialy a bad thing really, properly managed, it becomes common sense.
Don't let anyone talk you OUT of diving. Rather, feel encouraged by those who would encourage you. While being a shy person about being around others unknown, that is not a bad thing either. It equates to being selective who you feel comfortable with.
As they have suggested, a refresher course may be just the thing, it certainly would not hurt. But if your not wanting the more advanced classes, don't feel obligated or pushed into those either. An advanced class can't hurt, in fact, it will most likely be fun for you. If you don't want to dive, that's most certainly your decision and your none the poorer for it, after all, you have dove before, but if you feel that pull inside that leads you to dive again. Go for it. Treat it a little like you did when you first started diving, from the student point of view, some caution, some wild eyed curiosity with this very cool under-water world. I think it would be total "Bull-puckey" if ANY diver said they did not feel fear or apprehension at first, of even after a long duration between dives, ie, 5 years. But, with more diving, comes the realization that sharks arn't out to get you, with the proper equipment and education and diciplen, your not going to drown, the ocean won't swallow you up and a mermaid won't kidnap you (I know, the last would be a hopefull!), your eventually going to find it to be the time of your life and even discover that those work hours sure cut into your SCUBA schedule!
Ultimately, the choice has always been yours.
Hmm, lets see, to get wet or not to get wet.
In any regard, have fun.