Goombaking:
Baking soda is good. Just make sure you mix it w/some water and rub gently! Just remember to not exfoliate too much where you shave.
Also, if the soap you're using is drying your face, you need to switch.
Try 'gentle', 'sensitive', 'moisturizing', 'milky' soaps/cleansers.
In Japan, most face lotions/creams are more liquids (watery or milky) than creams (moisturizing, but not too much) and alot of people use 化粧水, which is a slightly moisturized water for the face that has minerals or some factors of pH to calm your skin's balance. This is really great for oily skin, since it's lighter than any type of 'light moisturizers' that I've ever found and is basically like water, but better!!! If you live in a big city, you can probably buy this at some place that sells Asian face stuff.

Shouldn't be too expensive.