Action Reaction Interesting Physics Ques?

I'm sort of just curious. If reaction is supposed to be equal to the action, why doesn't your ping pong paddle recoil when it hits the ping pong ball? Thanks!