This is a very common reaction for many people. We are bought up to believe that physically hurting another person is wrong, no matter what. That's still true, but it isn't as simple as it sounds. What about when someone wants to indulge in an activity that looks and feels like "hurting," but isn't?
If your partner's feeling conflicted, and understandably so, the key is consent and communication. It is vital that you take the time to explain to him how being spanked makes you feel, what it is about it that turns you on and why you don't feel he's hurting you in the way he might think. He needs to understand that and see how it works for you in order to be able to shift how he views the act from something potentially harmful to something you want and enjoy. Knowing that he is providing you with an exciting and desirable experience will help both of you find pleasure in the act.
The fact that your partner understands that what he is doing is potentially harmful is actually a good thing, as there are always risks involved with this kind of play. It's up to you as a couple to explore where that boundary between pain for pleasure and bad pain is. And the only way to really establish that is to experiment. Good communication is essential here to ensure things don't go too far. I really recommend that you think about having a safeword, not only for your protection but also for his as this way he can be reassured that if something is wrong you have a very clear, obvious way of communicating that.
As I said, having these kind of worries is a completely understandable and common reaction when first experimenting with impact play and it will certainly help your partner if you reassure him of this . A great book that deals with this well is "The Loving Dominant" by John and Libby Warren. I recommend it as a resource for both you and your partner.