Well Neon, it does look like you're working on a fair bit of conjecture here. He may not actually be gay, and may just be, shall we say, genteel. Even if he was gay, what's to say that he's interested in a relationship with you?
The simplest thing to do is to lay your cards on the table: "Look, you're a good bloke and all, but I have to be honest and say that I'm no boxer, and I don't fancy being punched in the ring", or something to that effect.

If he's cool with that, well and good. If he's not, then you've got the grounds to say that his intentions are making you really uncomfortable, and because of that you would no longer wish to maintain contact with him.