How can I detach myself from him?

I've been seeing this guy on and off who really isn't good for me. It's been almost 4 yrs, which is unbelievable. I guess I stay with him because deep down, I feel that I don't deserve any better or that I'm afraid of commitment and he suits that mold (he's "unavailable" so I only see him once in a while). He gets on my nerves and his values clash with mine. I might be holding on to him because I was intimate with him at times (no intercourse or oral s*x, but intimate kissing and touching). Anyway, how can I let go of him for good? When I break up (which I did at least 5 times), I end up missing him and only remembering the good. Help!