Occasionally isn't enough!?

Ive become virtually obsessed with this man, he has been around me for almost ten years, I became aware of feelings for him although they had no basis for the past five years, we have now had several casual encounters, he is unreadable, talks in genral, I want him more now that we have been so close but am scared of scaring him off, what can I do?