walking away from owner contract and credit

I have never been late on any payments for the two years I have had the owner financed agreement. I am being harrased by the seller. The escrow company has written the seller asking him not to contact me. I have written letters, verbally told him in person and on the phone but to no avail. The seller drives around my house "inspecting" how I am keeping the place. (I have paid to have three trailers full of trash hauled away) He thinks I am renting. He tells me he wants my payments made to him sooner than they are supposted to be.
He says until it is paid for in full he can do what he wants, even through it has been pointed out time and time again that he is wrong.
I can not get a restraining order since there is no bodly harm. Seriously, today I was out watering flowers and this jerk comes walking to the back of my house. He smiles and says since I do not want him to drive in here, then he walked so I can not get mad. !!!! My question, can I leave without my credit being ruined? I have excellent credit and worry he would be able to report me as a default.