How do I remove my name from a loan?

I own a house with my ex-husband and when we divorced, the agreement was that he would sell or re-fi to have my name taken off of the loan. He is unable to do this but there are no repercussions listed in the agreement like I thought there were. I have spoken with the lender but they say all he can do is try to re-fi. I have the right to remove him from the deed because of too many missed payments and force him to move out. I cannot afford the home nor do I want it. Do I have any other options?