House Deed

My grandfather signed the title to his house over to me. He was asking 20,000 for it. My husband and I have paid him $5,000 already. There was no document signed, it was just a verbal agreement. There was a representative of the court there when the title was signed over, saying that he was of sound mind and that it notarized by the court My aunt is having a fit that he signed it over to us because she wanted to sell it and keep the money. There is a family feud over this. He has had several health issues since the house has been signed over and we feel that he is not capable of making this kind of decision. There has also been a lady from Elderly Aging Services that came to my house and told me that they could take me to court and get the house back, even though my name is on it. I am wondering, is this true or is this a bluff?? If you could please answer this question for me, I would greatly appreciate it.