"I don't hate your place." Phil doesn't mind Clint's place at all. It's not always the most clean but he can live with that. It is very much Clint's place and there's something endearing about it.

"Why bother with the extra step? Let's just go to your place." As long as he is with Clint Phil is generally happy. After letting him have his space he is very happy to spend time with him again.