"It means that I don't know where Banner is, Thor's busy, I'm telling Nat, and that leaves Tony and I'll let him beat the shit out of me but I won't hand him something that easy to use against me - or carelessly fuck up with you. You don't really know any of the other people around well enough for it to be any of their business." He's not explaining to convince, he's explaining because Clint... well expressed interest in what he meant by trust.

And because they're on the floor, probably.