Ben Affleck is letting the ongoing chatter about his personal life get to him ... and is particularly frustrated with one rumor-- involving Kick Kennedy-- that just won't die. Sources with direct ...