We do not currently know of any antonyms for have a rapport with.
For similar words to have a rapport with, try: