We do not currently know of any antonyms for having a love affair with.
For similar words to having a love affair with, try: