We do not currently know of any antonyms for having influence with.
For similar words to having influence with, try: