We do not currently know of any antonyms for having the time of your life.
For similar words to having the time of your life, try: