The late night host bought the bluff-top property in the ’80s and held onto it for the rest of his life