The man behind an NYC club frequented by Taylor Swift and Elon Musk has ticked off an entire village in the Hamptons