Can infinite freedom be justified?

I was struck by something that Felix Mara wrote in The Architects' Journal as part of his review of the Arcelor Mittal Orbit. When he first saw the drawings, he said: 'I was determined to like it, as it would obviously be controversial and I happen to believe, as far as the appearance of their work is concerned, artists and architects should have licence to do whatever they want. There is no shortage of people intent on standing in their way.'
Irrespective of what you believe about this particular project, is this a valid position? Work by artists and by architects is not necessarily comparable.If the art is not 'public art' but will exist within a gallery or private space, then we are not obliged to see it, and one can easily defend the artists right to do what they wish, within certain very specific bounds of decency - and perhaps not even there.
But public art and architecture form part of our daily environment. There have been plenty of anomalies in both the law and the application of the planning system, and the new National Planning Policy Framework is only likely to throw up a new set. But surely this is not an argument for no regulation at all - for saying that any building, providing it is functional and energy efficient, can look like anything, wherever it is. There are some very bad buildings, and there are some highly inappropriate ones - possibly good buildings in the wrong place. Sustainability, we are always told, is about much more than just energy saving. It is about building sustainable environments. Should all architects (and artists) really have carte blanche, everywhere?