CRM 2011 Outlook Client Server Address is not Valid

While installing CRM I was using IFD/ADFS and the installation went really smooth but then the IFD access broke so I am unable to connect to my Outlook.
I am uploading the error message, any suggestions?

The server address (URL) is not valid
