Adding an on-premises SharePoint organization with the Use SSL parameter enabled fails with the following error: “Connecting to remote server <servername> failed with the following error message: The server certificate on the destination computer (<servername>:<port>) has the following errors: Encountered an internal error in the SSL library.”

User-added image


WinRM service is not configured to use HTTPS.


To make WinRM use HTTPS, please follow Microsoft article How To: Configure WINRM for HTTPS on each machine in your SharePoint farm.
Once done, ensure that a specified port matches the one configured as per the article above, and try adding organization again.