Why do I need the server certificate in my Azure Certificates store to use a Client Certificate?

I have a C# app, using HttpClient to connect to a remote server. The server owner has asked for Client Certificate authentication. I sent through the public key certificate to them, and loaded the .pfx certificate (which includes the private key) when I make the request.I receive the following error:

The credentials supplied to the package were not recognized

I now load the server owners public key certificate into my Azure Certificate store, in the "Public key certificates" area, and it works.

Why does loading their certificate make any difference? I already needed to ignore server side validation, as we are connecting via an IP address, as opposed to a URL. This is the configuration:

        var handler = new HttpClientHandler();        handler.ServerCertificateCustomValidationCallback =            (httpRequestMessage, cert, cetChain, policyErrors) =>            {                return true;            };        handler.SslProtocols = SslProtocols.Tls12;        handler.ClientCertificateOptions = ClientCertificateOption.Manual;        var certificate = new X509Certificate2(certificateData, password);        handler.ClientCertificates.Add(certificate);        using (var client = new HttpClient(handler))        {            ...        }

So, how is adding the server certificate doing anything? What have I missed?