.net core under linux throws “The remote certificate is invalid because of errors in the certificate chain: NotTimeValid” when communicating with HTTPS server having letsencrypt cert

i’ve stumbled on an issue – simple .net core code using HttpWebRequest failed to communicate with any server that had lets encrypt cert. no matter if the server used new or old certificate chain – i was getting:

System.Net.WebException: The SSL connection could not be established, see inner exception.
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: NotTimeValid
   at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Security.SslStream.ProcessAuthentication(Boolean isAsync, Boolean isApm, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClientHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at System.Net.HttpWebRequest.SendRequest(Boolean async)
   at System.Net.HttpWebRequest.GetResponse()
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetResponse()
   at System.Net.WebClient.GetWebResponse(WebRequest request)
   at System.Net.WebClient.DownloadBits(WebRequest request, Stream writeStream)
   at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
   at System.Net.WebClient.DownloadString(Uri address)
   at System.Net.WebClient.DownloadString(String address)
   at TestGetWebContent.Program.Main(String[] args) in C:\Users\[..]\source\repos\TestGetWebContent\TestGetWebContent\Program.cs:line 13

at the same time curl or wget could communicate with the same site without issues. i’ve tried updating local ca-certificates package – that did not help. then i’ve run the same binaries on two different servers – both debian bullseye and – surprise surprise! it worked on one but not on the other. in despair i’ve run strace and noticed that on the server where i had a problem program was reading content from ~/.dotnet/corefx/cryptography/x509stores/ca, on the other – where things worked nicely – that folder was not prsent. what did i do? i’ve deleted ~/.dotnet/corefx/cryptography/x509stores/ca and things started to work.

i have too little knowledge of .net core internalls to tell exactly why did it use own store of root certs and did not rely on CAs installed from the ca-certificates package.