I need to download several files (let's say 70 files) from https://nomads.ncep.noaa.gov on a 20.04 ubuntu machine everyday to run a weather forecasting simulations. I used to do it by curl (version: 7.68.0) like this:
$curl "https://nomads.ncep.noaa.gov/cgi-bin/XXX_tail_of_the address" -o destination -C -
It's worked fine with the resuming option for at least a year, until recently, when my organization decided to do some security upgrade on the network that I don't know what it is. Since then, the same command fails to download the files quite randomly. For example, some day, it does not download anything at all, but sometimes it downloads some of files but fails on the others. If it fails, it usually shows these errors:
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to nomads.ncep.noaa.gov:443
for this one the error come after about 20-25 seconds. Or
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to nomads.ncep.noaa.gov:443
which takes longer to come (2-5 minutes).
I tried many things and I'm pretty sure that the problem is on my organization side and nobody here knows what it is. I list the tests I have done:
1- the download task is OK if I use a normal browser on the same machine.
2- the download task is OK if I use another internet connection. However, I normally can't change my internet connection and I have to stick to what I have.
3- I tried curl with either of following options and every single of them showed the same random behavior:
--tlsv1.1--tlsv1.2--tlsv1.3--insecure4- I have the same outcome if I use wget instead of curl.
with kurly, It also fails to download with this error dial tcp: lookup nomads.ncep.noaa.gov on [::1]:53: read udp [::1]:48697->[::1]:53: read: connection refused. but I'm not sure if it shows the same random behavior!
I appreciate if anyone can help me to resolve this annoying problem. Many thanks.