I'm trying to get anime-list in this site, https://ww1.gogoanime.io
this is the code,
org.jsoup.Connection.Response usage = Jsoup.connect("https://ww1.gogoanime.io/anime-list-A") .header("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8") .header("accept-encoding", "gzip, deflate, sdch, br") .header("accept-language", "en-US,en;q=0.8") .header("cache-control", "max-age=0") .header("user-agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36") .header("upgrade-insecure-requests", "1") .ignoreHttpErrors(true) .followRedirects(true) .method(Connection.Method.GET) .timeout(30000) .execute();System.out.println(usage.parse());
This code works for other websites, however with this site the result is Cloudflare DDOS protectionI have added all the headers, but chrome can access this url without any problem.
Btw, if I didn't set,
ignoreHttpErrors(true)
to true, this will throw an exception 503. No matter what I do it won't go away until I change this to true. So I'm stuck at ddos protection page, which says will redirect to the website in 5 seconds.
I tried the below code too,
org.jsoup.Connection.Response usage = Jsoup.connect("https://ww1.gogoanime.io/anime-list-A") .header("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8") .header("accept-encoding", "gzip, deflate, sdch, br") .header("accept-language", "en-US,en;q=0.8") .header("cache-control", "max-age=0") .header("user-agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36") .header("upgrade-insecure-requests", "1") .ignoreHttpErrors(true) .followRedirects(true) .method(Connection.Method.GET) .timeout(30000) .execute();Thread.sleep(5000);org.jsoup.Connection.Response usg = Jsoup.connect("https://ww1.gogoanime.io/anime-list-A") .header("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8") .header("accept-encoding", "gzip, deflate, sdch, br") .header("accept-language", "en-US,en;q=0.8") .header("cache-control", "max-age=0") .header("user-agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36") .header("upgrade-insecure-requests", "1") .ignoreHttpErrors(true) .followRedirects(true) .cookies(usage.cookies()) .method(Connection.Method.GET) .timeout(30000) .execute();
This didn't work either. My browser access this url without any problem. So I think it's related to jsoup?
btw, I thought it was something about certificates, so I used this too.but it didn't work too.
TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() { public java.security.cert.X509Certificate[] getAcceptedIssuers() { return null; } public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType) { } public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType) { } } }; // Install the all-trusting trust manager try { SSLContext sc = SSLContext.getInstance("SSL"); sc.init(null, trustAllCerts, new java.security.SecureRandom()); HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory()); } catch (Exception e) { throw new RuntimeException(e); }