java - Jsoup returns 404 error -
jsoup-1.8.1
try {
document document = jsoup.connect(url).get();
return document.getelementsbytag("title").text();
} catch (exception e) {
system.out.println(e);
return null;
}org.jsoup.httpstatusexception: http error fetching url. status=404, url=http://ja.wikipedia.org/wiki/%e3%83%aa%e3%83%b3%e3%82%b4
decoded url here http://ja.wikipedia.org/wiki/りんご
when run in main function in local, runs expect. if execute in servlet, returns 404 error. non encoded url can executed correctly.
wikipedia doesn't allow bots crawling add useragent
, referrer
doc = jsoup.connect(url) .useragent("mozilla/5.0 (windows; u; windowsnt 5.1; en-us; rv1.8.1.6) gecko/20070725 firefox/2.0.0.6") .referrer("http://www.google.com") .get();
Comments
Post a Comment