问Java:无法从URL获取html
EN

Stack Overflow用户

提问于 2015-03-20 01:04:23

回答 3查看 2.8K关注 0票数 0

我试图从URL中读取html文件。我的代码可以处理大多数站点，但有些站点除外，比如维基。我想我需要设置java代理什么的？

这是我的密码：

    try {
        URL webPage = new URL("http://dota2.gamepedia.com/Dota_2_Wiki");

        URLConnection con = webPage.openConnection();
        con.setConnectTimeout(5000);
        con.setReadTimeout(5000);

        BufferedReader in = new BufferedReader(
                            newInputStreamReader(con.getInputStream()));

        String inputLine;
        while ((inputLine = in.readLine()) != null)
            System.out.println(inputLine);

        in.close();
    }
    catch (MalformedURLException exc){exc.printStackTrace();}
    catch (IOException exc){exc.printStackTrace();}

因此：

java.io.IOException: Server returned HTTP response code: 403 for URL: http://dota2.gamepedia.com/Dota_2_Wiki
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1838)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1439)
at com.Popov.Main.main(Main.java:17)

错误代码403:我如何访问它？顺便说一句，它在浏览器中工作正常。

html

url

java

回答 3

Stack Overflow用户

回答已采纳

发布于 2015-03-20 01:52:34

您的问题很可能是因为没有正确地设置用户代理。你们这些喜欢香草爪哇的家伙。这些是密码

private void sendGet() throws Exception {

    String url = "http://dota2.gamepedia.com/Dota_2_Wiki";

    URL obj = new URL(url);
    CookieHandler.setDefault(new CookieManager(null, CookiePolicy.ACCEPT_ALL));
    HttpURLConnection con = (HttpURLConnection) obj.openConnection();

    con.setRequestMethod("GET");
    con.setRequestProperty("User-Agent", USER_AGENT);

    int responseCode = con.getResponseCode();
    System.out.println("\nSending 'GET' request to URL : " + url);
    System.out.println("Response Code : " + responseCode);

    BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
    String inputLine;
    StringBuffer response = new StringBuffer();

    while ((inputLine = in.readLine()) != null) {
        response.append(inputLine);
    }
    in.close();

    System.out.println(response.toString());

}

请注意，您还需要设置cookie，因为当我尝试不使用cookie时，代码将给我许多重定向循环。

票数 2

Stack Overflow用户

发布于 2015-03-20 01:44:18

您可以简单地尝试使用汤汁 html parser.See示例代码；

public static void main(String[] args) throws IOException {

        Document doc = Jsoup
                .connect("http://dota2.gamepedia.com/Dota_2_Wiki")
                .userAgent(
                        "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36")
                .timeout(0).followRedirects(true).execute().parse();
        Elements titles = doc.select(".entrytitle");

        // print all titles in main page
        for (Element e : titles) {
            System.out.println("text: " + e.text());
            System.out.println("html: " + e.html());
        }

        // print all available links on page
        Elements links = doc.select("a[href]");
        for (Element l : links) {
            System.out.println("link: " + l.attr("abs:href"));
        }

    }