Jsoup将内容保存到数据库中

我有一个url数组，我想存储我在数据库中读取的url中的信息。 我的问题是我的数据列表太大url如果读取序列化每个url从上面露水存储在数据库中需要时间。

我知道有一种方法可以使用线程进行操作，但我不知道该怎么做，请帮助我。 或者不管你的方法

try { String lstUrls = "http://www.java2s.com/Tutorials/Java/Scala/index.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0020__Scala_Variables.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0040__Scala_Variable_Declarations.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0060__Scala_Semicolons.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0080__Scala_Code_Blocks.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0090__Scala_Comments.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0100__Scala_Type_Hierarchy.htm\n"; String[] urls = lstUrls.split("\n"); for (String url : urls) { Document doc = Jsoup.connect(url).userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36").get(); Elements select = doc.select("div.row"); String html = select.html(); System.out.println(html); /* insert html to database */ } } catch (IOException ex) { ex.printStackTrace(); } 

Answer 1

要使用多个线程来检索数据，您可以执行以下操作：

  Executor ex = Executors.newFixedThreadPool(3); String lstUrls = "http://www.java2s.com/Tutorials/Java/Scala/index.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0020__Scala_Variables.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0040__Scala_Variable_Declarations.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0060__Scala_Semicolons.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0080__Scala_Code_Blocks.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0090__Scala_Comments.htm\n" + "http://www.java2s.com/Tutorials/Java/Scala/0100__Scala_Type_Hierarchy.htm\n"; String[] urls = lstUrls.split("\n"); for (final String url : urls) { try { ex.execute(new Runnable() { @Override public void run() { try { Document doc = Jsoup .connect(url) .userAgent( "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36") .get(); Elements select = doc.select("div.row"); String html = select.html(); System.out.println(html); /* * insert html to database */ } catch (Exception e) { e.printStackTrace(); } } }); } catch (Exception e) { e.printStackTrace(); } } 

这将使用3个线程同时处理url，如果你想使用超过3个线程更改此行Executor ex = Executors.newFixedThreadPool(3); 并用你想要的任何数字替换3 。

您可以在此处找到有关Executors的更多信息

Answer 2

我建议在插入数据库之前压缩数据。

 //PreparedStatement.setBytes(1,compress(html)); public static byte[] compress(String str) throws Exception { if (str == null || str.length() == 0) { return null; } ByteArrayOutputStream obj = new ByteArrayOutputStream(); GZIPOutputStream gzip = new GZIPOutputStream(obj); gzip.write(str.getBytes("UTF-8")); gzip.close(); return obj.toByteArray(); } public static String decompress(byte[] bytes) throws Exception { GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(bytes)); BufferedReader bf = new BufferedReader(new InputStreamReader(gis,"UTF-8")); String outStr = ""; String line; while ((line = bf.readLine()) != null) { outStr += line; } return outStr; } 

第二种方法，将html数据保存到文件并仅存储数据库中的文件路径。

 long ts = System.currentTimeMillis(); String filePath = String.valueOf(ts)+".gz"; saveToFile(filePath ,html); -------- public static void saveToFile(String filePath, String text) { try { GZIPOutputStream gzos = new GZIPOutputStream(new FileOutputStream(filePath)); gzos.write(text.getBytes("UTF-8")); gzos.finish(); gzos.close(); } catch (IOException ex) { ex.printStackTrace(); } }