在Java中并行线程中写入文件的最佳方法是什么？

我有一个执行大量计算的程序，并经常将它们报告给一个文件。 我知道频繁的写操作可能会使程序运行速度降低很多，所以为了避免它，我希望有一个专门用于写操作的第二个线程。

现在我正在用我写的这个课做（不耐烦可以跳到问题的最后）：

public class ParallelWriter implements Runnable { private File file; private BlockingQueue q; private int indentation; public ParallelWriter( File f ){ file = f; q = new LinkedBlockingQueue(); indentation = 0; } public ParallelWriter append( CharSequence str ){ try { CharSeqItem item = new CharSeqItem(); item.content = str; item.type = ItemType.CHARSEQ; q.put(item); return this; } catch (InterruptedException ex) { throw new RuntimeException( ex ); } } public ParallelWriter newLine(){ try { Item item = new Item(); item.type = ItemType.NEWLINE; q.put(item); return this; } catch (InterruptedException ex) { throw new RuntimeException( ex ); } } public void setIndent(int indentation) { try{ IndentCommand item = new IndentCommand(); item.type = ItemType.INDENT; item.indent = indentation; q.put(item); } catch (InterruptedException ex) { throw new RuntimeException( ex ); } } public void end(){ try { Item item = new Item(); item.type = ItemType.POISON; q.put(item); } catch (InterruptedException ex) { throw new RuntimeException( ex ); } } public void run() { BufferedWriter out = null; Item item = null; try{ out = new BufferedWriter( new FileWriter( file ) ); while( (item = q.take()).type != ItemType.POISON ){ switch( item.type ){ case NEWLINE: out.newLine(); for( int i = 0; i < indentation; i++ ) out.append(" "); break; case INDENT: indentation = ((IndentCommand)item).indent; break; case CHARSEQ: out.append( ((CharSeqItem)item).content ); } } } catch (InterruptedException ex){ throw new RuntimeException( ex ); } catch (IOException ex) { throw new RuntimeException( ex ); } finally { if( out != null ) try { out.close(); } catch (IOException ex) { throw new RuntimeException( ex ); } } } private enum ItemType { CHARSEQ, NEWLINE, INDENT, POISON; } private static class Item { ItemType type; } private static class CharSeqItem extends Item { CharSequence content; } private static class IndentCommand extends Item { int indent; } } 

然后我通过这样做来使用它：

 ParallelWriter w = new ParallelWriter( myFile ); new Thread(w).start(); /// Lots of w.append(" things ").newLine(); w.setIndent(2); w.newLine().append(" more things "); /// and finally w.end(); 

虽然这非常有效，但我想知道： 有没有更好的方法来实现这一目标？

Answer 1

你的基本方法看起来很好。 我会按如下方式构造代码：

 import java.io.BufferedWriter; import java.io.File; import java.io.IOException; import java.io.Writer; import java.util.concurrent.BlockingQueue; import java.util.concurrent.LinkedBlockingQueue; import java.util.concurrent.TimeUnit; public interface FileWriter { FileWriter append(CharSequence seq); FileWriter indent(int indent); void close(); } class AsyncFileWriter implements FileWriter, Runnable { private final File file; private final Writer out; private final BlockingQueue queue = new LinkedBlockingQueue(); private volatile boolean started = false; private volatile boolean stopped = false; public AsyncFileWriter(File file) throws IOException { this.file = file; this.out = new BufferedWriter(new java.io.FileWriter(file)); } public FileWriter append(CharSequence seq) { if (!started) { throw new IllegalStateException("open() call expected before append()"); } try { queue.put(new CharSeqItem(seq)); } catch (InterruptedException ignored) { } return this; } public FileWriter indent(int indent) { if (!started) { throw new IllegalStateException("open() call expected before append()"); } try { queue.put(new IndentItem(indent)); } catch (InterruptedException ignored) { } return this; } public void open() { this.started = true; new Thread(this).start(); } public void run() { while (!stopped) { try { Item item = queue.poll(100, TimeUnit.MICROSECONDS); if (item != null) { try { item.write(out); } catch (IOException logme) { } } } catch (InterruptedException e) { } } try { out.close(); } catch (IOException ignore) { } } public void close() { this.stopped = true; } private static interface Item { void write(Writer out) throws IOException; } private static class CharSeqItem implements Item { private final CharSequence sequence; public CharSeqItem(CharSequence sequence) { this.sequence = sequence; } public void write(Writer out) throws IOException { out.append(sequence); } } private static class IndentItem implements Item { private final int indent; public IndentItem(int indent) { this.indent = indent; } public void write(Writer out) throws IOException { for (int i = 0; i < indent; i++) { out.append(" "); } } } } 

如果您不想在单独的线程中编写（可能在测试中？），您可以使用FileWriter的实现，该调用在调用者线程中的Writer上调用append 。

Answer 2

使用LinkedBlockingQueue是个不错的主意。 不确定我喜欢代码的一些风格……但原则似乎是合理的。

我可能会为LinkedBlockingQueue添加一个等于你总内存的某个百分比的容量..说10,000个项目..这样，如果你的写入速度太慢，你的工作线程将不会继续添加更多的工作，直到堆被烧毁。

Answer 3

与单个消费者线程交换数据的一种好方法是使用Exchanger。

您可以使用StringBuilder或ByteBuffer作为缓冲区与后台线程进行交换。 发生的延迟可能约为1微秒，不涉及创建任何对象，使用BlockingQueue则较低。

从我认为值得重复的例子来看。

 class FillAndEmpty { Exchanger exchanger = new Exchanger(); DataBuffer initialEmptyBuffer = ... a made-up type DataBuffer initialFullBuffer = ... class FillingLoop implements Runnable { public void run() { DataBuffer currentBuffer = initialEmptyBuffer; try { while (currentBuffer != null) { addToBuffer(currentBuffer); if (currentBuffer.isFull()) currentBuffer = exchanger.exchange(currentBuffer); } } catch (InterruptedException ex) { ... handle ... } } } class EmptyingLoop implements Runnable { public void run() { DataBuffer currentBuffer = initialFullBuffer; try { while (currentBuffer != null) { takeFromBuffer(currentBuffer); if (currentBuffer.isEmpty()) currentBuffer = exchanger.exchange(currentBuffer); } } catch (InterruptedException ex) { ... handle ...} } } void start() { new Thread(new FillingLoop()).start(); new Thread(new EmptyingLoop()).start(); } } 

Answer 4

我知道频繁的写操作可以减慢程序的速度

如果你使用缓冲，可能没有你想象的那么多。