My app reads text file line by line and record offset of each line until the end of file. offset changes only when readLine is first executed. It does not change any more after that. I tested with bufferSize from 10 to 16384. What is wrong with my code? I use RandomAccessFile instead of FileInputStream because seek() is faster than skip() when file is big.
String line;
long offset;
RandomAccessFile raf = new RandomAccessFile("data.txt", "r");
FileInputStream is = new FileInputStream(raf.getFD());
InputStreamReader isr = new InputStreamReader(is, encoding);
BufferedReader br = new BufferedReader(isr, bufferSize);
while (true) {
offset = raf.getFilePointer(); // offset remains the same after 1st readLine. why?
if ((line = br.readLine()) == null) // line has correct value.
return;
………………………………
}
In order to update the file pointer in a RandomAccessFile you need to use the read() methods that are part of the RandomAccessFile object.
Making a separate Reader won't update it.
If you need to use BufferedReader you can always wrap a RandomAccessFile in your own InputStream implementation so reads in the inputStream delegate to read in the RandomAccessFile:
I've had to do this before. It's not hard:
public final class RandomAccessFileInputStream extends InputStream{
private final RandomAccessFile randomAccessFile;
private long bytesRead=0;
/**
* The number of bytes to read in the stream;
* or {@code null} if we should read the whole thing.
*/
private final Long length;
private final boolean ownFile;
/**
* Creates a new {@link RandomAccessFileInputStream}
* of the given file starting at the given position.
* Internally, a new {@link RandomAccessFile} is created
* and is seek'ed to the given startOffset
* before reading any bytes. The internal
* {@link RandomAccessFile} instance is managed by this
* class and will be closed when {@link #close()} is called.
* @param file the {@link File} to read.
* @param startOffset the start offset to start reading
* bytes from.
* @throws IOException if the given file does not exist
* @throws IllegalArgumentException if the startOffset is less than 0.
*/
public RandomAccessFileInputStream(File file, long startOffset) throws IOException{
assertStartOffValid(file, startOffset);
this.randomAccessFile = new RandomAccessFile(file, "r");
randomAccessFile.seek(startOffset);
this.length = null;
ownFile =true;
}
/**
* Creates a new {@link RandomAccessFileInputStream}
* of the given file starting at the given position
* but will only read the given length.
* Internally, a new {@link RandomAccessFile} is created
* and is seek'ed to the given startOffset
* before reading any bytes. The internal
* {@link RandomAccessFile} instance is managed by this
* class and will be closed when {@link #close()} is called.
* @param file the {@link File} to read.
* @param startOffset the start offset to start reading
* bytes from.
* @param length the maximum number of bytes to read from the file.
* this inputStream will only as many bytes are in the file.
* @throws IOException if the given file does not exist
* @throws IllegalArgumentException if either startOffset or length are less than 0
* or if startOffset < file.length().
*/
public RandomAccessFileInputStream(File file, long startOffset, long length) throws IOException{
assertStartOffValid(file, startOffset);
if(length < 0){
throw new IllegalArgumentException("length can not be less than 0");
}
this.randomAccessFile = new RandomAccessFile(file, "r");
randomAccessFile.seek(startOffset);
this.length = length;
ownFile =true;
}
private void assertStartOffValid(File file, long startOffset) {
if(startOffset < 0){
throw new IllegalArgumentException("start offset can not be less than 0");
}
if(file.length() < startOffset){
throw new IllegalArgumentException(
String.format("invalid startOffset %d: file is only %d bytes" ,
startOffset,
file.length()));
}
}
/**
* Creates a new RandomAccessFileInputStream that reads
* bytes from the given {@link RandomAccessFile}.
* Any external changes to the file pointer
* via {@link RandomAccessFile#seek(long)} or similar
* methods will also alter the subsequent bytes read
* by this {@link InputStream}.
* Closing the inputStream returned by this constructor
* DOES NOT close the {@link RandomAccessFile} which
* must be closed separately by the caller.
* @param file the {@link RandomAccessFile} instance
* to read as an {@link InputStream}; can not be null.
* @throws NullPointerException if file is null.
*/
public RandomAccessFileInputStream(RandomAccessFile file){
if(file ==null){
throw new NullPointerException("file can not be null");
}
this.randomAccessFile = file;
length = null;
ownFile =false;
}
@Override
public synchronized int read() throws IOException {
if(length !=null && bytesRead >=length){
return -1;
}
int value = randomAccessFile.read();
if(value !=-1){
bytesRead++;
}
return value;
}
@Override
public synchronized int read(byte[] b, int off, int len) throws IOException {
if(length != null && bytesRead >=length){
return -1;
}
final int reducedLength = computeReducedLength(len);
int numberOfBytesRead = randomAccessFile.read(b, off, reducedLength);
bytesRead+=numberOfBytesRead;
return numberOfBytesRead;
}
private int computeReducedLength(int len) {
if(length ==null){
return len;
}
return Math.min(len, (int)(length - bytesRead));
}
/**
* If this instance was creating
* using the {@link #RandomAccessFileInputStream(RandomAccessFile)}
* constructor, then this method does nothing- the RandomAccessFile
* will still be open.
* If constructed using {@link #RandomAccessFileInputStream(File, long)}
* or {@link #RandomAccessFileInputStream(File, long, long)},
* then the internal {@link RandomAccessFile} will be closed.
*/
@Override
public void close() throws IOException {
//if we created this randomaccessfile
//then its our job to close it.
if(ownFile){
randomAccessFile.close();
}
}
}
EDIT
I've tried running your code example by using my RandomAccessFileInputStream and the problem is even with setting a buffer size, the BufferedReader for some reason is still buffering so the file pointer increments by 8912 whenever the underlying inputStream is read. Even if the buffering was working as expected, the buffer will ALWAYS read past the next line so offset will never be the position of the end of the line.
If you don't want to buffer the data AND don't want to write your own implementation that reads lines. You can use DataInputStream which has a deprecated readLine() method. The method is deprecated because it "does not properly convert bytes to characters" however if you are using ASCII characters it should be fine.
InputStream in = new RandomAccessFileInputStream(raf);
DataInputStream dataIn = new DataInputStream(in))
...
if ((line = dataIn.readLine()) == null)
...
Works as expected. the offset only updates the exact number of bytes for each line. However, since it's not buffered it will be slower to read the file.