Search code examples
linuxvimcharacter-encodingdiffflat-file

Saving a flat-file through Vim add an invisible byte to the file that creates a new line

The title is not really specific, but I have trouble identifying the correct key words as I'm not sure what is going on here. For the same reason, it is possible that my question has a duplicate, as . If that's the case: sorry!

I have a Linux application that receive data via flat files. I don't know exactly how those files are generated, but I can read them without any problem. Those are short files, only a line each.

For test purpose, I tried to modify one of those files and reinjected it again in the application. But when I do that I can see in the log that it added a mysterious page break at the end of the message (resulting in the application not recognising the message)...

For the sake of example, let's say I receive a flat file, named original, that contains the following:

ABCDEF

I make a copy of this file and named it copy.

  • If I compare those two files using the "diff" command, it says they are identical (as I expect them to be)
  • If I open copy via Vi and then quit without changing nor saving anything and then use the "diff" command, it says they are identical (as I also expect them to be)
  • If I open copy via Vi and then save it without changing anything and then use the "diff" command, I have the following (I added the dot for layout purpose):

diff original copy 1c1 < ABCDEF \ No newline at end of file --- .> ABCDEF

And if I compare the size of my two files, I can see that original is 71 bytes when copy is 72.

It seems that the format of the file change when I save the file. I first thought of an encoding problem, so I used the ":set list" command on Vim to see the invisible characters. But for both files, I can see the following:

ABCDEF$

I have found other ways to do my test, But this problem still bugged me and I would really like to understand it. So, my two questions are:

  1. What is happening here?
  2. How can I modify a file like that without creating this mysterious page break?

Thank you for your help!

Solution

  • What happens is that Vim is set by default to assume that the files you edit end with a "newline" character. That's normal behavior in UNIX-land. But the "files" your program is reading look more like "streams" to me because they don't end with a newline character.

    To ensure that those "files" are written without a newline character, set the following options before writing:

    :set binary noeol

    See :help 'eol'.