Efficiency in writing to disk in Fortran

Question

I am trying to write a bunch of large matrices in fortran to disk. Matrices are V, C and d. All have size (2 , n1, n2, n3, n4, n5).

These are large matrices. And fortran is taking around 3 hours to write them.

do ind1=1,n1
do ind2=1,n2
do ind3=1,n3
do ind4=1,n4

     write(filename,'(a,i0,a,i0,a,i0,a,i0,a,i0,a)')'PF',t,'_',ind1,'_',ind2,'_',ind3,'_',ind4,'.txt'

     OPEN(UNIT=25,FILE=filename,STATUS='replace',ACTION='write')
     do ind5=1,n5
      WRITE(25,*) c(2,ind1,ind2,ind3, ind4,ind5)
     end do
     do ind5=1,n5
      WRITE(25,*) v(2,ind1,ind2,ind3, ind4,ind5)
     end do
     do ind5=1,n5
      WRITE(25,*) d(1,ind1,ind2,ind3, ind4,ind5)
     end do
     CLOSE(UNIT=25)

end do
end do
end do
end do

Any smart way of doing this faster?

Answer 1

Let me start by saying that I agree with the comments, the best way would be to move to something like HDF5 or NetCDF which should give you both the portability of the data and also high performance.

That said if you want to stick with simple Fortran I/O it's usually a good idea to keep the number of I/O transactions to a minimum. I suspect all your opens are not a good idea, I would stick it all in one file. If that's OK you then have 3 routes as shown below and run on my laptop (which admittedly has a SSD). Anyway it should give you some ideas.

ian-admin@agon ~/test $ cat io.f90
Program test_io

  Implicit None

  Integer, Parameter :: wp = Selected_real_kind( 12, 70 )

  Integer, Parameter :: n1 = 201
  Integer, Parameter :: n2 = 50
  Integer, Parameter :: n3 = 2
  Integer, Parameter :: n4 = 2
  Integer, Parameter :: n5 = 21

  Real( wp ), Dimension( 1:2, 1:n1, 1:n2, 1:n3, 1:n4, 1:n5 ) :: stuff

  Integer :: start, finish, rate

  Integer :: i1, i2, i3, i4, i5

  Call Random_Number( stuff )

  Call System_Clock( start, rate )
  Do i1 = 1, n1
     Do i2 = 1, n2
        Do i3 = 1, n3
           Do i4 = 1, n4
              Do i5 = 1, n5
                 Write( 10, * ) stuff( 1, i1, i2, i3, i4, i5 )
              End Do
           End Do
        End Do
     End Do
  End Do
  Call System_Clock( finish, rate )
  Write( *, * ) 'Loops : ', Real( finish - start ) / Real( rate )

  Call System_Clock( start, rate )
  Write( 11, * ) stuff( 1, :, :, :, :, : )
  Call System_Clock( finish, rate )
  Write( *, * ) 'Array : ', Real( finish - start ) / Real( rate )

  Call System_Clock( start, rate )
  Write( 12 ) stuff( 1, :, :, :, :, : )
  Call System_Clock( finish, rate )
  Write( *, * ) 'Unform: ', Real( finish - start ) / Real( rate )

End Program test_io
ian-admin@agon ~/test $ gfortran -O -Wall -Wextra -std=f2003 io.f90
ian-admin@agon ~/test $ ./a.out
 Loops :    2.28500009    
 Array :    1.80200005    
 Unform:    5.79999983E-02
ian-admin@agon ~/test $