Search code examples
cblasalgebracublas

How do do BLAS/cuBLAS treat the factors alpha and beta in their routines?

Many linear algebra routines have constants such as alpha and beta as arguments. For example cublas?GEMM performs the following operation:

C := alpha*op( A )op( B ) + betaC

Suppose I set beta to 0.

  1. Will the cuBLAS still perform an unnecessary scalar-matrix multiplication and matrix-matrix addition? What about other libraries such as BLAS/LAPACK/MKL?

  2. If the necessary operations is not performed: Do I need to do something to ensure this, or is it avoided automatically?

  3. Are there other values for alpha/beta for which there are other optimizations? For example, suppose I instead set beta=1, will scaling by beta operation be skipped?

  4. Why does the cuBLAS documentation and BLAS documentation specify these factors in DGEMM as const double but in examples a double value is passed to them? What's the difference?

I would be surprised if these libraries did waste operations in the manner I described, but I didn't find an explicit discussion about it anywhere other than the cuBLAS documentation mentioning:

if beta == 0 then C does not have to be a valid input.

Solution

  • Even the reference implementation optimises here. No serious implementation with do the operation regardless of values for alpha or beta.

    1. No it will not.
    2. N/A
    3. Just leave beta=0. to ignore C. beta=1. to skip scaling
    4. The reason is compatibility with FORTRAN. There were no const variables in FORTRAN prior to F90. The BLAS interfaces were defined prior to F90 and everybody sticks to the conventions. I you want to have a C interface with proper keywords, look at the c-specific interfaces like sblas_dgemm.

    Here is the reference implementation for DGEMM. http://www.netlib.org/lapack/explore-html/d7/d2b/dgemm_8f_source.html. Look for Quick return if possible., And if alpha.eq.zero. etc