How can I make sure ASDF loads a dynamic foreign library every time my system is loaded?

I am looking for the idiomatic way to set up my project whether by an option in the .asd file or other way so that a dynamic foreign library that I depend on is loaded automatically using cffi.

Here is what I'm trying: in a file called armadillo.lisp I have

    ;;; Load the Armadillo Libraries
(push "/users/mcheema/src/c++/lisp-armadillo/" cffi:*FOREIGN-LIBRARY-DIRECTORIES*)
(cffi:define-foreign-library lisp-armadillo-lib
  (t "lisp-armadillo.dylib"))
(cffi:load-foreign-library 'lisp-armadillo-lib)

when I first start up slime I will type

(ql:register-local-projects)
(asdf:load-system :cl-armadillo)

This throws an error:

CL-USER> (asdf:load-system :cl-armadillo)
;Compiling "/Users/mcheema/src/lisp/cl-armadillo/src/armadillo.lisp"...;
Evaluation aborted on #<CFFI::FOREIGN-LIBRARY-UNDEFINED-ERROR #x3020017A461D>.

If I then Just cut and paste the same cffi code fragment above into my slime REPL I get:

CL-USER> (push "/users/mcheema/src/c++/lisp-armadillo/" cffi:*FOREIGN-LIBRARY-DIRECTORIES*)
(cffi:define-foreign-library lisp-armadillo-lib
  (t "lisp-armadillo.dylib"))
(cffi:load-foreign-library 'lisp-armadillo-lib)


#<FOREIGN-LIBRARY LISP-ARMADILLO-LIB "lisp-armadillo.dylib">

EDIT: To answer Dirk's comment. Currently both specials.lisp and io.lisp are empty files save a defpackage and one form for the overall project version number.

(in-package :cl-user)
(defpackage cl-armadillo-asdf
  (:use  :cl :asdf))
(in-package :cl-armadillo-asdf)
(defsystem cl-armadillo
  :author  "Munawar Cheema"
  :license "LLGPL"
  :version "0.0.1"
  :depends-on (:uiop :cffi)
  :components (
               (module "src"
                       :components
                       ((:file "armadillo" :depends-on ("io"))
                        (:file "io" :depends-on ("specials"))
                        (:file "specials"))))
  :description "Create a simple interface to the armadillo library"
  :long-description "Create a simple interface to the armadillo library"
;  :in-order-to ((test-op (load-op cl-armadillo-test))) ; see fukamachi/cl-project.asd
  )

I'll include armadillo.lisp to point of failure as well

(in-package :cl-user)
(defpackage cl-armadillo
  (:use
   #:cl
   #:uiop
   #:cffi
   #:cl-armadillo.specials
   #:cl-armadillo.io)
  (:nicknames :arm)
  (:export matrix-destroy
           matref
           make-matrix-ones
           make-matrix-randu
           make-matrix-zeros
           make-matrix-eye))

(in-package :cl-armadillo)

;;; Load the Armadillo Libraries
(push "/users/mcheema/src/c++/lisp-armadillo/" cffi:*FOREIGN-LIBRARY-DIRECTORIES*)
(cffi:define-foreign-library lisp-armadillo-lib
  (t "lisp-armadillo.dylib"))
(cffi:load-foreign-library 'lisp-armadillo-lib)

Solution

After some trial and error I figured out how to make sure my code works with asdf:load-system everytime. The key was to wrap the definition of the library and update of cffi:*foreign-library-directories in an eval-when form:

(eval-when
    (:compile-toplevel :load-toplevel :execute)
  (pushnew #P"/users/mcheema/src/c++/lisp-armadillo/" cffi:*FOREIGN-LIBRARY-DIRECTORIES*)
  (cffi:define-foreign-library lisp-armadillo-lib
    (t "lisp-armadillo.dylib"))
  (cffi:load-foreign-library 'lisp-armadillo-lib))

Code Edited to reflect pushnew suggestion in comments