lisp package not available at read time when loading is deferred

I made a small repro for this issue using only lisp.

file1.lisp :-

(defpackage :my_package
 (:use :cl))
(in-package :my_package)

(defun subscribe (x) (print x)(terpri))

(export '(subscribe))

(in-package :cl-user)

file2.lisp :-

(defun loader ()
 (load "file1.lisp")
 (my_package:subscribe "hello"))

(loader)

Now, running this gives the same error :-

sbcl --load file2.lisp 

This is SBCL 2.1.11.debian, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.

debugger invoked on a SB-C::INPUT-ERROR-IN-LOAD in thread
#<THREAD "main thread" RUNNING {1001834103}>:
  READ error during LOAD:

    Package MY_PACKAGE does not exist.

      Line: 3, Column: 22, File-Position: 59

      Stream: #<SB-INT:FORM-TRACKING-STREAM for "file /home/omkarwagh/swarajya/scratch/file2.lisp" {10009BF613}>

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT   ] Abort loading file "/home/omkarwagh/swarajya/scratch/file2.lisp".
  1: [CONTINUE] Ignore runtime option --load "file2.lisp".
  2:            Skip rest of --eval and --load options.
  3:            Skip to toplevel READ/EVAL/PRINT loop.
  4: [EXIT    ] Exit SBCL (calling #'EXIT, killing the process).

(SB-C:COMPILER-ERROR SB-C::INPUT-ERROR-IN-LOAD :CONDITION #<SB-INT:SIMPLE-READER-PACKAGE-ERROR "Package ~A does not exist." {10018375A3}> :STREAM #<SB-INT:FORM-TRACKING-STREAM for "file /home/omkarwagh/swarajya/scratch/file2.lisp" {10009BF613}>)
4

I guess that the problem is that we don't know the package at read time because the relevant file has not been loaded yet. So the question is, what is the recommended practice in this case?

Original question

I have a roswell script that I've attached below. I have an asdf system called "my_system" which has only one module with one file that contains only one package called "my_package".

Somehow, the asdf system is being loaded but when I try to actually use any of the functions in it, then it fails with an error saying that "my_package" is not found

#!/bin/sh                                                                                                                                
#|-*- mode:lisp -*-|#
#|                                                                                                                                       
exec ros -Q -- $0 $(readlink -f $(dirname $(readlink -f $0)))/asdf.conf "$@"                                                             
|#                                                                                                                                       
(progn ;;init forms
  (ros:ensure-asdf)
  #+quicklisp(ql:quickload '() :silent t)
  )
  
(defpackage :ros.script.test.3880638094                                                                                                
  (:use :cl))
(in-package :ros.script.test.3880638094)
  
(require "asdf")                                                                                                                         

;(asdf:load-system "my_system")
  
;(print #'my_package:subscribe)                                                                                                          

(defun main (mydir &rest argv)                                                                                                           
  (declare (ignorable argv))
  (asdf:initialize-source-registry `(:source-registry :inherit-configuration (:include ,mydir)))                                         
  (asdf:load-system "my_system")                                                                                                             
  (sleep 0.2)                                                                                                                            
  (print "here3")(terpri)
  (print (list-all-packages))                                                                                                            
  (do-external-symbols (s (find-package :my_package)) (print s))
  (print #'my_package::subscribe)(terpri)                                                                                                
  )
;;; vim: set ft=lisp lisp:

The error is :-

Unhandled SB-C::INPUT-ERROR-IN-LOAD in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                                 {1004460113}>:
  READ error during LOAD:

    Package MY_PACKAGE does not exist.

However, if I comment out this line :-

  (print #'my_package::subscribe)(terpri)

I do in fact, see the package in the list of packages as well as the symbol in question :-

#<PACKAGE "MY_PACKAGE">
MY_PACKAGE:SUBSCRIBE

Solution

As you said, the error is signalled because the package does not exist at the read time (when the form is read as text and translated to symbols).

The most straightforward option to change the code above this is to create the symbol when function is executed, e.g., (funcall (intern "SUBSCRIBE" 'my-package)). Variant of this (using read-from-string instead of intern) can be seen in the Swank or Slynk loader.

If you use asdf (which is implied in your question), you should probably use uiop to do this for you - see uiop:symbol-call. For example, documented way to define test-op is with :perform (test-op (o c) (symbol-call :fiveam '#:run! :foobar)))

However, most libraries structure files in such a way that this issue does not arise - the file1.lisp would be loaded before file2.lisp (e.g., using asdf system definition).