Subtlety about Common Lisp scoping (dynamic vs lexical)

Question

After reading documentation about the declaration SPECIAL, the special operator LET, the macro DEFVAR, and several questions here at StackOverflow about the dynamic versus lexical scoping in Common Lisp, as, for instance, this, I still can't understand the following behaviour after evaluating these forms in SBCL.

;; x is a free variable
CL-USER> (defun fn ()
           (print x))
; in: DEFUN FN
;     (PRINT X)
; 
; caught WARNING:
;   undefined variable: X
; 
; compilation unit finished
;   Undefined variable:
;     X
;   caught 1 WARNING condition
FN

CL-USER> (describe 'x)
COMMON-LISP-USER::X
  [symbol]
; No value

CL-USER> (let ((x 'dinamic_1st_binding))
           (declare (special x))
           (print x)
           (fn)
           (let ((x 'dinamic_2nd_binding))
             (declare (special x))
             (print x)
             (fn))
           (let ((x 'lexical_1st_binding))
             (print x)
             (fn))
           (values))

DINAMIC_1ST_BINDING 
DINAMIC_1ST_BINDING 
DINAMIC_2ND_BINDING 
DINAMIC_2ND_BINDING 
LEXICAL_1ST_BINDING 
DINAMIC_1ST_BINDING 
; No value

;; x is defvar'ed as a top level form
CL-USER> (defvar x 'dinamic_global_binding)
X

CL-USER> (describe 'x)
COMMON-LISP-USER::X
  [symbol]

X names a special variable:
  Value: DINAMIC_GLOBAL_BINDING
; No value

CL-USER> (let ((x 'dinamic_1st_binding))
           (declare (special x))
           (print x)
           (fn)
           (let ((x 'dinamic_2nd_binding))
             (declare (special x))
             (print x)
             (fn))
           (let ((x 'lexical_1st_binding))
             (print x)
             (fn))
           (values))

DINAMIC_1ST_BINDING 
DINAMIC_1ST_BINDING 
DINAMIC_2ND_BINDING 
DINAMIC_2ND_BINDING 
LEXICAL_1ST_BINDING 
LEXICAL_1ST_BINDING 
; No value

Why does the third call to fn, before the variable x is defvar'ed, prints DINAMIC_1ST_BINDING and after the variable x is defvar'ed it prints LEXICAL_1ST_BINDING?

Answer 1

Let's go step-by-step.

Form 1

(defun fn ()
  (print x))

This defines fn.

Since the referenced x variable is not declared lexically, we get a warning.

Free referenced variables are usually locally assumed special with a free declaration¹. This is not defined in the standard, but most implementations do it and signal a warning.

The same principle applies to evaluating the top-level form (setq x ...) in a file or in a REPL.

None of these situations are supposed to globally declare x as special.

Form 2

(describe 'x)

x is just a symbol. Nothing happened to it globally.

Form 3

(let ((x 'dinamic_1st_binding))
  (declare (special x))
  (print x)
  (fn)
  (let ((x 'dinamic_2nd_binding))
    (declare (special x))
    (print x)
    (fn))
  (let ((x 'lexical_1st_binding))
    (print x)
    (fn))
  (values))

The first binding to x is locally declared special. So, fn will pick it up.

The second binging to x is more of the same, just creating a new dynamic binding for x, which is unwound after the call to fn.

The third binding to x is a lexical binding, because every binding is lexical unless there's a global special declaration or a local special bound declaration¹ for the bound variable.

Thus, fn, which picks up the most recent dynamic binding of x, will pick up dinamic_1st_binding.

The xs in the print forms use whatever enclosing meaning of x, thus picking the special x when it's declared special and the lexical x when not.

Form 4

(defvar x 'dinamic_global_binding)

This globally declares x as special and sets its symbol-value to dinamic_global_binding.

Every use of the symbol x as a variable is now tainted with this global special declaration. From this point on, there is no standard way for code to either bind or refer to a variable named x as a lexical variable.

Form 5

(describe 'x)

We now observe the side effects of the previous form. x is globally special, and its current dynamic value is dinamic_global_binding.

Form 6

(let ((x 'dinamic_1st_binding))
  (declare (special x))
  (print x)
  (fn)
  (let ((x 'dinamic_2nd_binding))
    (declare (special x))
    (print x)
    (fn))
  (let ((x 'lexical_1st_binding))
    (print x)
    (fn))
  (values))

All bindings of x are special. ∎

---

1. A declaration is bound if it gives meaning to the establishment of a binding in the form that performs the binding, and is free otherwise, i.e. if it gives meaning only to references/uses of a binding.

So, the following example:

(defun fn2 ()
  (print y))

(let ((y 1))
  (fn2)
  (locally (declare (special y))
    (fn2)))

does not make let bind y dynamically. It only makes the code lexically under locally that refers to y treat it as a dynamic variable through a free declaration. In this case, both calls to fn2 will signal an error.

The following:

(let ((y 1))
  (declare (special y))
  (print y)
  (let ((y 2))
    (print y)
    (locally (declare (special y))
      (print y))
    (print y)))

will print:

1
2
1
2

The first binding of y has a bound special declaration, so it's a dynamic binding.

The second binding of y has no bound special declaration and y is not globally special, so it's a lexical binding.

Thus, the access to y is also conditioned by (missing) declarations about y.

The second declaration is a free declaration, making references to the variable y be treated as dynamic. So, the immediatlely previous binding to a variable named y is not changed by this free declaration.

The following example:

(let ((y 1))
  (print y)
  (locally (declare (special y))
    (print y)))

will fail in the second print form, because its reference to y is dynamic and there is no established dynamic binding for y, i.e. y is unbound and an unbound-variable error is signaled.