remove-text-properties doesn't seem to affect `display` text property

Question

I'm confused why using remove-text-properties to remove the display text property doesn't change the display in the buffer. Instead it seems I must completely remove all the text properties using set-text-properties to nil. For example, why doesn't remove-text-properties work in place of set-text-properties here:

(defvar my-regex "#\\([[:alnum:]]+\\) \\([0-9]+\\)")
(defvar-local my--fontified-p nil)

(defun my-remove-display ()
  "Remove the display, eg. '#blah<2020>' -> '#blah 2020."
  (save-excursion
    (goto-char (point-min))
    (while (re-search-forward my-regex nil 'move)
      ;; why can't I use remove-text-properties here to get rid of 'display?
      (set-text-properties (match-beginning 0) (match-end 0) nil))))

(defun my-toggle-display ()
  "Toggle font-locking and display of '#blah 2020'."
  (interactive)
  (if (setq my--fontified-p (not my--fontified-p))
      (progn
        (font-lock-add-keywords
         nil
         `((,my-regex
            (0 (prog1 nil
                 (put-text-property
                  (1+ (match-beginning 0)) (match-end 0)
                  'display
                  (format "%s<%s>"
                          (match-string-no-properties 1)
                          (match-string-no-properties 2)))))
            (0 'font-lock-constant-face t))))
        (font-lock-flush)
        (font-lock-ensure))
    (my-remove-display)
    (font-lock-refresh-defaults)))

;;; Example that gets fontified
;; #blah 2020

Answer 1

It works for me:

(defun my-remove-display ()
  "Remove the display, eg. '#blah<2020>' -> '#blah 2020."
  (save-excursion
    (goto-char (point-min))
    (while (re-search-forward my-regex nil 'move)
      (remove-text-properties (match-beginning 0) (match-end 0) '(display)))))

You didn't show the remove-text-properties code you tried. Is this what you tried? Did you perhaps pass 'display instead of '(display)?