How do I properly hook Interrupt 28h in assembly for DOS, and restore it?

I'm trying to set the handler of Interrupt 28h to my own routine, restore all the registers and flags involved, and restore the original Interrupt handler. I'm using NASM Assembler, under DOSBox and MS-DOS 6.22 in VirtualBox.

I've thought about debugging, but doing so on a TSR program sounds impossible. I've tried pushing the Data Segment onto the Code Segment, and saving the original Data Segment for restoring later, but it seems to hang the machine even after restoring the Data Segment.

section .text   ;Code Section
org 100h        ;DOS Executable Start
mov ah,35h      ;Get Interrupt Vector
mov al,28h      ;Of Interrupt 28h
int 21h         ;Call DOS Kernel
push cs         ;Push Code Segment
pop ds          ;Onto Data Segment
mov [oldseg],es ;Save Old Interrupt Vector Segment
mov [oldoff],bx ;Save Old Interrupt Vector Offset
mov ah,25h      ;Set Interrupt Vector
mov dx,resstart ;To Resstart
int 21h         ;Call DOS Kernel
mov dx,resend   ;Set Data Offset to Resend
sub dx,resstart ;Subtract Resstart
shr dx,4h       ;Shift Right 4 Bits for Paragraph
inc dx          ;One Extra Paragraph for PSP
mov ah,31h      ;Terminate and Stay Resident
xor al,al       ;Return Code
int 21h         ;Call DOS Kernel

resstart:       ;Resident Code Start
push ax         ;Save AX
push es         ;Save ES
push di         ;Save DI
push cx         ;Save CX
push ds         ;Save DS
push dx         ;Save DX
mov ah,00h      ;Set Video Mode
mov al,13h      ;To Mode 13h
int 10h         ;Call BIOS Video
mov ax,0A000h   ;VGA Segment
mov es,ax       ;Stored in ES
xor di,di       ;VGA Offset in DI
mov cx,0FA00h   ;Fill Entire Screen
mov al,09h      ;With Light Blue Color
rep stosb       ;Repeat Store AL at ES:DI
mov ah,25h      ;Set Interrupt Vector
mov al,28h      ;Of Interrupt 28h
mov ds,[oldseg] ;Restore Old Interrupt Vector Segment
mov dx,[oldoff] ;Restore Old Interrupt Vector Offset
int 21h         ;Call DOS Kernel
pop dx          ;Restore DX
pop ds          ;Restore DS
pop cx          ;Restore CX
pop di          ;Restore DI
pop es          ;Restore ES
pop ax          ;Restore AX
iret            ;Return and Restore Flags
resend:         ;Resident Code End

section .data
oldseg dw 0     ;Old Interrupt Vector Segment
oldoff dw 0     ;Old Interrupt Vector Offset

After returning the original interrupt vector address, and setting the new interrupt vector address to "resstart", the program should terminate and stay resident. After this, Interrupt 28h would be triggered automatically since DOS has nothing else to do, which would in turn run my Interrupt handler.

The Interrupt handler sets the video mode to 13h, tries to fill the entire screen with a light blue color, restores the original Interrupt 28h handler, restores all registers and flags involved, and returns to DOS. Executing this program yields no results, the system doesn't even hang. While running the part of setting video mode 13h and filling the entire screen with blue on its own separately, it works perfectly fine.

Solution

mov dx,resend ;Set Data Offset to Resend
sub dx,resstart ;Subtract Resstart
shr dx,4h ;Shift Right 4 Bits for Paragraph
inc dx ;One Extra Paragraph for PSP

In this .COM program you're saving and setting the interrupt vector correctly. However you don't calculate accurately the amount of paragraphs to keep by the DOS.TerminateAnd StayResident function.

The inc dx is needed to round up to the nearest paragraph higher. Certainly not to account for the PSP. That would require 16 paragraphs since the PSP has 256 bytes.

The memory that was allocated to this .COM program starts with the PSP and so the DX count must start there also.

mov     dx, resend 
shr     dx, 4
inc     dx
mov     ax, 3100h   ; DOS.TerminateAndStayResident
int     21h

Tip If you align this resend label to a paragraph boundary, the inc dx is no longer required.

If your present code worked partially in an emulator like virtualbox it's because the memory formerly occupied by your program was not yet overwritten by e.g. the program shell. Emulators, unlike DOS, have the luxury to execute the command interpreter from a far distance.

the screen does fill with blue using virtualbox, though the system hangs

I would hang too if someone switch off the lights while I'm in the middle of writing something! That's what your handler does when it suddenly changes the video mode...

For a TSR program we usually jump over the part that is to be kept resident, so the space occupied by the one-time setup can be recycled by the system.

One more trick you can use, is to write the offset and segment of the old interrupt vector directly in the instructions that will restore the vector. No more problems with segment registers in the handler.

This is my rewrite of your program:

    org     100h
Start:
    jmp     Setup

MyInt28:
    push    ax
    push    es
    push    di
    push    cx
    push    ds
    push    dx
    mov     ax, 0013h   ; BIOS.SetVideoMode
    int     10h
    mov     ax, 0A000h
    mov     es, ax
    xor     di, di
    mov     cx, 64000/2
    mov     ax, 0909h
    cld
    rep stosw
PatchA:
    mov     ax, 0       ; Don't change this to 'xor ax,ax'
    mov     ds, ax
PatchB:
    mov     dx, 0       ; Don't change this to 'xor dx,dx'
    mov     ax, 2528h   ; DOS.SetInterruptVector
    int     21h
    pop     dx
    pop     ds
    pop     cx
    pop     di
    pop     es
    pop     ax 
    iret

Setup:                  ; Resident part ends here.
    mov     ax, 3528h   ; DOS.GetInterruptVector
    int     21h         ; -> ES:BX
    mov     [PatchA + 1], es
    mov     [PatchB + 1], bx
    mov     dx, MyInt28
    mov     ah, 25h     ; DOS.SetInterruptVector
    int     21h
    mov     dx, (256+Setup-Start+15)/16
    mov     ax, 3100h   ; DOS.TerminateAndStayResident
    int     21h