So, I'm currently working on an OS, and I'm stuck in a very dumb situation: switching to VESA/VBE
My current code:
mov ax, 0x4F02
mov bx, 0x4118
int 0x10
It changes Qemu resolution, but now I have a problem with plotting pixels. The formula of the pixel offset is :
uint32 pixel_offset = y * pitch + (x * (bpp/8)) + framebuffer;
The problem is that bpp, pitch and framebuffer are supposed to be given by
mov ax, 0x4F01
int 0x10
and
mov ax, 0x4F00
int 0x10
First,
mov ax, 0x4F00
int 0x10
softlock qemu. And even if it didn't, what do I do to access the info.
What I want is a function that switches to VESA/VBE written ENTIRELY in assembly (nasm) and then a function written in c my kernel can access.
EDIT : I REALLY can't include any C in my bootloader. I need assembly code.
Don't use the "fixed mode numbers" (e.g. mode 0x0118) because the mode may not be supported and if it is supported it could be something completely different than what you're expecting because the "fixed mode numbers" were deprecated about 25 years ago by VBE version 2.
With this in mind, the basic steps are:
a) Check if the computer supports any kind of video using "BIOS Int x010, function 0x1A (Get Display Combination Code)". There's no point bothering to set a video mode if it's a server without any video card. There's also no point using VBE if the video card is an ancient EGA.
b1) Use "VBE function 0x00 (Return VBE Controller Information)" to get a list of valid video mode numbers. If this function doesn't work, fall back to "ancient VGA only" or give up (assume there's no video card, even if there is).
b2) For each mode number that was listed; use "VBE function 0x01 (Return Mode Information)" to figure out what the mode is. Ideally; you'd filter out anything your code doesn't support (e.g. funky old "16 color planar" modes, anything that needs more than 64 KiB of video RAM but doesn't support LFB, anything that uses YUV and not RGB, ...) while creating a list of possibilities. If this function doesn't work, just skip the video mode. Note: You will also want to keep track of the "VBE version number" (from the VBE controller information) and use that to make sense of the information returned by "VBE function 0x01 (Return Mode Information)" (e.g. to determine if the video card is supposed to return some of the information that's only present for later version/s of VBE).
b3) Advanced/optional step. While creating your list of possible video modes; you might want to cross-reference with information from the monitor and try to filter out video modes that the monitor doesn't support. Note that a video mode returned by VBE may be supported by the video card and not supported by the monitor; so unless you check you can't guarantee that anything (except 640*480 video modes) will actually work. WARNING: Parsing EDID is horrible (mostly because they've tried to pack as much as possible in the least bits, with several different formats for different video timings, where some require lookup tables of your own); and trying to match "mode timing" (reported by monitor, including refresh rate, etc) to "video mode" (used by VBE, without any timing/refresh information) is not an exact science.
b4) After you've obtained the full list of video modes (filtered by what your OS supports and maybe by what the monitor supports); use some sort of something to figure out which mode is the "best" mode. This can be a lot more complex than you might think (e.g. figure out the probability that the monitor supports the mode, how much the monitor likes the video mode/how closely it matches monitor's native resolution, how much the OS likes the mode for performance reasons, and how close the video mode is to the user's preference if they have one; then do some sort of calculation to combine all the separate factors into a single final score for the video mode; and choose the video mode with the best final score). Note: If the list is empty, fall back to horrible old VGA modes (e.g. 320*200 with 256 colors) or give up (assume the computer has no video card, even if there is, and continue booting).
c) Set the selected video mode using "VBE function 0x02 (Set Mode)". If you get an error at this point, mark the selected video mode as "borked" in your list of possible video modes, then go back to "step b4".
d) After setting the video mode successfully:
if it was an 8-bit indexed video mode use "VBE function 0x08 (Set DAC Palette Format)" then "VBE function 0x09 (Set Palette Data)" to setup the palette in a known/sane way (I prefer an "2-bit red, 3-bit green, 2-bit blue" palette). Note: You want to force the video card's DAC into the default "6-bit per channel" mode, because the better 8-bit per channel isn't always supported and this avoids the hassle of having 2 different pieces of code to construct data for the DAC.
extract the details (horizontal and vertical resolution, address of frame buffer, which pixel format it is, bytes between rows of pixels, etc) from the information you got earlier from "VBE function 0x01 (Return Mode Information)" so you can pass this information on to whatever code does your drawing.
Note: To make it easier to support many possible video modes (which is necessary to make code work on more different computers); I recommend doing all drawing to a buffer in RAM using a "standard for you" pixel format (e.g. maybe 32-bpp, ARGB) and then converting this data into what the video mode actually wants when copying the data from your buffer in RAM to the framebuffer. That way all your code to draw stuff (lines, rectangles, characters, window decoration, pictures of kittens, whatever) only has to care about horizontal and vertical resolution.
Note: You will need to find the parameters and descriptions of all of the functions I've mentioned and make sure you understand everything they do and all information they return. You can find a list of all BIOS functions online (search for "Ralph Brown's Interrupt List"), and you can find multiple versions of the VBE specs online.