To be true, VESA doesn't even support 1280x800... Thats where 915 resolution (more here) steps in. With this program on linux you can modify the vidoe bios of your integrated intel gma to have non standard vesa modes (basically it replaces the selected video mode with a new one you specified). I used it to achieve 1440x900 on linux, so I think it will support higher resolutions than 1280x800. (one more thing: 915resolution doesn't support x4500! I had to create a new entry for it in the program, but you basically just need your pci id and have to copy the 96GM codes)
My train of thougth was:If I can use 915resolution to have vesa modes for my lcdm than maybe OSX will use these vesa modes,too. But 915 resolution is a linux utility, how can I start this at boot time? Its' GRUB2 (here) time! Because grub2 has 915resolution module which patches the video bios at boot time! But you need a linux distribution with grub2-915resolutin, thats why I choose archlinux. I installed it on a separate partition (after the osx install), downloaded the package (from here:http://aur.archlinux.org/packages.php?ID=22927), compilled, fiddled with the config file, installed, tried to boot OSX with it (info here:http://www.insanelymac.com/forum/index.php?showtopic=150004&pid=1063106&mode=threaded&start=#entry1063106) and when it worked I modified the 915resolution modul, recompilled it,replaced the original with it and now I can set my 1280x800 from the bootloader and If I boot osx from here then I can see my resolution when I type in ?video at boot, and can use "Graphics Mode" = "1280x800x32"
Hope this helps, but if not I try to help.
Read below, I tried to explain it.