If my OSx86 cannot recongize my driver, can I set 1280x800 instead of 1024x768?

Any possible ways?

Which video card are you current using? The resolution 1280x800 looks like it is on a laptop. First you should try to install the proper driver for the card so you could have proper hardware acceleration for those effect only available if you have QE(Quartz Extreme) support such as the ripple effect when you start a new widget in Dashboard. If your driver is not yet supported, you might try to modify /Library/Preferences/SystemConfiguration/com.apple.Boot.plist and add the following 2 lines before the </dict> tag:

<key>Graphics Mode</key>
<string>1280x800x32</string>

Not sure it this work but you can try to see if it help.