Okay, I have an Asus A8N-E with ALC850, too, and I think I know what the problem is:
The ALC850 actually has 8-channel, not just 6-channel! So every time it is accessed using 6-channel output, there's two "ignored" channels "left" and hence the output channels are rotated by 2!
Try it with some AC3 testfile in VLC. Every time you play it, it shifts 2 channels, there are 3 different layouts until it repeats. So speaker-assignment is not really random...
If you try Fraunhofer's mp3 surround player with the encoded 6channel_id.wav the layout stays the same. But it does not react in any way to what you set in the Audio/MIDI config tool... This may be due to VLC using AuHAL for an output (like it is supposed to be) while the Fraunhofer mp3 surround player probably uses the old static-windows-channelorder method VLC used before that ignores the Audio/MIDI Speaker layout (CoreAudio?)..
And if you keep in mind the 3 steps VLC cycles through and f.ex. switch LS/RS with L/R you'll notice that VLC actually reacts to the speaker layout set in Audio/MIDI config, thanks to AuHAL!
I also noticed that Left/Right front (aka Normal stereo) is much louder than the other channels, but I think this can be fixed using the volume sliders in Audi/MIDI config...
So we need to find the person that's responsible for the AC97 driver, especially the Realtek ALC850 and ask him nicely to provide 8(7.1)-channel output (which OS X can do, just check out the Pulldown in the "Assign channel" window in Audio/MIDI!). If that's too hard to do, another option would be to provide two channels of silence for the missing Left/Right side speakers..
A way to check that the default channel layout is correct (so that programs like the mp3 surround player that don't use AuHAL but a fixed channel order with Core Audio) still work would be to get some generic 5.1 (or 7.1 if that's even readily available!) USB-device, plug it in and check with the mp3-surround player. Don't worry, all the ones I tried worked, there's a generic USB class for such devices.. I recommend the Terratec Aureon 5.1 USB, all these 5.1-headphones make it kinda hard to tell where the sound is really coming from! ;-)
btw: The M-Audio Sonica Theater can do 7.1 on the Mac - but they have their own software for OS X. Still, i think they also work with Audio/MIDI if you set them to "passthrough" IIRC, but from what I remember they placed the two extra channels BEHIND the listener as some sort of "enhanced back center", not directly on the sides (like the Realtek and other PC-manufacturers aim to do)... This could be somewhat compensated for using the Speaker assignment in Audio/MIDI, just move LS/RS to L/R and the two "back centers" (can't remember what they were called) to LS/RS..