
For all I know, I'm the only person in the world who's ever had this problem, but this past weekend I was trying connect my 360 to the VGA port on my new Samsung LCD television.
Connecting over component worked fine, but every time I plugged in the VGA cable, I'd get a black screen and my TV would display a "Mode Not Supported" error message.
Occasionally, a jagged, garbled image would pop up on the screen of the Xbox dashboard, but mostly it was just nothing. I called Samsung, and the tech suggested I plug the component cables back in, reset the resolution to 480p, turn off the console and plug it back in with the VGA cable.
No dice.
While I was on hold waiting for the "level two" tech, I decided to fiddle around with the system some more.
With the VGA cable plugged in (but still no image), I turned on the console. Once it was booted up and I could hear the beeps and swooshes as I toggled through the menu options, I pushed the joystick to the right to get to the console settings tab. Once the swooshes stopped, I knew I was at the right-most tab.
Then I hit the X button, since I knew that the display settings button was the top option on the console settings page, then hit X again to change the display resolution.
I then scrolled all the way to the bottom of the list (again with no image on the screen, just going from memory and sound effects), highlighted what I hoped was the 1920x1080 resolution setting and hit X.
Presto! The screen turns on and everything is perfect!
It sounds like the problem is that the 360 couldn't detect that it was connected to a VGA input and automatically adjust its resolution. The resolution had to be set manually, even though you can't see anything on the screen until you adjust the resolution.
I know Microsoft just released an update for the 360 that's supposed to improve connectivity via the VGA port, but I've been without Internet access the last few weeks (another story), so I don't know if that update solves this issue.
So if anyone else out there runs into this problem, just make sure you memorize the buttons presses to change the display resolution so you can do it literally with your eyes closed.
Doug note: It's about as difficult to switch from Component inputs to HDMI on the PS3.
Victor note: Looks like I wasn't the only one (see comments). Glad to help, Georgios!