Additionally, for future reference:
If VLC (or any other application for that matter) goes into an infinite loop and causes the system to hang like that again, you don't have to reboot. Instead you can use Ctrl+Alt + one of the F keys (F1 to F6) to hide the desktop environment and switch to a full-screen virtual terminal session. e.g. Ctrl+Alt+F3.
This will bring you to a text-based login screen which prompts you for your username and password.
After logging into your account via the command-line login, you can kill vlc with the command:
This will kill all instances of vlc, so if you have more than one instance open it will kill all of them!
If you have multiple instances of VLC open and you only want to kill the crashed one, you could use something like top or pstree to find the PID/Process ID of the crashed/thrashing instance of VLC and then kill it with the kill command:
where {pid} is the process ID of the dead VLC. So if the PID of the crashed VLC is 5054, you would use: kill -9 5054
Note: In the above I have used the -9 switch/option which sends the SIGKILL signal. SIGKILL forces the program to stop regardless of what it is doing and risks some data loss. Ideally you'd first try to kill a process with -15, which is the SIGTERM signal. SIGTERM politely asks the program to finish what it is doing and exit to avoid data loss/corruption.
However, if VLC is in that bad a state, it will not respond to SIGTERM. So it's probably best to go straight for a forced, SIGKILL style shutdown!
For more on the kill command and the signals you can pass to applications see this article:
http://www.linux.org/threads/kill-commands-and-signals.4423/
Once you have killed VLC, type exit to log out and go back to the text login screen, then use Ctrl+Alt+F7 to switch back to your desktop and you should see that VLC has closed.
That might save you a bit of time the next time a program crashes/goes into an infinite loop on you and the system hangs! Rather than do a hard-reboot, simply switch to a virtual terminal and kill whatever process is causing the problem and then switch back to the desktop afterwards!