Imagine that you register your VBL handler for process A, but this process gets preempted, the CPU time is given to process B. While the process B is running, your registered VBL occurs. What should the system do with your user handler? Forcibly preempt the process B and switch the tasks to process A? Or miss the turn? Or what?
Well, I guess it could work similar like Drag&drop (or it's AV?) -- just use some global shared memory -- VBL could write data into some buffer and user process could read it from there, too. Not a perfect / clean way but it's quite easy.
Hardware interrupt handlers have to be registered in kernel space. So you have to create a device driver, which would do what you want within the VBL interrupt, and would be controlled from your user space application (the demo) by means of such calls as Fopen(), Fread(), Fwrite(), Fcntl() and such.
Ouch... when you write it like that, I have a feeling I'll keep the old fashioned way with supervisor :)
This reminds me Evil / DHS was working on multitasking-friendly demoshell for ct60, I'd like to see how he managed to bypass stuff like this :)