I have been looking for software that can allow me to mask out any part of the video that is required.

I have looked at VirtualDub-1.9.11, but can not see how I mask out, I do see the options in the menu but clicking it does nothing that i can see. Also F1 was not much help, does anyone know what I am doing wrong or what I should do once the option in the menu is selected to start the mask...?