The pam_usbng package has been written to make everyday-use of Linux systems more comfortable.
When the token is plugged out, authentication will be rejected. Otherwise, authentication is permitted with or without a passphrase, as you deserve.
Note that, thus securly implemented, pam_usbng has not been developed for use in very sensitive environment.
pam_usbng comes with a couple of utilities which will make work easier: A configuration wizard, a daemon serving a dynamic scripting interface and a tool which parses configuration files and writes authentication data on a device. This makes the whole project easy to use while remaining scriptable.