Does it have to be a task? I know you can with a dispatch script. you could simply create a script that says:
if VAR = TRUE
CANCEL SOLDIWORKS ENTERPRISE PDM COMMAND
then in you task, set that variable when you need to cancel out.
I am pretty sure you will have to create your own task addin. I think your task could set a flag variable to true if the task completed successfully. Then your workflow could have a holding state and an automatic transition to approved state iwith condition of the flag being true.
I second Greg : Does it have to be a task? Or at least... do a task AND an addin
With addins I know for a fact that you have absolute control over files being added / workflow transitions etc and therefore you have access to the post_add, pre_add etc flags
File moves from State A -> B through transition 1
File is now in "pre-transition" state, addin fires to check for certain things...cancells transition
Alternatively... just add another state (waiting for approval ). If the design checker fails, task moves the files to approved state, else to WIP state