This function doesn't work for me...SW 2017.3.0.
If I select closed line (contour) in the perimeter of the PCB, this function will shrink the PCB board to the selected contour...
And the name of the function is rather confusing...
What will bee defined from selected object ?
Found a workaround:
Create contour > Convert > Create region from Selected Primitives > Region Properties > Kind > Board cutout
You are 100% correct, I misread your question and defined the procedure to modify the board shape not a cutout.
Unfortunately I do not know of a method for doing this from existing (selected) objects. You currently have to define the shape manually. Before they get this implemented you can send the board over (PUSH) to SOLIDWORKS and create the cutout there. I actually like this as SOLIDWORKS has some great sketching features not available in any ECAD tool.
I have created an Enhancement request for this: 1-14929600051