Enable component selection for MPxSubsceneOverride of apiMeshShape sample

In Maya 2016 Extention 2, Autodesk has completed the native Viewport 2 implementation, and it is now the main viewport. There are many new APIs introduced and I've already posted a blog about Make MPxSurfaceShape selectable in Viewport 2 that was a different change in Maya 2016 Extension 2. Today, I am going to talk about another change for the MPxSubsceneOverride.

MPxSubsceneOverride::getInstancedSelectionPath and MPxSubsceneOverride::getSelectionPath are very important for the native Viewport 2 selection, especially for the component selection. When selecting with the Marquee tool in Maya, it will check the **hilite **object and try to intersect components with the renderItems. Once Maya has the component list from MPxComponentConverter, it will merge the component list from MPxComponentConverter to the object's active component selection list.

For MPxSubsceneOverride, because the sample could be instancing a shape with multiple transforms, Maya will try to call getInstancedSelectionPath and getSelectionPath to find a proper dagPath of the component renderItem. When it cannot be done, Maya will keep using the hilite object path (e.g. transform in this case). Thus, MPxSurfaceShape::hasActiveComponents will not return true and there is no activeComponents info for the shape if it returns false by default.

So, if you are trying to select vertex with MPxSubsceneOverride of apiMeshShape, the component info is actually stored on the transform1 instead apiMeshShape1. To correct this, you should implement MPxSubsceneOverride::getInstancedSelectionPath like this:

bool apiMeshSubSceneOverride::getInstancedSelectionPath(const MHWRender::MRenderItem& renderItem, 
                                                        const MHWRender::MIntersection& intersection, 
                                                        MDagPath& dagPath) const
{
    // Return shape path for the component selection
    if(intersection.selectionLevel() == MHWRender::MSelectionContext::kComponent)
    {
        MFnDagNode dagNode(fObject);    
        dagNode.getPath(dagPath);
        return true;
    }

    // Use transform path by default.
    return false;
}

This will store the component's selection in the shape and let MPxSurfaceShape::hasActiveComponents and MPxSurfaceShape::activeComponents of apiMeshShape return the value that the sample is expecting.

The updated apiMeshShape is available here, so please check it out. For more usage reference, please see the gpuSubsceneOverride in our devkit.