In the last post, I talked about the NeuronDataReader SDK and the general process to use these APIs, now we will talk about how to make a Maya plug-in client to receive and parse the animation data.
With this project, we are going to create a Maya plug-in which will:
- fetch the animation data from Axis Neuron application to Maya,
- create a Neuron specific skeleton, and assign the animation data to the skeleton,
- retarget the animation from the skeleton to another characterized model.
The workflow would be something like:
- Register the frame data callback to fetch BVH binary stream data of the skeleton. Note that the callback is invoked by a separated thread of NeuronDataReader lib. We will cache the data within the callback.
- Create MPxThreadedDeviceNode-derived node to fetch the data from cache, use Maya system lock MSpinLock to avoid data access conflict. Set the animation data to output array of translation and rotation.
- Create Neuron specific skeleton, and connect each items in translation and rotation array to the specific neuron joint.
- Retarget the animation from the skeleton to another characterized model.
Let’s check the details on each step.
Step 1 - We already mentioned in my previous post how to receive data from the Axis Neuron application using the NeuronDataReader SDK. Here, we need to define the callback first, but remember that the callback should be invoked within a separated working thread, you cannot handle any UI related work within the callback directly. Here we will cache the data of current frame, and our device thread will try to fetch the frame data later. Of course, we need to lock the resource to mutually exclusive access to the shared frame data by 2 different threads, I will talk more about this in step 2.
Here is the some code samples:
--- Define the skeleton data structure for every frame:
struct FrameData{
int nFrame;
float data[60][6];
};
--- Declare the current frame data as a static member of NeuronForMayaDevice:
static FrameData curFrameData;
--- Declare the callback to receive the frame data:
static void myFrameDataReceived(void* customedObj, SOCKET_REF sender, BvhDataHeaderEx* header, float* data);
--- Implementation of myFrameDataReceived callback
void NeuronForMayaDevice::myFrameDataReceived(void* customedObj, SOCKET_REF sender, BvhDataHeaderEx* header, float* data)
{
if( !bLive )
return;
BOOL withDisp = header->WithDisp;
BOOL withReference = header->WithReference;
UINT32 count = header->DataCount;
static int nFrame = 0;
// add mutex
spinLock.lock();
// push the data for each frame into a queue
curFrameData.nFrame = nFrame++;
for(UINT32 i = 0; i < 60; ++i )
for( UINT32 j = 0; j< 6; j++ )
curFrameData.data[i][j] = data[i*6+j];
spinLock.unlock();
}
Besides registering the callbacks, we also added a customized command “NeuronForMayaCmd” to connect or disconnect to the Axis Neuron application with TCP/IP protocol,
class NeuronForMayaCmd : public MPxCommand
{
public:
NeuronForMayaCmd() { mDeviceName=""; mStart=false;};
virtual ~NeuronForMayaCmd();
MStatus doIt( const MArgList& args );
static void* creator();
static MSyntax newSyntax();
private:
MStatus parseArgs( const MArgList& args );
static SOCKET_REF socketInfo;
MString mDeviceName;
bool mStart;
};
Here is the implementation of the customized command “NeuronForMayaCmd”:
#define kStartFlag "-s"
#define kStartFlagLong "-start"
#define kDeviceNameFlag "-dn"
#define kDeviceNameFlagLong "-device name"
SOCKET_REF NeuronForMayaCmd::socketInfo = NULL;
NeuronForMayaCmd::~NeuronForMayaCmd() {}
void* NeuronForMayaCmd::creator()
{
return new NeuronForMayaCmd();
}
MSyntax
NeuronForMayaCmd::newSyntax()
{
MSyntax syntax;
syntax.addFlag(kStartFlag, kStartFlagLong, MSyntax::kBoolean);
syntax.addFlag(kDeviceNameFlag, kDeviceNameFlagLong, MSyntax::kString );
return syntax;
}
MStatus
NeuronForMayaCmd::parseArgs( const MArgList& args )
{
MStatus status = MStatus::kSuccess;
MArgDatabase argData(syntax(), args);
if (argData.isFlagSet(kStartFlag))
status = argData.getFlagArgument(kStartFlag, 0, mStart);
if( argData.isFlagSet(kDeviceNameFlag))
status = argData.getFlagArgument(kDeviceNameFlag, 0, mDeviceName );
return status;
}
MStatus NeuronForMayaCmd::doIt( const MArgList& args )
{
MStatus status;
status = parseArgs( args );
if( status != MStatus::kSuccess)
{
MGlobal::displayError( "parameters are not correct." );
return status;
}
MSelectionList sl;
sl.add( mDeviceName );
MObject deviceObj;
status = sl.getDependNode(0, deviceObj );
if(status != MStatus::kSuccess )
{
MGlobal::displayError("Please create your device first.");
return status;
}
MFnDependencyNode fnDevice(deviceObj);
MString ip = fnDevice.findPlug( "inputIp", &status ).asString();
int port = fnDevice.findPlug("inputPort", &status).asInt();
if( mStart ){
// to register the 3 callback to fetch data from Neuron
BRRegisterFrameDataCallback(NULL, NeuronForMayaDevice::myFrameDataReceived );
BRRegisterCommandDataCallback(NULL, NeuronForMayaDevice::myCommandDataReceived );
BRRegisterSocketStatusCallback (NULL, NeuronForMayaDevice::mySocketStatusChanged );
socketInfo = BRConnectTo(const_cast (ip.asChar()), port);
if(socketInfo == NULL )
MGlobal::displayError("Failed to connect to device.");
}
else
{
// stop socket
BRCloseSocket( socketInfo);
}
return MStatus::kSuccess;
}
Step 2 - To fetch the animation data, we create a customized device node (NeuronForMayaDevice) that is derived from MPxThreadedDeviceNode. This is the recommended way to receive data from any device, you can get the detail information on how to use this class at SDK online help.
Within the “NeuronForMayaDevice” class, we define 2 output array attributes, outputRotations and outputTranslations. Both of them are array of float3 type, containing the rotation & translation information of all the Neuron Skeleton bones for current frame. They will be connected to Neuron skeleton bones to stream animation data. But, we will talk about this later.
Here I need to mention 2 issues: the 1st issue is actually a limitation of Maya; to implement “NeuronForMayaDevice”, we need to call setRefreshOutputAttributes() and createMemoryPools() within the node’ postConstructor() function. The setRefreshOutputAttributes() takes an array, but there is a limitation that the array size only can be 1, so you cannot add attributes outputTranslations and outputRotations into the array directly. To workaround this limitation, I added an assistant attribute named outputData and make both outputTranslations & outputRotations be affected by this attribute. outputData will then be added into setRefreshOutputAttributes(), so when the threadHandler actually push data in the memory pool, the device thread will mark the “outputData” attribute dirty for that pool, and because our outputData attribute affects both outputTranslations and outputRotations attributes, Maya will mark them dirty too, and will call compute from the DG thread if someone needs the data.
Another issue is about the multiple threads access to the frame data buffer; as I mentioned in the previous post, the registered callback works in a separated working thread, so you cannot access UI within the callback. We will cache the skeleton data of current frame into a data buffer, and fetch the data buffer from the device thread. Here, we use the Maya system lock MSpinLock to implement the mechanism. If you want to know why and how to use the API, you can check the Threading and Maya section of SDK help for the details.
The threaded device node looks like this:
--- Declare NeuronForMayaDevice:
class NeuronForMayaDevice : public MPxThreadedDeviceNode
{
public:
NeuronForMayaDevice();
virtual ~NeuronForMayaDevice();
virtual void postConstructor();
virtual MStatus compute( const MPlug& plug, MDataBlock& data );
virtual void threadHandler();
virtual void threadShutdownHandler();
static void* creator();
static MStatus initialize();
// 3 callbacks to fetch data from Axis Neuron application
static void myFrameDataReceived(void* customedObj, SOCKET_REF sender, BvhDataHeaderEx* header, float* data);
static void myCommandDataReceived(void* customedObj, SOCKET_REF sender, CommandPack* pack, void* data);
static void mySocketStatusChanged(void* customedObj, SOCKET_REF sender, SocketStatus status, char* message);
public:
static MObject inputIp;
static MObject inputPort;
static MObject inputRecord;
static MObject outputTranslate;
static MObject outputTranslations;
static MObject outputRotations;
static MSpinLock spinLock;
static MTypeId id;
private:
static FrameData curFrameData;
static bool bRecord;
static bool bLive;
};
--- C++ implementation file for NeuronForMayaDevice
FrameData NeuronForMayaDevice::curFrameData;
MSpinLock NeuronForMayaDevice::spinLock;
bool NeuronForMayaDevice::bLive = false;
bool NeuronForMayaDevice::bRecord = false;
MTypeId NeuronForMayaDevice::id( 0x00081051 );
MObject NeuronForMayaDevice::inputIp;
MObject NeuronForMayaDevice::inputPort;
MObject NeuronForMayaDevice::inputRecord;
MObject NeuronForMayaDevice::outputTranslate;
MObject NeuronForMayaDevice::outputTranslations;
MObject NeuronForMayaDevice::outputRotations;
NeuronForMayaDevice::NeuronForMayaDevice()
{}
NeuronForMayaDevice::~NeuronForMayaDevice()
{
destroyMemoryPools();
}
void NeuronForMayaDevice::postConstructor()
{
MObjectArray attrArray;
attrArray.append( NeuronForMayaDevice::outputTranslate );
setRefreshOutputAttributes( attrArray );
createMemoryPools( 24, 1, sizeof(FrameData));
}
void NeuronForMayaDevice::threadHandler()
{
MStatus status;
setDone( false );
while ( ! isDone() )
{
// Skip processing if we are not live
if ( ! isLive() )
continue;
MCharBuffer buffer;
status = acquireDataStorage(buffer);
if ( ! status )
continue;
beginThreadLoop();
{
FrameData* frameData = reinterpret_cast(buffer.ptr());
// add mutex here
spinLock.lock();
frameData->nFrame = curFrameData.nFrame;
for(UINT32 i = 0; i < 60; ++i )
for( UINT32 j = 0; j< 6; j++ )
frameData->data[i][j] = curFrameData.data[i][j];
spinLock.unlock();
pushThreadData( buffer );
}
endThreadLoop();
}
setDone( true );
}
void NeuronForMayaDevice::threadShutdownHandler()
{
// Stops the loop in the thread handler
setDone( true );
}
void* NeuronForMayaDevice::creator()
{
return new NeuronForMayaDevice;
}
MStatus NeuronForMayaDevice::initialize()
{
MStatus status;
MFnNumericAttribute numAttr;
MFnTypedAttribute tAttr;
MFnStringData fnStringIp;
MObject stringIp = fnStringIp.create("127.0.0.1");
inputIp = tAttr.create("inputIp", "ii", MFnData::kString, stringIp, &status);
MCHECKERROR(status, "create input Ip");
tAttr.setWritable(true);
ADD_ATTRIBUTE(inputIp)
inputPort = numAttr.create("inputPort", "ip", MFnNumericData::kInt, 7001, &status );
MCHECKERROR(status, "create input Port");
numAttr.setWritable(true);
numAttr.setReadable(false);
//numAttr.setConnectable(false);
ADD_ATTRIBUTE(inputPort)
inputRecord = numAttr.create("record", "ird", MFnNumericData::kBoolean, false, &status );
MCHECKERROR(status, "create input Record");
numAttr.setWritable(true);
numAttr.setConnectable(false);
ADD_ATTRIBUTE(inputRecord)
outputTranslate = numAttr.create("outputTranslate", "ot", MFnNumericData::k3Float, 0.0, &status);
MCHECKERROR(status, "create outputTranslate");
numAttr.setHidden(true);
ADD_ATTRIBUTE(outputTranslate);
outputTranslations = numAttr.create("outputTranslations", "ots", MFnNumericData::k3Float, 0.0, &status);
MCHECKERROR(status, "create outputTranslations");
numAttr.setArray(true);
numAttr.setUsesArrayDataBuilder(true);
ADD_ATTRIBUTE(outputTranslations);
outputRotations = numAttr.create("outputRotations", "ors", MFnNumericData::k3Float, 0.0, &status);
MCHECKERROR(status, "create outputRotations");
numAttr.setArray(true);
numAttr.setUsesArrayDataBuilder(true);
ADD_ATTRIBUTE(outputRotations);
ATTRIBUTE_AFFECTS( live, outputTranslate);
ATTRIBUTE_AFFECTS( frameRate, outputTranslate);
ATTRIBUTE_AFFECTS( inputIp, outputTranslate);
ATTRIBUTE_AFFECTS( inputPort, outputTranslate);
ATTRIBUTE_AFFECTS( live, outputTranslations);
ATTRIBUTE_AFFECTS( frameRate, outputTranslations);
ATTRIBUTE_AFFECTS( inputIp, outputTranslations);
ATTRIBUTE_AFFECTS( inputPort, outputTranslations);
ATTRIBUTE_AFFECTS( live, outputRotations);
ATTRIBUTE_AFFECTS( frameRate, outputRotations);
ATTRIBUTE_AFFECTS( inputIp, outputRotations);
ATTRIBUTE_AFFECTS( inputPort, outputRotations);
ATTRIBUTE_AFFECTS( outputTranslate, outputTranslations);
ATTRIBUTE_AFFECTS( outputTranslate, outputRotations);
return MS::kSuccess;
}
MStatus NeuronForMayaDevice::compute( const MPlug& plug, MDataBlock& block )
{
MStatus status;
if( plug == outputTranslate || plug.parent() == outputTranslate ||
plug == outputTranslations || plug.parent() == outputTranslations ||
plug == outputRotations || plug.parent() == outputRotations )
{
bLive = isLive();
MCharBuffer buffer;
if ( popThreadData(buffer) )
{
FrameData* curData = reinterpret_cast(buffer.ptr());
printf( "current frame is %d \n ", curData->nFrame);
MArrayDataHandle translationsHandle = block.outputArrayValue( outputTranslations, &status );
MCHECKERROR(status, "Error in block.outputArrayValue for outputTranslations");
MArrayDataBuilder translationsBuilder = translationsHandle.builder( &status );
MCHECKERROR(status, "Error in translationsBuilder = translationsHandle.builder.\n");
MArrayDataHandle rotationsHandle = block.outputArrayValue( outputRotations, &status );
MCHECKERROR(status, "Error in block.outputArrayValue for outputRotations");
MArrayDataBuilder rotationsBuilder = rotationsHandle.builder( &status );
MCHECKERROR(status, "Error in rotationsBuilder = rotationsHandle.builder.\n");
for(UINT32 i=0; i< 60; ++i )
{
float3& translate = translationsBuilder.addElement(i, &status).asFloat3();
MCHECKERROR(status, "ERROR in translate = translationsBuilder.addElement.\n");
translate[0] = curData->data[i][0];
translate[1] = curData->data[i][1];
translate[2] = curData->data[i][2];
float3& rotate = rotationsBuilder.addElement(i, &status).asFloat3();
MCHECKERROR(status, "ERROR in translate = translationsBuilder.addElement.\n");
rotate[0] = curData->data[i][3];
rotate[1] = curData->data[i][4];
rotate[2] = curData->data[i][5];
}
status = translationsHandle.set(translationsBuilder);
MCHECKERROR(status, "set translationsBuilder failed\n");
status = rotationsHandle.set(rotationsBuilder);
MCHECKERROR(status, "set rotationsBuilder failed\n");
block.setClean( plug );
releaseDataStorage(buffer);
return ( MS::kSuccess );
}
else
{
return MS::kFailure;
}
}
return ( MS::kUnknownParameter );
}
Step 3 - So far, we have registered the callback to receive the animation data from the Axis neuron app, and cached the frame data as shared data buffer. Then, we created a device thread node to fetch the data and then pass them to 2 output attributes (outputTranslations, outputRotations). Now, we will create the Neuron specific skeleton, and connect each item in the translation and rotation arrays (outputTranslations, outputRotations) to the corresponding joint in the Neuron skeleton.
Neuron skeleton is a little different than the Maya internal skeleton. It supports 60 joints at most, and is corresponding with its MotionCapture device (Perception Neuron). You can reference the SDK Doc “Appendix B: BVH header template for the hierarchy” for more information. To create the Neuron skeleton, you can use several different ways to do that and make it more flexible. But here we will just create a Maya default skeleton, and export it to a MA file as a reference (by Reference -> Export Selection as Reference), then modify the MA file to generate the MEL procedure according to the Neuron skeleton hierarchy. It is not very flexible but quick and effective at this stage.
global proc createNeuronSkeleton(string $characterNodeName)
{
createNode transform -n ($characterNodeName + "_Character");
setAttr ".s" -type "double3" 0.1 0.1 0.1 ;
createNode joint -n ($characterNodeName + "_Hips") -p ($characterNodeName + "_Character");
addAttr -s false -ci true -sn "ch" -ln "Character" -at "message";
setAttr ".ro" 2;
setAttr ".typ" 1;
createNode joint -n ($characterNodeName + "_RightUpLeg") -p ($characterNodeName + "_Hips");
addAttr -s false -ci true -sn "ch" -ln "Character" -at "message";
setAttr ".ro" 2;
setAttr ".sd" 2;
setAttr ".typ" 2;
……
}
When the Neuron skeleton is created, we need to create a NeuronForMayaDevice node, and connect each item within outputTranslations and outputRotations attributes to the corresponding joint in the Neuron skeleton, the script should be like:
When the Neuron skeleton is created, we need to create a NeuronForMayaDevice node, and connect each item within outputTranslations and outputRotations attributes to the corresponding joint in the Neuron skeleton, the script should be like:
global proc createUnitconversions(string $characterNodeName)
{
int $i;
for($i = 0; $i < 60; ++$i)
{
createNode unitConversion -n ($characterNodeName + "_unitConversion" + $i);
setAttr ".cf" 0.017453292519943295;
}
}
global proc connectNodes(string $characterNodeName, string $neuronDeviceName)
{
int $i;
string $NeuronSkelNames[] = {"Character","Hips","RightUpLeg","RightLeg","RightFoot","LeftUpLeg","LeftLeg","LeftFoot","Spine","Spine1","Spine2","Spine3","Neck","Head","RightShoulder","RightArm","RightForeArm","RightHand","RightHandThumb1","RightHandThumb2","RightHandThumb3","RightInHandIndex","RightHandIndex1","RightHandIndex2","RightHandIndex3","RightInHandMiddle","RightHandMiddle1","RightHandMiddle2","RightHandMiddle3","RightInHandRing","RightHandRing1","RightHandRing2","RightHandRing3","RightInHandPinky","RightHandPinky1","RightHandPinky2","RightHandPinky3","LeftShoulder","LeftArm","LeftForeArm","LeftHand","LeftHandThumb1","LeftHandThumb2","LeftHandThumb3","LeftInHandIndex","LeftHandIndex1","LeftHandIndex2","LeftHandIndex3","LeftInHandMiddle","LeftHandMiddle1","LeftHandMiddle2","LeftHandMiddle3","LeftInHandRing","LeftHandRing1","LeftHandRing2","LeftHandRing3","LeftInHandPinky","LeftHandPinky1","LeftHandPinky2","LeftHandPinky3"};
for($i = 0; $i < 60; ++$i)
{
connectAttr ($neuronDeviceName + ".ots[" + $i + "]") ($characterNodeName + "_" + $NeuronSkelNames[$i] + ".t");
connectAttr ($neuronDeviceName + ".ors[" + $i + "]") ($characterNodeName + "_unitConversion" + $i + ".i");
connectAttr ($characterNodeName + "_unitConversion" + $i + ".o") ($characterNodeName + "_" + $NeuronSkelNames[$i] + ".r");;
}
}
If you have your Axis Neuron app running, set the IP and port, run command NeuronForMayaCmd to connect to the Axis Neuron app, make it live, you will see the Neuron Skeleton be animated like this:
Step 4 – At this stage everything should be ready to drive the Neuron skeleton with the real-time Motion Capture data from the device. Now, it is your turn to play with it ;). One of my colleagues created a wonderful Ironman model, and we retargeted the Neuron skeleton to the Ironman model to make it be animated. Here is the result we got. It works well, but not perfect, there are some issues we need to resolve. I will discuss about this in next post.
One more thing, you can access all the source code from Github if you are interested.
Just amazing, thanks for sharing!
Posted by: Raphael | November 16, 2015 at 04:24 PM