Difference between revisions of "TensorRT/PluginFAQ"

From eLinux.org
Jump to: navigation, search
m (Why we need clone() for Plugin interface?)
m (Why we need clone() for Plugin interface?)
 
Line 4: Line 4:
 
* The following shows the callback flow of all IPlugin key APIs from network parsing, to engine building to inferencing (take reference from TensorRT 5.1.5).<br /> NOTE: the digit ‘0x12458e00’ is the plugin object ptr and can be used to track when it gets destroyed.  
 
* The following shows the callback flow of all IPlugin key APIs from network parsing, to engine building to inferencing (take reference from TensorRT 5.1.5).<br /> NOTE: the digit ‘0x12458e00’ is the plugin object ptr and can be used to track when it gets destroyed.  
 
<pre>
 
<pre>
         '''parsing:'''
+
         parsing:
 
CropAndResizePluginCreator::createPlugin() --> 0x12458e00
 
CropAndResizePluginCreator::createPlugin() --> 0x12458e00
 
CropAndResizePlugin::CropAndResizePlugin()
 
CropAndResizePlugin::CropAndResizePlugin()
Line 16: Line 16:
 
CropAndResizePlugin::isOutputBroadcastAcrossBatch()
 
CropAndResizePlugin::isOutputBroadcastAcrossBatch()
 
CropAndResizePlugin::getOutputDimensions()
 
CropAndResizePlugin::getOutputDimensions()
'''parsing done'''
+
parsing done
  
  '''engine building:'''   --> graph optimization, tactic and format selection, memory optimization and etc.
+
  engine building:    --> graph optimization, tactic and format selection, memory optimization and etc.
 
CropAndResizePlugin::clone() --> 0xf9e8e0
 
CropAndResizePlugin::clone() --> 0xf9e8e0
 
CropAndResizePlugin::CropAndResizePlugin()
 
CropAndResizePlugin::CropAndResizePlugin()
Line 45: Line 45:
 
CropAndResizePlugin::destroy() --> 0x12459300
 
CropAndResizePlugin::destroy() --> 0x12459300
 
CropAndResizePlugin::~CropAndResizePlugin()
 
CropAndResizePlugin::~CropAndResizePlugin()
'''engine building done'''
+
engine building done
  
'''createExecutionContext and inference:'''
+
createExecutionContext and inference:
 
CropAndResizePlugin::clone() --> 0x166b2840
 
CropAndResizePlugin::clone() --> 0x166b2840
 
CropAndResizePlugin::CropAndResizePlugin()
 
CropAndResizePlugin::CropAndResizePlugin()
Line 61: Line 61:
 
CropAndResizePlugin::destroy() --> 0x6380f750
 
CropAndResizePlugin::destroy() --> 0x6380f750
 
CropAndResizePlugin::~CropAndResizePlugin()
 
CropAndResizePlugin::~CropAndResizePlugin()
'''Finished'''
+
Finished
 
</pre>
 
</pre>
 
* The following shows the callback flow from engine/plan deserializing to inferencing (take reference from TensorRT 5.1.5).  
 
* The following shows the callback flow from engine/plan deserializing to inferencing (take reference from TensorRT 5.1.5).  
 
<pre>
 
<pre>
Deserializing:
+
        Deserializing:
 
CropAndResizePluginCreator::deserializePlugin --> 0xd10c5b0
 
CropAndResizePluginCreator::deserializePlugin --> 0xd10c5b0
 
CropAndResizePlugin::CropAndResizePlugin
 
CropAndResizePlugin::CropAndResizePlugin

Latest revision as of 23:35, 20 August 2019

Why we need clone() for Plugin interface?

TensorRT API documentation provides an explanation about this. Simply, it intends to share immutable resources, like input/output dimensions, workspace ptr or weights and etc, among different execution contexts and these immutable resources or parameters are created per each engine, and copied over the invocation of clone().

  • The following shows the callback flow of all IPlugin key APIs from network parsing, to engine building to inferencing (take reference from TensorRT 5.1.5).
    NOTE: the digit ‘0x12458e00’ is the plugin object ptr and can be used to track when it gets destroyed.
        parsing:
		CropAndResizePluginCreator::createPlugin() --> 0x12458e00
		CropAndResizePlugin::CropAndResizePlugin()
		CropAndResizePlugin::setPluginNamespace
		CropAndResizePlugin::getNbOutputs
	
		CropAndResizePlugin::clone() --> 0x12459300
		CropAndResizePlugin::CropAndResizePlugin()
		CropAndResizePlugin::getNbOutputs()
		CropAndResizePlugin::getOutputDataType()
		CropAndResizePlugin::isOutputBroadcastAcrossBatch()
		CropAndResizePlugin::getOutputDimensions()
	parsing done

 	engine building:    --> graph optimization, tactic and format selection, memory optimization and etc.
		CropAndResizePlugin::clone() --> 0xf9e8e0
		CropAndResizePlugin::CropAndResizePlugin()
		CropAndResizePlugin::supportsFormat() 
		CropAndResizePlugin::getOutputDataType()	

		CropAndResizePlugin::clone() --> 0x6380f750
		CropAndResizePlugin::CropAndResizePlugin() 
		CropAndResizePlugin::supportsFormat()
		CropAndResizePlugin::configurePlugin()
		CropAndResizePlugin::getWorkspaceSize()
		CropAndResizePlugin::initialize()
		
		CropAndResizePlugin::getSerializationSize()
		CropAndResizePlugin::serialize()
		
		CropAndResizePlugin::destroy() --> 0xf9e8e0
		CropAndResizePlugin::~CropAndResizePlugin()
		
		CropAndResizePlugin::getSerializationSize()
		CropAndResizePlugin::serialize()
		
		CropAndResizePlugin::destroy() --> 0x12458e00
		CropAndResizePlugin::~CropAndResizePlugin()
		
		CropAndResizePlugin::destroy() --> 0x12459300
		CropAndResizePlugin::~CropAndResizePlugin()
	engine building done

	createExecutionContext and inference:
		CropAndResizePlugin::clone() --> 0x166b2840
		CropAndResizePlugin::CropAndResizePlugin()
		CropAndResizePlugin::attachToContext()
		CropAndResizePlugin::enqueue()
		
		ropAndResizePlugin::detachFromContext
		CropAndResizePlugin::terminate
		CropAndResizePlugin::destroy() --> 0x166b2840
		CropAndResizePlugin::~CropAndResizePlugin()
		
		CropAndResizePlugin::terminate()
		CropAndResizePlugin::destroy() --> 0x6380f750
		CropAndResizePlugin::~CropAndResizePlugin()
	Finished
  • The following shows the callback flow from engine/plan deserializing to inferencing (take reference from TensorRT 5.1.5).
        Deserializing:
		CropAndResizePluginCreator::deserializePlugin --> 0xd10c5b0
			CropAndResizePlugin::CropAndResizePlugin
		CropAndResizePlugin::initialize()	
	Deserializing done
	
	createExecutionContext and infer:
		CropAndResizePlugin::enqueue()
		CropAndResizePlugin::terminate()
		CropAndResizePlugin::destroy()
		CropAndResizePlugin::~CropAndResizePlugin()
	Finished

NOTE: TensorRT 5.1 has a known issue that initialize() and terminate() don’t appear in pairs (terminate() will be invoked twice by different objects). If you want to allocate immutable resource for your plugin in initialize(), you better make it as std::shared_ptr so that it won’t get released twice from terminate().