Holds properties for configuring a builder to produce an engine. More...
Public Member Functions | |
virtual void | setMinTimingIterations (int32_t minTiming)=0 |
Set the number of minimization iterations used when timing layers. More... | |
virtual int32_t | getMinTimingIterations () const =0 |
Query the number of minimization iterations. More... | |
virtual void | setAvgTimingIterations (int32_t avgTiming)=0 |
Set the number of averaging iterations used when timing layers. More... | |
virtual int32_t | getAvgTimingIterations () const =0 |
Query the number of averaging iterations. More... | |
virtual void | setEngineCapability (EngineCapability capability)=0 |
Configure the builder to target specified EngineCapability flow. More... | |
virtual EngineCapability | getEngineCapability () const =0 |
Query EngineCapability flow configured for the builder. More... | |
virtual void | setInt8Calibrator (IInt8Calibrator *calibrator)=0 |
Set Int8 Calibration interface. More... | |
virtual IInt8Calibrator * | getInt8Calibrator () const =0 |
Get Int8 Calibration interface. More... | |
virtual void | setMaxWorkspaceSize (std::size_t workspaceSize)=0 |
Set the maximum workspace size. More... | |
virtual std::size_t | getMaxWorkspaceSize () const =0 |
Get the maximum workspace size. More... | |
virtual void | setFlags (BuilderFlags builderFlags)=0 |
Set the build mode flags to turn on builder options for this network. More... | |
virtual BuilderFlags | getFlags () const =0 |
Get the build mode flags for this builder config. More... | |
virtual void | clearFlag (BuilderFlag builderFlag)=0 |
clear a single build mode flag. More... | |
virtual void | setFlag (BuilderFlag builderFlag)=0 |
Set a single build mode flag. More... | |
virtual bool | getFlag (BuilderFlag builderFlag) const =0 |
Returns true if the build mode flag is set. More... | |
virtual void | setDeviceType (const ILayer *layer, DeviceType deviceType)=0 |
Set the device that this layer must execute on. More... | |
virtual DeviceType | getDeviceType (const ILayer *layer) const =0 |
Get the device that this layer executes on. More... | |
virtual bool | isDeviceTypeSet (const ILayer *layer) const =0 |
whether the DeviceType has been explicitly set for this layer More... | |
virtual void | resetDeviceType (const ILayer *layer)=0 |
reset the DeviceType for this layer More... | |
virtual bool | canRunOnDLA (const ILayer *layer) const =0 |
Checks if a layer can run on DLA. More... | |
virtual void | setDLACore (int32_t dlaCore)=0 |
Sets the DLA core used by the network. More... | |
virtual int32_t | getDLACore () const =0 |
Get the DLA core that the engine executes on. More... | |
virtual void | setDefaultDeviceType (DeviceType deviceType)=0 |
Sets the default DeviceType to be used by the builder. More... | |
virtual DeviceType | getDefaultDeviceType () const =0 |
Get the default DeviceType which was set by setDefaultDeviceType. More... | |
virtual void | reset ()=0 |
Resets the builder configuration to defaults. More... | |
virtual void | destroy ()=0 |
De-allocates any internally allocated memory. More... | |
virtual void | setProfileStream (const cudaStream_t stream)=0 |
Set the cudaStream that is used to profile this network. More... | |
virtual cudaStream_t | getProfileStream () const =0 |
Get the cudaStream that is used to profile this network. More... | |
virtual int32_t | addOptimizationProfile (const IOptimizationProfile *profile) noexcept=0 |
Add an optimization profile. More... | |
virtual int32_t | getNbOptimizationProfiles () const noexcept=0 |
Get number of optimization profiles. More... | |
virtual void | setProfilingVerbosity (ProfilingVerbosity verbosity)=0 |
Set verbosity level of layer information exposed in NVTX annotations. More... | |
virtual ProfilingVerbosity | getProfilingVerbosity () const =0 |
Get verbosity level of layer information exposed in NVTX annotations. More... | |
virtual void | setAlgorithmSelector (IAlgorithmSelector *selector)=0 |
Set Algorithm Selector. More... | |
virtual IAlgorithmSelector * | getAlgorithmSelector () const =0 |
Get Algorithm Selector. More... | |
virtual bool | setCalibrationProfile (const IOptimizationProfile *profile) noexcept=0 |
Add a calibration profile. More... | |
virtual const IOptimizationProfile * | getCalibrationProfile () noexcept=0 |
Get the current calibration profile. More... | |
virtual void | setQuantizationFlags (QuantizationFlags flags)=0 |
Set the quantization flags. More... | |
virtual QuantizationFlags | getQuantizationFlags () const =0 |
Get the quantization flags. More... | |
virtual void | clearQuantizationFlag (QuantizationFlag flag)=0 |
clear a quantization flag. More... | |
virtual void | setQuantizationFlag (QuantizationFlag flag)=0 |
Set a single quantization flag. More... | |
virtual bool | getQuantizationFlag (QuantizationFlag flag) const =0 |
Returns true if the quantization flag is set. More... | |
virtual bool | setTacticSources (TacticSources tacticSources)=0 |
Set tactic sources. More... | |
virtual TacticSources | getTacticSources () const =0 |
Get tactic sources. More... | |
Protected Member Functions | |
virtual | ~IBuilderConfig () |
Holds properties for configuring a builder to produce an engine.
|
inlineprotectedvirtual |
|
pure virtual |
Set the number of minimization iterations used when timing layers.
When timing layers, the builder minimizes over a set of average times for layer execution. This parameter controls the number of iterations used in minimization. The builder may sometimes run layers for more iterations to improve timing accuracy if this parameter is set to a small value and the runtime of the layer is short.
|
pure virtual |
Query the number of minimization iterations.
By default the minimum number of iterations is 2.
|
pure virtual |
Set the number of averaging iterations used when timing layers.
When timing layers, the builder minimizes over a set of average times for layer execution. This parameter controls the number of iterations used in averaging.
|
pure virtual |
Query the number of averaging iterations.
By default the number of averaging iterations is 1.
|
pure virtual |
Configure the builder to target specified EngineCapability flow.
The flow means a sequence of API calls that allow an application to set up a runtime, engine, and execution context in order to run inference.
The supported flows are specified in the EngineCapability enum.
|
pure virtual |
Query EngineCapability flow configured for the builder.
By default it returns EngineCapability::kDEFAULT.
|
pure virtual |
Set Int8 Calibration interface.
The calibrator is to minimize the information loss during the INT8 quantization process.
|
pure virtual |
Get Int8 Calibration interface.
|
pure virtual |
Set the maximum workspace size.
workspaceSize | The maximum GPU temporary memory which the engine can use at execution time. |
|
pure virtual |
Get the maximum workspace size.
By default the workspace size is 0, which means there is no temporary memory.
|
pure virtual |
Set the build mode flags to turn on builder options for this network.
The flags are listed in the BuilderFlags enum. The flags set configuration options to build the network.
builderFlags | The build option for an engine. |
|
pure virtual |
Get the build mode flags for this builder config.
Defaults to 0.
|
pure virtual |
clear a single build mode flag.
clears the builder mode flag from the enabled flags.
|
pure virtual |
Set a single build mode flag.
Add the input builder mode flag to the already enabled flags.
|
pure virtual |
Returns true if the build mode flag is set.
|
pure virtual |
Set the device that this layer must execute on.
deviceType | that this layer must execute on. If DeviceType is not set or is reset, TensorRT will use the default DeviceType set in the builder. |
|
pure virtual |
Get the device that this layer executes on.
|
pure virtual |
whether the DeviceType has been explicitly set for this layer
|
pure virtual |
reset the DeviceType for this layer
|
pure virtual |
Checks if a layer can run on DLA.
|
pure virtual |
Sets the DLA core used by the network.
dlaCore | The DLA core to execute the engine on (0 to N-1). Default value is 0. |
It can be used to specify which DLA core to use via indexing, if multiple DLA cores are available.
|
pure virtual |
Get the DLA core that the engine executes on.
|
pure virtual |
Sets the default DeviceType to be used by the builder.
It ensures that all the layers that can run on this device will run on it, unless setDeviceType is used to override the default DeviceType for a layer.
|
pure virtual |
Get the default DeviceType which was set by setDefaultDeviceType.
By default it returns DeviceType::kGPU.
|
pure virtual |
Resets the builder configuration to defaults.
When initializing a builder config object, we can call this function.
|
pure virtual |
De-allocates any internally allocated memory.
When destroying a builder config object, we can call this function.
|
pure virtual |
Set the cudaStream that is used to profile this network.
stream | The cuda stream used for profiling by the builder. |
|
pure virtual |
Get the cudaStream that is used to profile this network.
|
pure virtualnoexcept |
Add an optimization profile.
This function must be called at least once if the network has dynamic or shape input tensors. This function may be called at most once when building a refittable engine, as more than a single optimization profile are not supported for refittable engines.
profile | The new optimization profile, which must satisfy profile->isValid() == true |
|
pure virtualnoexcept |
Get number of optimization profiles.
This is one higher than the index of the last optimization profile that has be defined (or zero, if none has been defined yet).
|
pure virtual |
Set verbosity level of layer information exposed in NVTX annotations.
Control how much layer information will be exposed in NVTX annotations.
|
pure virtual |
Get verbosity level of layer information exposed in NVTX annotations.
Get the current setting of verbosity level of layer information exposed in NVTX annotations. Default value is ProfilingVerbosity::kDEFAULT.
|
pure virtual |
Set Algorithm Selector.
selector | The algorithm selector to be set in the build config. |
|
pure virtual |
Get Algorithm Selector.
|
pure virtualnoexcept |
Add a calibration profile.
Calibration optimization profile must be set if int8 calibration is used to set scales for a network with runtime dimensions.
profile | The new calibration profile, which must satisfy profile->isValid() == true or be nullptr. MIN and MAX values will be overwritten by kOPT. |
|
pure virtualnoexcept |
Get the current calibration profile.
|
pure virtual |
Set the quantization flags.
The flags are listed in the QuantizationFlag enum. The flags set configuration options to quantize the network in int8.
flags | The quantization flags. |
|
pure virtual |
Get the quantization flags.
|
pure virtual |
clear a quantization flag.
Clears the quantization flag from the enabled quantization flags.
|
pure virtual |
Set a single quantization flag.
Add the input quantization flag to the already enabled quantization flags.
|
pure virtual |
Returns true if the quantization flag is set.
|
pure virtual |
Set tactic sources.
This bitset controls which tactic sources TensorRT is allowed to use for tactic selection.
By default, kCUBLAS is always enabled. kCUBLAS_LT is enabled for x86 platforms, as well as non-x86 platforms if CUDA >= 11.0
Multiple tactic sources may be combined with a bitwise OR operation. For example, to enable cublas and cublasLt as tactic sources, use a value of:
1U << static_cast<uint32_t>(TacticSource::kCUBLAS) | 1U << static_cast<uint32_t>(TacticSource::kCUBLAS_LT)
|
pure virtual |
Get tactic sources.
Get the tactic sources currently set in the engine build configuration.